Serenity-IRC Teaching This purpose of this class is to introduce you to the principles of Routing as it is a key part of the Serenity-IRC network. It will provide you with the knowledge to both identify and detect when servers need attention. It will also introduce you to the basic commands available to Operators of all levels, and strategies for when events occur. 1. Basics of routing on the Serenity-IRC network The Serenity-IRC network as we see it as end users, is as a continuous IRC network where all users around the world come together. The Serenity-IRC network is made up of servers running our very own flavour of Internet Relay Chat Daemon called SereneIRCD. This software handles the intricacies of users connecting to it, but also in talking to other servers to create this "one" network we call Serenity-IRC. The majority of the servers on our network are referred to as "Client Servers". These are servers which allow users to connect and chat. They are part of the "pool", the collection of servers that do accept client connects. Our main pool's DNS address is irc.Serenity-IRC.net. The other type of server is called a "Hub Server". These help route traffic to and from individual servers rather than passing everything down a single line, if all the servers just "held hands". At present, most of our servers are configured to be hubs if we need them to be, those configured as hubs are Serenity, Nightfall, Aurora, Anvil, and Gainesville 2. Basic Routing Tools There are several simple commands that you will use to get basic information back from the network regarding its status. 2.1 Routing map At regular intervals, the Routing team sends out an update of the current configuration of the Serenity-IRC network. They list the servers and also which are the Primary and Secondary servers they should connect to. The Primary is the preferred choice of connection for each server, the Secondary being a backup if needs arise. Client servers will attempt to connect to their Primary and Secondary servers if they get disconnected. Hub servers are manually connected. 2.2 /map and /links The trusty Serenity-IRC staff member, has these two commands at his/her disposal to keep an eye on the global situation of our servers. Each presents the same basic information, but in its own way. Either command will show you the other servers connected to Serenity-IRC, relative to wherever you are logged in. A small item of Networking jargon might be useful at this stage. That is that the way that a network is made up can sometimes be referred to like a tree. The structure of the network consists of "branches" which are the connections to each server, and "leaves" which are the Client servers. As our network has, at the moment, two main Hub servers we can't say that a single Hub server counts as the "trunk" of the tree, but you get the idea :) ./links will display the result in its own window. The information presented is the full server name, a number, and then the brief description line that each server has. The number in brackets represents how many "hops" that particular server is away from you. Your own server is zero hops away from you, for obvious reasons, and so on. Example: Nightfall.FL.US.Serenity-IRC.Net (1) - Clear nights, endless stars surround you /map will display the result in your Status window. This information too shows you the leaves of the tree, with a little ASCII art thrown in to graphically show the links. It also shows you the number of users connected to each server and the percentage of total load that server is holding for the whole network. [21:42.23] --- Aurora.CA.US.Serenity-IRC.Net (Users: 50) (44%) [21:42.23] --- |-Nightshade.CA.US.Serenity-IRC.Net (Users: 13) (11%) [21:42.23] --- |-Anvil.NJ.US.Serenity-IRC.Net (Users: 21) (18%) [21:42.23] --- |-ChickenLittle.NL.EU.Serenity-IRC.Net (Users: 13) (11%) [21:42.23] --- | `-MrPotato.NL.EU.Serenity-IRC.Net (Users: 6) ( 5%) [21:42.23] --- | `-CowZone.UK.EU.Serenity-IRC.Net (Users: 4) ( 3%) [21:42.23] --- `-Services.Serenity-IRC.Net (Users: 6) ( 5%) [21:42.23] --- End of /MAP You may now want to run your own fresh copy of /map also. It is this Users information that can sometimes be useful for spotting potential Denial of Service (DoS) attacks on the network. 2.3 The /connect command Let's deal with connect first, as this is the more used of the two commands. It is the Duck Tape of the Routing world. Connect allows you to join a split server back onto the network. You must refer to the Routing Map before attempting to connect the server to just anywhere. If you have just seen a server disconnect from its Primary connection, don't automatically stick it back to the Primary. It didn't disconnect without a reason. So try the Secondary. And visa versa if it was already on its Secondary. The connect command is used in two ways, "remote connect" and "local connect". 2.3.1 Remote connect If you have just seen a server disconnect from its hub, and you are "remote", ie not on either server, then you can issue a "remote connect" command. /CONNECT client.* port hub.* Example: /CONNECT anvil.* 7827 aurora.* The port of 7827 is the port that servers accept other server connections from, not client connections. 2.3.2 Local connect If in the previous example, you happened to be Opered on Aurora.* you could issue a local connect to bolt the split server on to Aurora. /CONNECT anvil.* 7827 No need for the hub.* variable as the command will attempt to connect the server to your server. Also command will work without port number. 2.4 The /squit command If /connect is the Duck Tape, then Squit is the sharp knife held against the twine holding together this peaceful network. This command is not to be used lightly or without knowing the implications and the reasons. However, I include it for completeness as it is briefly touched on in the Oper Manual so you deserve to know what it does. /squit will disconnect and sever a link between two servers. As you might be able to see, to sever a link between two servers can have serious consequences, even if all it means is an influx of users into #Serenity-IRC wondering if there's a problem. However there are times when an squit is considered. 1) If a hub is about to be taken out of commission, then squit-ing a server to then manually reconnect it to its Secondary or other link. 2) If the connection to the server is intermittent and a Jupe needs to be placed on the server. 3) If the connection to a hub is providing a lagging link, but not sufficient for the server to be dropped, as a last resort, an squit to allow the server to be reconnected to its alternative hub might be considered. The command is : /squit server.* Reason Example: /squit nightshade.* Rerouting to alternative hub Please be careful with your typing when squit-ing. As a server is allowed to be shortened just to its name followed by .*, the addition of an asterisk can lead to some bad wildcards. In the past, there is tale of an attempt to squit Serenity which ended up being /squit ser* which would have lead to Services being squit too. 3. Server Splits So then. All should be fine in the land of routing. Servers happily talking amongst themselves, making sure all the messages are delivered to the right places. But what causes servers to disappear or disconnect? Example [13:09.06] --- *** Global -- from MrPotato.NL.EU.Serenity-IRC.Net: No response from CowZone.UK.EU.Serenity-IRC.Net[0.0.0.0], closing link [13:09.06] --- *** Global -- from ChickenLittle.NL.EU.Serenity-IRC.Net: Server MrPotato.NL.EU.Serenity-IRC.Net[0.0.0.0] closed the connection Well why would a server split? Where has it gone? Gone to the local shop for a newspaper? 1. The server could be dead. - Someone might have unplugged it to plug the kettle in. 2. Software or server problems. - Maintenance work is being carried out. Reboots, upgrades, or people just not realising the IRCD is running. 3. A communications or routing error on the internet. - The path between the two servers has gotten so lagged that the IRCD has given up on it. 3.1 What to do when a server splits off First things first, reach for your Basic Routing Tools, most important of which is your Routing Map. Discover which server has split off and also if it is a Hub or a Client server. This is usually announced in Global notices. Hub Servers are connected to each other and are not configured to AutoConnect. Client Servers are connected to the Hubs and have a default AutoConnect to their Primary Server. Other methods for discovering which server has split include - checking the /map against the Routing map - checking with each Hub server to see what is missing from its viewpoint by using /stats x hub.* (example, /stats x nightfall.*) 3.1.1 A Client Server has split off First off, find out whether it had been connected to its Primary hub or its Secondary hub before it disconnected. This should be announced in the Global notice if you were around to see that. If it was already connected to its Primary Hub, you need to Remote Connect or Local Connect it to its Secondary hub. If it was not connected to its Primary hub, you can leave it and the Client Server will auto-reconnect to its Primary Hub. 3.1.2 A Hub Server has split off Issue a Remote Connect or Local Connect to link the splitted hub to its Primary or Secondary Hub, referring to your Routing map. 3.2 "It's Dead Jim, kick it if you don't believe me" Ok, how can you tell if a server is dead, or has just taken really bad acting lessons and is hiding behind a polystyrene rock to avoid being killed as its just realised its the unnamed crew member on an Away team? (and that's not a team who have gone /away either :P) If your patient is a split-off Client Server, then you can try connecting with Telnet to port 7827 to see if ircd is up and running telnet Aurora.CA.US.Serenity-IRC.Net 7827 If you get back something equivalent to, :Aurora.CA.US.Serenity-IRC.Net NOTICE AUTH :*** Looking up your hostname... :Aurora.CA.US.Serenity-IRC.Net NOTICE AUTH :*** Found your hostname or Trying 66.40.130.103... Connected to 66.40.130.103 this is a good sign as the IRCD is still up, working and accepting connections. So if it is still split off from the network, then its time to get our your Routing tools and bring it back home to Serenity. If on the other hand you get.... Unable to connect to remote host : Connection refused this is probably a fair indication that all is not well with that Server. In case the Serenity-IRC DNS server is down, you can get the IP address for the hub in question. The easiest way to get hold of that, you can use the /stats c command. /stats c will show you the IP addresses and names of the other servers the server you are Oper'd on at the moment will connect to. /stats c client.* will show you the same information for a remote server. Example: /stats c Aurora.* [22:26.14] --- C *@==== Aurora ==== * ==== Services ==== 0 0 [22:26.14] --- N *@66.40.130.103 * Services.Serenity-IRC.Net 0 20 [22:26.14] --- C *@66.40.130.103 * Services.Serenity-IRC.Net 0 20 [22:26.14] --- C *@==== Aurora ==== * ==== Primary ==== 0 0 [22:26.14] --- N *@209.133.9.65 * Anvil.NJ.US.Serenity-IRC.Net 0 50 [22:26.14] --- C *@209.133.9.65 * Anvil.NJ.US.Serenity-IRC.Net 7827 50 [22:26.14] --- C *@==== Aurora ==== * ==== Secondary ==== 0 0 [22:26.14] --- N *@72.20.44.126 * Bebop-n-Boogie.CA.US.Serenity-IRC.Net 0 20 [22:26.14] --- C *@72.20.44.126 * Bebop-n-Boogie.CA.US.Serenity-IRC.Net 7827 20 [22:26.14] --- C *@==== Aurora ==== * ==== Leaves ==== 0 0 [22:26.14] --- N *@79.170.91.47 * ChickenLittle.NL.EU.Serenity-IRC.Net 0 20 [22:26.14] --- C *@79.170.91.47 * ChickenLittle.NL.EU.Serenity-IRC.Net 7827 20 [22:26.14] --- N *@89.16.173.177 * CowZone.UK.EU.Serenity-IRC.Net 0 40 [22:26.14] --- C *@89.16.173.177 * CowZone.UK.EU.Serenity-IRC.Net 7827 40 [22:26.14] --- N *@212.79.239.43 * MrPotato.NL.EU.Serenity-IRC.Net 0 30 [22:26.14] --- C *@212.79.239.43 * MrPotato.NL.EU.Serenity-IRC.Net 7827 30 [22:26.14] --- N *@72.20.35.195 * Nightshade.CA.US.Serenity-IRC.Net 0 30 [22:26.14] --- C *@72.20.35.195 * Nightshade.CA.US.Serenity-IRC.Net 7827 30 [22:26.14] --- c :End of /STATS report This lists a servers C/N lines, which I'll avoid the lengthy explanation of, suffice to say here is a handy reference of the Primary and Secondary hubs of Aurora.* Now you can check if Aurora is awake by telnet 66.40.130.103 7827, for example 4. IRCD.conf There was a request for a brief account of what goes into making up an ircd.conf file. ircd.conf is a text file that controls how the IRCD process runs on each server. It consists of a load of lines, each prefixed by a capital letter to define different types of settings. I'll quickly run through the different types and what they do. M line - This sets the server name, description and the port number it runs on. A line - Sets the server administrator's information. Y line - These define different types of connection class. We use them to define different thresholds for users, opers and other servers etc connecting to servers. Y lines include information on how often a silent client should be Ping'd, how quickly users can connect on it... and also how much information is allowed back up in a send queue before it finally decides enough is enough and disconnects the connection. I line - No, not some kind of makeup, but define who may connect to the server. Usually left set as *@* P line - Sets which ports the server will listen for traffic on. We usually operate 6667, 6668, 6669 and 7000. One of these will have been set in the M line. O line - If you don't know what an O line is... what are you doing here? Ok, just in case you're really new, O lines set out who can be Oper'd up on a server. And you have one to be here (I hope) X line - like the little X in the corner of your software to close it, this sets out how to restart or kill a server. U line - This is the secret service line to a server! It sets where the Services server is so it can listen out for secret commands to amend channel modes etc. C and N lines - Another commonly talked about set of lines. These set out connections to other servers. They have an IP address, password and also a Y line associated with each one. What's the difference? Well C lines say who your server can Connect to, and N lines say who your server can accept coNnections from :) (tenuous use of Caps) H line - Sets which servers can be hubs when connected. K line - Kill lines are specified only in ircd.conf files for commonly bad things we want to keep away, or users who have really done a lot to earn our disrespect. Common K lines hard coded into ircd.conf files are usually protecting us against abusive idents. A line - These are the Autokill lines, which are temporary and not set within ircd.conf, but are set by the U lined server (see if you've been paying attention :P) Q line - These reserve nicknames that we either want for the network, or don't want users to be using. Common Q line nicknames involve abusive nicknames. As Q lines are local to each server, then NickServ can be used to quickly ban nicknames across the whole network rather than edit and rehash each server. 5. Summary In this class we have covered the following: The basics of routing Routing tools - Routing map, /map and /links /Connect - remote connect and local connect /squit Server splits - how to detect them and how to fix them How to check if a server is dead and you have also been introduced to /stats x hub.* and /stats c client.* A quick summary of each of the types of line you might find in an ircd.conf file. It is very important to refer to the Routing map, and also being confident with what you are attempting. The golden rule with being an effective Op on Serenity-IRC is, if you don't know or are not 100% sure of what you are doing, ASK FIRST. Many thanks for paying attention through this detailed class. However I hope that you will feel more confident to know what to do next time you see a server split and 200 screaming users end up at the bottom of the Log Ride... wondering when they are going to get back to the theme park and be presented with a ill-taken flash photo of their little jaunt screaming around the corridors of the Internet. However, if they do come back into #help from their trip, shouts of "arms in the air" will be lost on them... but might make you chuckle... Have fun References: -- ircd.conf Programming -- (written for DALnet release ircd) v.01 by Roddy Vagg, updated by Remmy