Handling Crashes in InspIRCd
From the makers of InspIRCd.
Contents |
Introduction
Before I start, I apologise for some of this manual going off the right of the screen.
If InspIRCd causes a Segmentation Fault this is usually indicted by the message:
[07:25] * Error! Segmentation fault! save meeeeeeeeeeeeee *splat!*
Or maybe in the logfile:
Sat Apr 22 18:16:57 2006: *** fell down a pothole in the road to perfection ***
The best method of bug fixing is to perform certain debugging operations on it. After just coming across a segfault in one of InspIRCd's modules, i thought i would write this tutorial to describe how i tracked down the problem.
Coredumps
NOTE: InspIRCd, at least in later versions, should try to generate coredumps, however using gdb is sometimes more reliable, even if a core dump is made it is generally recommended you use the following method.
But if you're set on loading the core dump yourself, then run:
gdb bin/inspircd /path/to/inspircd.core
then skip to the part of this manual which reads:
Program received signal SIGSEGV, Segmentation fault.
NOTE: Please don't just paste core dumps to our bugtracker, it's very unlikely we'll be able to make use of them on our systems, a backtrace is much more helpful!
Using gdb
If you know what action or command caused InspIRCd to crash, continue reading.
The first thing we need to do is start a debugger. In this case, we will use gdb to assist us. From the inspircd directory, fire up gdb. All versions of InspIRCd past 1.0.3 (or older?) should be able to handle launching gdb for you, simply run:
me@mymachine:~/build/inspircd$ ./inspircd debug
And the launcher script will open gdb and launch InspIRCd for you, the launcher script automatically forces debug logging, so you'll get a great big heap of debug logging. This is good, it helps debug your crash.
NOTE: InspIRCd can generate a lot of debug logging, and by a lot, we mean a lot, you'll need to watch out for it filling up your disk.
Your IRCd is now running. You must now connect to it, and force it to crash (always a fun job). When the IRCd crashes, there will be NO OUTPUT on IRC, it will appear to 'hang', but gdb will say otherwise.
Getting Crash Infomation
Now we're in gdb with either a corefile or a just-crashed InspIRCd, we can see where the crash happened:
Program received signal SIGSEGV, Segmentation fault. 0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46 46 Srv->SendServ(user->fd, buf);
Instantly, this tells us where the problem has occured, the file, and line in question.
IF YOU HAVE SET OPTIMISATION FLAGS ON, THIS DATA WILL NOT BE HERE! PLEASE RE-COMPILE THE IRCD WITH THEM OFF AND TRY AGAIN
If you have received this data, we can get a little more info. The first thing required would be a backtrace, to describe the events leading up to the crash.
(gdb) bt #0 0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46 #1 0x2835fcd1 in ModuleRandQuote::OnUserConnect (this=0x80f8550, user=0x810b000) at m_randquote.cpp:101 #2 0x0807fb3a in ConnectUser (user=0x810b000) at inspircd.cpp:2429 #3 0x0809e5ca in handle_user (parameters=0xbfbd8030, pcnt=4, user=0x810b000) at commands.cpp:1063 #4 0x08081529 in process_command (user=0x810b000, cmd=0xbfbd8280 "USER") at inspircd.cpp:2872 #5 0x08081e62 in process_buffer (cmdbuf=0xbfbd84b0 "USER Craig i-br0ked-it 192.168.0.1 :Craig McLure", user=0x810b000) at inspircd.cpp:3009 #6 0x08084a9d in InspIRCd () at inspircd.cpp:3613 #7 0x0807dbfd in main (argc=2, argv=0xbfbfeb80) at inspircd.cpp:2067 (gdb)
Although most of this won't make any sense to you, it is important to the developers.
Some other useful data would include the value of some variables, in this case we see the segfault is caused by a Srv->SendServ(); so something useful to know would be what the module was trying to send. For this, we use the command 'print buf' which will give us the details of what was stored in the variable 'buf' when the IRCd Crashed.
(gdb) print buf $1 = "NOTICE Craig :Linux is only free if your time is worthless\000ay file is missing.\000EN=307 CHANMODES=qaohvb,Lk,l,psmntirRMcKCQTNVGS NETWORK=ChatSpike SILENCE=999 OVERRIDE KNOCK :are supported by this serve"...
Giving any other variables would be useful, for example user->fd.
Writing a Bug Report
From all this data, it is now possible to build something very helpful for the bugtracker, here is what i'm about to submit:
I've been experiencing problems with m_randquote crashing with a segfault whenever a user connects. I've used gdb to retreive some info into the crashing, some output is below: Program received signal SIGSEGV, Segmentation fault. 0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46 46 Srv->SendServ(user->fd, buf); (gdb) bt #0 0x2835f071 in handle_randquote (parameters=0xbfbd7500, pcntl=0, user=0x810b000) at m_randquote.cpp:46 #1 0x2835fcd1 in ModuleRandQuote::OnUserConnect (this=0x80f8550, user=0x810b000) at m_randquote.cpp:101 #2 0x0807fb3a in ConnectUser (user=0x810b000) at inspircd.cpp:2429 #3 0x0809e5ca in handle_user (parameters=0xbfbd8030, pcnt=4, user=0x810b000) at commands.cpp:1063 #4 0x08081529 in process_command (user=0x810b000, cmd=0xbfbd8280 "USER") at inspircd.cpp:2872 #5 0x08081e62 in process_buffer (cmdbuf=0xbfbd84b0 "USER Craig i-br0ked-it 192.168.0.1 :Craig McLure", user=0x810b000) at inspircd.cpp:3009 #6 0x08084a9d in InspIRCd () at inspircd.cpp:3613 #7 0x0807dbfd in main (argc=2, argv=0xbfbfeb80) at inspircd.cpp:2067 (gdb) print buf $1 = "NOTICE Craig :Linux is only free if your time is worthless\000ay file is missing.\000EN=307 CHANMODES=... (gdb) print user->fd $2 = 13 (gdb) The Text when connecting: * Your host is frostycoolslug.chatspike.net, running version InspIRCd-1.0(Beta1) * This server was created 11:56:55 Mar 28 2005 * frostycoolslug.chatspike.net InspIRCd-1.0(Beta1) iowghraAsORVSxNCWqBzvdHtGI lvhopsmntikrRcaqOALQbSeKVfHGCuzN * MESHED WALLCHOPS MODES=13 CHANTYPES=# PREFIX=(ohv)@%+ MAP SAFELIST MAXCHANNELS=20 MAXBANS=60 NICKLEN=32 TOPICLEN=307 KICKLEN=307 MAXTARGETS=20 :are supported by this server * AWAYLEN=307 CHANMODES=qaohvb,... * Message of the day file is missing. * Craig sets modes [Craig :+x] * Error! Segmentation fault! save meeeeeeeeeeeeee *splat!*
Including the 005 numeric (the line containing the MESHED token) is always of use, and so sometimes is /modules output as these indicate other modules which may be changing the outcome of the program.
I can almost guarantee that this bug will be corrected in under 24 hours of it being reported, as a lot of information is provided as to the cause making finding a solution easier. just saying "InspIRCd crashes on connect when m_randquote.so is loaded" would mean we would have to spend significant time tracking it down, and doing testing ourselves.
As a final note, this bug was fixed within three hours of adding it to the bugtracker, seems two declerations of Srv were at fault :)
Notes
Using screen
If you're having problems getting InspIRCd to generate corefiles, quite possible on limited shell hosts and the like, then you may want to run Insp inside 'screen', this lets you run it inside gdb without having to have a terminal permenantly open, but still allowing you to get backtraces if or when it crashes. Brain tells me that all you have to do is:
(@LaptopBrain) om, its easy, screen ./inspircd debug (@LaptopBrain) and screen -r to resume later (@LaptopBrain) (no other params unless you have other screened sessions)
So just run the command:
screen ./inspircd debug
And then once you've got the IRCd to crash:
screen -r
And continue from as if you'd run `./inspircd debug` normally.
Of course, this is coming from Brain, so poke him if it crashes and burns horribly. Not me.
Logfiles
Just to reiterate here, Insp's debug logfiles can get very, very large, watch out!
Obtaining backtraces using other debuggers
XDB
$ xdb bin/inspircd core (xdb) t
DBX
$ dbx bin/inspircd core (dbx) where
SDB
$ sdb bin/inspircd core (sdb) t
DEBUG
$ debug -c core bin/inspircd debug> stack
DDE
$ dde -ui line core bin/inspircd dde> bt
ADB
$ adb bin/inspircd core $c
Note: In this example the literal $c must be typed.
