Bug / GDB / Memory Leak

Posted by Greven on Wed 20 Aug 2003 07:39 PM — 25 posts, 79,940 views.

Greven Canada Wed 20 Aug 2003 07:39 PM #0

Hello, I don't think this was posted elsewhere, but if it was, please excuse me.

I am using Samson's Dynamic Command Support snippet on my SWR 1 modified mud. I am running it through a shell account running Slackware Linux 7.1, and compiling with gcc. I seem to have a memory leak somewhere, and its causing me problems.

Whenever it is supposed to give me a message with a call to the bug function, I am crashing. Using gdb, it tells me this:


#0  0x2ab64c44 in _IO_seekoff () from /lib/libc.so.6
#1  0x2ab64207 in ftell () from /lib/libc.so.6
#2  0x80dcc15 in bug (str=0x822bc60 "Error locating %s in symbol table. %s") at db.c:3871
#3  0x81a3961 in skill_function (name=0x7ffff53f "adsfasdfasf") at tables.c:51
#4  0x8097440 in do_cedit (ch=0x86534e8, argument=0x7ffff53f "adsfasdfasf") at act_wiz.c:6527

When I frame 2, it tells me the the code that seems to be bugging it is:


	    iChar = ftell( fpArea );

I did a print on iChar, and got :

$3 = 2147466537

And a print on fpArea produced:

$4 = (FILE *) 0x860b310

This whole thing only oocurs when one of the prototype areas are loaded. Otherwise, its not a problem.

I don't understand memory management very well, so if anyone has any suggestions on what I might do to track down this problem, and memory leaks in general, because I seem to have a lot of the, it would be muchly appreciated.

Thanks

Greven Canada Wed 20 Aug 2003 07:42 PM #1

Please excuse the code tags, I forgot to click the box

Nick Gammon Australia Forum Administrator Wed 20 Aug 2003 11:02 PM #2

You know you can edit posts? I edited it and ticked the box for you.

Sounds like a crash, not a memory leak. Sounds like fpArea might not be a valid file pointer, for some reason. Hard to say why.

Greven Canada Thu 21 Aug 2003 01:33 AM #3

I don't understand how fpArea is used in db.c. In the bug function, fpArea isn't initialized as anything, but it is declared at the beginning of db.c. When I used gdb it game me a memory adress. How would I go about trying to debug this and could anyone explain to me how this works. I've been working and mind isn't really working all that well.

Nick Gammon Australia Forum Administrator Thu 21 Aug 2003 08:18 AM #4

My guess, which is not looking at the code right now, is that fpArea is a File Pointer (fp) to the Area file - ie. so if the bug report happens when loading the area file it can report which line the problem is in.

Greven Canada Thu 21 Aug 2003 06:54 PM #5

Are you saying that when I'm loading the area, its set fpArea to itself, and then when further calls are being made tot he bug function, it is still pointing to the area file? If so, why would that cause a crash? It seem that fpArea is a valid pointer, all be it to the wrong file. Also, the bug is not happening when the area is being loaded, its any call to the nug function after the area has been loaded that is the cause of the crashes. I really don't understand what ftell() does, either, which doesn't help.

Is there perhaps a way to print the name of fpArea? I tried something like this:


		sprintf(buf, "Error in file: %s", strArea);
		log_string( buf );

which I just copied from a little below that, but in my log file its prints out "Error in file: $". I really don't understand how this works or what I could do to try and debug this. I've stopped the crashes by putting a return at the top of the function, but that not any kind of a fix.

I really appreciate all the help. Thanks.

Greven Canada Fri 22 Aug 2003 07:44 AM #6

THANK YOU! heh, giant help, and it fixed the problem right up. I do, of course, still have issues, maybe someone can suggest something. I grabbed fdcheck from Bobarak's string about file descriptor 0, and I seem to have lots of unused descriptors, which increase with a copyover. He notes that he fixed the issue with extra descriptors, but not how. If you remember, how did you do it? Also, can anyone suggest a good way to try and track down memory leaks? Thanks.

Boborak USA Fri 22 Aug 2003 09:10 AM #7

Wow. Good timing. I disappear for a couple months (to Iraq none the less) and then my name pops up ;-)

Anyway, if you're getting a number of extra file descriptors after a copyover, you have files that are being opened (hence a new descriptor is created) but then not being closed. Since copyover doesn't terminate these descriptors (the whole point of copyover really), they are carried over and then duplicated. To clear up this issue, you need to search for all of the instances in which a file is opened with FPOPEN and then verify that file is then closed with FPCLOSE. It's tedious but necessary. However, there is a shortcut to this if you are running a *nix machine. I think I explained this in another post but I couldn't find it in a quick search so here's a recap.

1. determine your mud server's PID by typing 'ps -axf' (sans quotes)

you get something like this:


  470 tty1     S      0:00 /sbin/agetty 38400 tty1 linux
  471 tty2     S      0:00 /sbin/agetty 38400 tty2 linux
  472 tty3     S      0:00 /sbin/agetty 38400 tty3 linux
  473 tty4     S      0:00 /sbin/agetty 38400 tty4 linux
  474 tty5     S      0:00 /sbin/agetty 38400 tty5 linux
  475 tty6     S      0:00 /sbin/agetty 38400 tty6 linux
  605 ?        S      0:00 /bin/bash ./startup
  609 ?        SN     0:31  \_ ../src/swreality 4000

609 is my pid

2. change directory (cd) to /proc/{pid}/fd/
i.e. /proc/609/fd/
note: you must be logged on as the user who started the mud server or as root.

3. type ls -la

you'll get something like this:


lr-x------    1 thresher mud-admi       64 Aug 22 02:07 0 -> /dev/null
l-wx------    1 thresher mud-admi       64 Aug 22 02:07 1 -> /home/swcop/smaug/test/log/1000.log
l-wx------    1 thresher mud-admi       64 Aug 22 02:07 2 -> /home/swcop/smaug/test/log/1000.log
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 3 -> /dev/null
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 4 -> /dev/null
lrwx------    1 thresher mud-admi       64 Aug 22 02:07 5 -> socket:[3857]

Each number (file) is a file descriptor. Ignore the /dev/null descriptors and any sockets. The sockets are 'connections' to the mud. Look for duplicate files listed (except certain log files, like my example) Look at this screen before and AFTER a copyover. They SHOULD be identical or in a best case scenario you'd have less fd's.
If for example you looked at this screen and you saw all of your areas listed, that would mean your code that loads the areas was not closing the files.

Here is a hypothetical situation of duplicated fd's


lr-x------    1 thresher mud-admi       64 Aug 22 02:07 0 -> /dev/null
l-wx------    1 thresher mud-admi       64 Aug 22 02:07 1 -> /home/swcop/smaug/test/log/1000.log
l-wx------    1 thresher mud-admi       64 Aug 22 02:07 2 -> /home/swcop/smaug/test/log/1000.log
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 3 -> /dev/null
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 4 -> /dev/null
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 6 -> /home/swcop/smaug/test/system/bugs.dat
lr-x------    1 thresher mud-admi       64 Aug 22 02:07 7 -> /home/swcop/smaug/test/system/bugs.dat
lrwx------    1 thresher mud-admi       64 Aug 22 02:07 5 -> socket:[3857]

I know the file bugs.dat SHOULD be closed after each use. The simple fact it is still open is a warning to me. But since it's duplicated I really know something's wrong. I can then surf my "bugs" code for all instances in which I open bugs.dat and ensure I am taking proper measures to close it.

I hope this helps and I didn't confuse you. Any questions be sure to ask.

-Bobo

Greven Canada Fri 22 Aug 2003 09:30 AM #8

Hello

Well, just before I read your not I shut down the mud to clear the extra file descriptors, so I didn't have many. I followed your instructions, and at the time the only files I had were 2 instances of my log file, but each time I did a copyover, I gained a socket connection. I have 8 sockets listed, but only 2 people connected at the time, me and someone testing something on my coding port. This doesn't seem right to me. I dunno. Thank for the help, that worked nicely. Also, I hope you don't mind that I copied fdcheck for the purposes of this debugging. Do you mind if I keep it around? If you do, that not a problem. You've been a great help.

Boborak USA Fri 22 Aug 2003 09:43 AM #9

First off that code snippet was provided on the SML archives by someone else who was providing it to help others look for fd issues. I copied it here to spread the love ;-)

Additional socket connections is an issue I haven't had a problem with and can't give you a tried and true method at tracking the problem down. However, I'll browse my code a bit and see what I can come up with. In the meantime what 'copyover' code are you using and have you made any major additions to the code that may deal with the sockets?

Boborak USA Fri 22 Aug 2003 01:29 PM #10

Assuming you haven't added any IMC code or anything else that may play with sockets there's only two places I can think of off hand that you should look.

First look in comm.c for the section of code that sets up your mud's port. This is an important piece of code regarding copyover. It should look something like this:


    /*
    * Run the game.
    */
    log_string("Booting Database");
    boot_db(fCopyOver);
    log_string("Initializing socket");
    if (!fCopyOver)
       control = init_socket (port);

If you're missing the "if (!fCopyOver)", then obviously that's part of your problem. A new port should NOT be initalized on copyover. Considering the rest of the copyover code is in place that should fix your problem.

Now if you DO have that if check, then obviously there's something else not kosher ;-) You'll have to give me more specifics about your particular setup.

-Bobo

Greven Canada Fri 22 Aug 2003 03:23 PM #11

Well, here is what it looks like:


    /*
     * Run the game.
     */

    log_string("Booting Database");
    boot_db(fCopyOver);
    log_string("Initializing socket");
    if (!fCopyOver) /* We have already the port if copyover'ed */
     {
        control = init_socket (port);
        control2 = init_socket( port+1 );
        conclient= init_socket( port+10);
        conjava  = init_socket( port+20); 
     }
    sprintf( log_buf, "Star Wars Reality ready on port %d.", port );                                                                                                                          
    log_string( log_buf );
    game_loop( );
    /*
     * That's all, folks.
     */
    log_string( "Normal termination of game." );
    exit( 0 );
    return 0;

I haven't touched anything in the way of connection sockets, in fact, the only thing I've done in comm.c was add samson's
customizable color code and change the way character creation looks. Otherwise its stock SWR 1.0 . Hope that
makes some sense.

Sorry, I'm using "Swr-Copyover-7" that can be found on
kyndig in the SWR area, its a port of "Erwin's Copyover-7
with some modifications to work on a swr based mud".

Boborak USA Sun 24 Aug 2003 09:26 AM #12

Which ports do you actually use for your mud? I have a feeling your mud is set up to run with one port, but the copyover snippet you installed is for a multi-port setup. Stock SWR1.0 is a single-port setup.

-Bobo

Greven Canada Sun 24 Aug 2003 02:45 PM #13

My main port is 4848, but I can connect to it through 4848, 4849, 4858, and 4868. I tested it through my client to make sure, and it does work that way. I haven't touched any part of that, and I have no idea why it would initialize on 4 seperate ports anyways. Thanks for the help.

Boborak USA Sun 24 Aug 2003 02:58 PM #14

Most multi-port smaug setups I have seen, are pretty useless, but aren't exactly a bad thing either. "Extra" sockets are not really a huge resource hog so they don't really hurt much. But if you want to clear up that "extra" socket issue on copyovers, try killing the excess ports to see if that helps.

Find this code:


if (!fCopyOver) /* We have already the port if copyover'ed */
     {
        control = init_socket (port);
        control2 = init_socket( port+1 );
        conclient= init_socket( port+10);
        conjava  = init_socket( port+20); 
     }

Comment out the last three lines, do a hard reboot and then do a copyover and see if you get that extra socket. Hope that helps.

-Bobo

Greven Canada Mon 25 Aug 2003 04:58 AM #15

Hello

I commented out the lines, and I start with three less, but I am still getting an extra one with the copyovers. I'm considering maybe getting a different copyover code, because I would like to solve this problem, but I really don't have that much time anymore, and this might just be quicker. What function starts initializing the sockets? If I need to, I might gdb it, and set a break on the function and check where its being called from.

Thanks,
Greven

Boborak USA Tue 26 Aug 2003 02:35 PM #16

Try setting a break for init_socket it's an OS level function but you can at least do a backtrace for each break to find where the code is being called.

Samson USA Wed 27 Aug 2003 03:39 PM #17

I couldn't tell if you guys had resolved the fpArea problem yet or not, but just in case you haven't, the solution to that is fairly simple.

Open up db.c, and wherever you see:


fclose( fpArea );

Add:


fpArea = NULL;

This has always been necessary, but since Smaug came about way back in the day and was originally written on Redhat 5/6 it was never an issue back then. Not NULL'ing the file pointer didn't cause issues. When Redhat updated the Glibc libraries somewhere along the way, this became a huge problem and was discussed at length on the SML. Most Smaug derivs have now corrected for this with the above simple fix. Evidently SWR missed the boat on that one. :)

Greven Canada Thu 28 Aug 2003 03:53 PM #18

Yes, I've nulled the fpArea's, but it seem that I've come along to another problem with bug:


		while ( getc( fpArea ) != '\n' )

I've been away for a couple of days, and my shell account ran out of space due to a massive log file that had alot of wierd stuff spit into it. This caused some of my player's pfiles to become corrupted, so when they load, boom. One had a problem with this .home file, but got that under control, it just crashed. However, another of my players pfile itself got corrupted, and when she would log on, it would freeze, and my host is getting pissed. I commented out bug and loadedup her file and made her save to correct her pfile, but I still have this problem. The above line was frame 6, I got it from gdb it and making it freeze when I did a loadup on here character. Why would this suddenly start to act up now? The server is using the same version of linux that it has since I started with them. I dunno, any help would be great. Also, I gdb init_socket, and it would only come up when I booted, not when I did a copyover, but the extra description would still be there. When I'm setting the break point though, its not grabbing a system function, its grabbing the one in comm.c. Are they much different? I assume they are

Boborak USA Thu 28 Aug 2003 05:16 PM #19

Eww. Pfile corruption is no fun, but unfortunately it is likely because of something you may have added. You may have to delete those pfiles to solve the problem or worst yet, the problem may actually be a code issue. Unfortunately it's hard to give guidance without some examples of the bugs you're getting. Posting a single line of code without telling what function or file it's from doesn't help alot ;-) The good news is I know where that particular line comes from and what it does. Bad news is, it really doesn't explain your pfile problem. Need error messages! ;-) Post some of your log here and we'll see what we can do with it.

-Bobo

Greven Canada Thu 28 Aug 2003 05:56 PM #20

The problem with the pfiles was that they were only half written due to lack of server space, and would end in something like:


#OBJECT
Vnum         10317
Values       20 0 0 0 0 0
End

#OBJECT
N

The pfile stuff I can fix, I commented out bug(), did a loadup, force them to save, and quit, works nicely.

Problem is that its not logging anything to the log file because its the bug function thats screwed up. That line of code comes from bug(), and the most that my log files have is this:


Wed Aug 27 22:05:17 2003 :: Sock.sinaddr:  198.53.93.106, port 62660.
Wed Aug 27 22:05:18 2003 :: Preloading player data for: Katrina (4K)
Wed Aug 27 22:05:20 2003 :: Loading player data for: Katrina (4K)

And then it stops cause I have to kill it from my shell. If I run it inside gdb, and I cause the freeze when I loadup a bad pfile while bug is not commented out, I pause the game from within gdb and backtrace to get this:


#0  0x2ab681bd in ?? ()
#1  0x2ab680d8 in ?? ()
#2  0x2ab6555e in ?? ()
#3  0x80dcc7e in bug (str=0x81fa560 "fread_word: EOF encountered on read.\n\r") at db.c:3867
#4  0x80dbd64 in fread_word (fp=0x8678b00) at db.c:3335
#5  0x8135973 in fread_obj (ch=0x8677388, fp=0x8678b00, os_type=0) at save.c:1857
#6  0x813161b in load_char_obj (d=0x8675e30, name=0x7fffcfd8 "Katrina", preload=0 '\000') at save.c:956
#7  0x808fe6e in do_loadup (ch=0x8672640, argument=0x7ffff587 "katrina") at act_wiz.c:4406
#8  0x80fa46d in interpret (ch=0x8672640, argument=0x7ffff587 "katrina") at interp.c:354
#9  0x80c645b in game_loop () at comm.c:624
#10 0x80c55b7 in main (argc=2, argv=0x7ffffa04) at comm.c:255
#11 0x2ab2baa7 in ?? ()

Hope that helps, sorry about the vagueness.

Nick Gammon Australia Forum Administrator Thu 28 Aug 2003 08:59 PM #21

Put a breakpoint in bug and work out why it is crashing. For instance, if fpArea null or not?

Greven Canada Thu 28 Aug 2003 10:13 PM #22

Its not crashing, its freezing, but no, fpArea is not null. Whether its valid or not, I don't know, but it is returning a pointer. The only places that fpArea is called from are db.c and save.c, and both are nulling fpArea.

Nick Gammon Australia Forum Administrator Fri 29 Aug 2003 12:51 AM #23

Well, they can't be if it isn't null when the bug is called, unless something else is corrupting it.

OK, put a breakpoint on the lines which you added which sets fpArea to null, and make sure *they* are called.

Greven Canada Fri 29 Aug 2003 12:57 AM #24

Well, I looked into the code, and load_char_obj actually uses the playerfile and set fpArea as such:


	/* Cheat so that bug will show line #'s -- Altrag */
	fpArea = fp;
	strcpy(strArea, strsave);

So that explains why fpArea is not null when encountering an error while loading a character, but doesn't help me in the slightest in trying to debug it, as I don't understand what the getc() function is doing or why.