Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to verify your details, confirm your email, resolve issues, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.

Due to spam on this forum, all posts now need moderator approval.

 Entire forum ➜ MUSHclient ➜ Bug reports ➜ Lua script gobbling memory
Lua script gobbling memory

It is now over 60 days since the last post. This thread is closed.     Refresh page

Posted by Robin Lionheart   (14 posts)  Bio
Date Sun 15 Oct 2006 06:57 AM (UTC)

Amended on Sun 15 Oct 2006 09:21 AM (UTC) by Robin Lionheart

Message
In MUSHclient 3.80 and 3.81 (at least), my Lua script rapidly [strike]leaks[/strike]consumes mass quantities of memory at about 300K/second. On one occasion, after a couple hours, MUSHclient crashed with a shower of tiny "Out of memory" dialog boxes, and Task Manager reported it was eating 1.2GB of RAM!

[Ed: Rewrote to make clear that my script is to blame]
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #1 on Sun 15 Oct 2006 07:10 AM (UTC)
Message
It depends a bit on what the scripts do.

Lua uses memory allocation for things like strings, tables, functions and so on.

I haven't had a problem myself, but if you made a script that made millions of something it would probably eventually run out of memory.

Are you suggesting this is a new problem in version 3.80? That is, it didn't happen in version 3.79?

Can you post the problem script, or confirm that the script does not create many strings/tables etc?
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top
Posted by Robin Lionheart   (14 posts)  Bio
Date Reply #2 on Sun 15 Oct 2006 08:09 AM (UTC)

Amended on Sun 15 Oct 2006 02:34 PM (UTC) by Robin Lionheart

Message
It does string concatenation operations [Ed: pretty much whenever I call Send(), including] assembling large strings [Ed: (about 4KB)] from the contents of tables.

I've noticed that when I minimize the program with Alt-Space Minimize or the minimize button in the upper right corner of the window, the memory usage in Task Manager shot down from 500,000K to about 10,000K. It seems this gets MUSHclient to garbage collect its memory space.

So I tried using 4NT to do 
activate "MUSHclient*" min
and 
activate "MUSHclient*" restore
, but it apparently doesn't garbage collect when it minimizes in response to IDE messages.
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #3 on Sun 15 Oct 2006 08:21 AM (UTC)
Message
The gradual decrease in memory is more likely to be that the operating system is swapping memory of inactive applications out to the page file on disk.

If you want to force a garbage collection simply do:


collectgarbage ("collect")


I suggest you read this technical note by one of the authors of Lua:

http://www.lua.org/notes/ltn009.html

He describes how doing string concatenation can be quite expensive, and a solution that is much more efficient.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #4 on Sun 15 Oct 2006 08:26 AM (UTC)
Message
Quote:

It does string concatenation operations assembling large strings ...


I would like to clarify that this isn't really an established "bug" at this stage.

If you concatenate large strings, then it is likely to "consume memory". This is a different thing to "leak memory". The word "leak" implies there is something wrong.

The recently-implemented spell-checker stores around 70,000 words in Lua tables, and I haven't had a problem with "out of memory" errors.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top
Posted by Robin Lionheart   (14 posts)  Bio
Date Reply #5 on Sun 15 Oct 2006 08:34 AM (UTC)

Amended on Sun 15 Oct 2006 10:42 AM (UTC) by Robin Lionheart

Message
Sorry to misattribute the problem to a memory leak.

I believe the primary culprit was a status line parser (which I added to my script after I upgraded from 3.79).

Changing 
local status_line = hp .. " " .. mana .. " " .. move .. " " .. time .. " " .. append_to_status
SetStatus(status_line)

to
SetStatus(hp, " ", mana, " ", move, " ", time, " ", append_to_status)

apparently caused MUSHclient to stop hemorrhaging memory.

Now it consumes memory more slowly, so I'm going to have to tighten up my script. I constantly used 
Send(string1 .. string2 .. string3)
where I could have used 
Send(string1, string2, string3)
to avoid filling the heap with waste strings.

However, I do wonder why garbage collector allowed the memory usage to get up as high as 1.2GB over a couple hours; I recently let it get up to 200KB and minimized the program to flush it down to about 10KB again.

[Edited to combine 4 successive posts into 1 and correct my misstatements above]
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #6 on Sun 15 Oct 2006 10:39 AM (UTC)
Message
It sounds strange. You haven't changed the garbage collection thresholds have you?

Your new method would certainly reduce the number of strings Lua would have to create, so that is more efficient, but I would have expected its garbage collector to cope.

I might see if I can reproduce the problem tomorrow. It sounds like a simple script like your SetStatus line, run in a loop, should show whether or not there is a major problem.

What operating system are you running on? (XP or something else?).

Can you give an estimate of the number of times a minute you would update the status line like that? Once a second? Or much more often?

Also, how many lines are you allocating to the output window? Over time, a lengthy output buffer will necessarily use quite a bit of memory.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top
Posted by Robin Lionheart   (14 posts)  Bio
Date Reply #7 on Sun 15 Oct 2006 10:50 AM (UTC)

Amended on Sun 15 Oct 2006 12:24 PM (UTC) by Robin Lionheart

Message
I didn't know I could change the garbage collection thresholds. The number of lines in my output buffer is 2500.

I'm using speedwalks and other macros which cause dozens of status/prompts to come in all at once, to be caught by a regex trigger breaking each into pieces and passing them to a script routine (without sending it to output) that then sends them to the status line. I figure the most it could be being called is about 100 times a minute, multiplied by the number of tabs I have open using this script.

I'm using Windows XP x64 Edition with an AMD Athlon 64 X2 multiprocessor CPU.

I've moved most of the code to the trigger to let trigger substitutions do the work of string concatenations:

<triggers>
  <trigger
   custom_colour="5"
   enabled="y"
   group="prompt"
   keep_evaluating="y"
   match="^&lt;(?P&lt;H&gt;\d+/\d+) (?P&lt;V&gt;\d+/\d+) (?P&lt;M&gt;\d+/\d+)(?: (?P&lt;X&gt;\d+/?\d*))?(?: (?P&lt;T&gt;\d+[ap]m))?&gt;$"
   name="StatusPrompt"
   omit_from_log="y"
   omit_from_output="y"
   regexp="y"
   send_to="12"
   sequence="1"
   variable="game_status"
  >
  <send>game_status = "H:%&lt;H&gt; V:%&lt;V&gt; M:%&lt;M&gt; %&lt;X&gt; %&lt;T&gt;"
update_status_line()</send>
  </trigger>
</triggers>


with the following in the script file:

game_status = ""
append_to_status = ""

function update_status_line()
	SetStatus(game_status, " ", append_to_status)
end


Doing this and repeatedly sending a flood of 500 say commands now causes my script to eat from 12K-24K/sec (and the scrollback should be full long since).

Typing 
/collectgarbage("collect")
after doing so appears to have no effect on the memory usage displayed in Windows Task Manager.
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #8 on Sun 15 Oct 2006 08:43 PM (UTC)
Message
Quote:

I didn't know I could change the garbage collection thresholds.


I would advise you not to, the Lua manual suggests you can make things worse, I was just checking.

Thanks for your detailed response. You clearly have a problem, I am trying to narrow down what it might be.

I don't think it is the SetStatus part itself, nor the Lua script engine, per se. I ran this test in the Immediate (scripting) window on version 3.81, using Windows XP:


print ("memory usage before = ", collectgarbage ("count"))
hp = 4200
mana = 2200
move = 53
time = 1234
append_to_status = "the quick brown fox"

t1 = os.time ()
for i = 1, 500000 do
  local status_line = hp .. " " .. mana .. " " .. move .. " " .. time .. " " .. append_to_status
  SetStatus(status_line)
end -- loop
t2 = os.time ()
print ("time taken = ", t2-t1)

print ("memory usage after = ", collectgarbage ("count"))


This does 500,000 lots of the string concatenation you originally had (with the .. syntax) and then a SetStatus. I made up some figures, I presume they are similar to yours.

Looking at the XP task manager, these were my memory usage figures:


  • Before opening the world = 7,100 Kb
  • Before doing the SetStatus test = 8,556 Kb
  • After doing the SetStatus = 9,184 Kb


First, I am confused about your reported figures, where you said you got the usage down to 10 Kb. I think you must mean 10 Mb. (1 Mb = 1000 Kb).

Even with no world open my copy uses 7,100 Kb (that is, over 7 Mb).

Opening a world seems to take up about 1.5 Mb, which seems OK to me. However after doing 500,000 SetStatus and Lua string concatenations, the memory only rises aby 0.5 Mb.

Plus, I checked the memory usage inside Lua by doing this:

/collectgarbage ("count")

My results were:


memory usage before = 83.6337890625
time taken = 8
memory usage after = 83.849609375


Thus the memory that Lua thinks it has used is only 0.2 Kb - hardly anything.

I think your memory consumption must be elsewhere, although I must admit I am puzzled as to why the operating system reports 500 Kb of usage, where Lua reports less than 1 Kb.

While I investigate this further, it might help if you can make a test script that reproduces the problem. Along the lines of what I have done above with SetStatus in a loop, you might try pushing a lot of data through one of your other script routines to see if you can find what is taking all the memory.

You mentioned the use of tables, it would be possible to consume a lot of memory if you did something like this:


t = {} --> make a table once

table.insert (t, "something") -- for each line from the MUD


If you were doing something like this then the table t would gradually get larger, and not be garbage-collected.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top
Posted by Nick Gammon   Australia  (23,173 posts)  Bio   Forum Administrator
Date Reply #9 on Sun 15 Oct 2006 10:50 PM (UTC)
Message
I have rerun the above test, and this time the Windows memory does not seem to increase at all (or by more than 1K or so) when running the above script. It is hard to know what is going on, when tests run at different times give totally different results.
- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

28,683 views.

It is now over 60 days since the last post. This thread is closed.     Refresh page

Go to topic:           Search the forum

[Go to top] top

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

Home