Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to "verify" your details, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.
 Entire forum ➜ MUSHclient ➜ Plugins ➜ Learning Mapper Database getting too big

Learning Mapper Database getting too big

It is now over 60 days since the last post. This thread is closed.     Refresh page


Posted by Faedara   (106 posts)  Bio
Date Fri 28 Oct 2022 05:52 AM (UTC)
Message
I'm having trouble with the learning mapper causing bloat, plugin failing to load, and subsequently crashing the plugin and deleting the entire database. The MUD I'm playing on (AwakeMUD) is rather large, admittedly, and room descriptions can be very descriptive, leading to 2k+ rooms added to the database.

Is there some method I can use to avert this crash, and subsequent loss of the map? I've been combing the forums for weeks and not found a solution.

The eternally clueless <3
Currently looking for a mentor/tutor to help me learn everything about Lua from the ground up. I can't understand 28*41 until I learn what causes 2+2.
Top

Posted by Nick Gammon   Australia  (23,070 posts)  Bio   Forum Administrator
Date Reply #1 on Sat 29 Oct 2022 06:41 AM (UTC)
Message
You shouldn't lose everything. At the very least I would suggest making a backup of the mapper data (the state file) which you will find in the plugins "state" directory.

Its name will have "99c74b2685e425d3b6ed6a7d" in it.

For example, under Wine, the directory is:


~/.wine/drive_c/Program Files (x86)/MUSHclient/worlds/plugins/state


My state file is:


04cbe372cc03045372494c73-99c74b2685e425d3b6ed6a7d-state.xml
^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^
  My world ID               The plugin ID


Yours will have a different hex number at the start, which is the unique ID of your world file.

If you copy this somewhere else then you have saved the "good" file from your last run. Of course, don't copy it if the file seems corrupted. Preferably make a script to copy each time under a different name with the date/time as part of the name. Then you can always revert back to a good copy of it.

If things go wrong, put the copy back under its original name.

For a longer term solution I'll look at storing the mapper database as an SQLite3 database rather than into a plugin state file. That should be much more robust. However that may take a few days to write and debug, so as a temporary solution I would be making backups.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Fiendish   USA  (2,533 posts)  Bio   Global Moderator
Date Reply #2 on Sat 29 Oct 2022 01:48 PM (UTC)

Amended on Sat 29 Oct 2022 01:51 PM (UTC) by Fiendish

Message
Side note: The state file maximum size limit isn't very large (looks like 5mb? https://github.com/nickgammon/mushclient/blob/6b79a6d88ea811f797b143623e839eed07fa9802/xml/xmlparse.h#L8). I had to switch from storing intermediate data in the state file to storing it in a separate file for one of my plugins because people were experiencing crashes during reload of my chat capture plugin when they had a large amount of text captured. https://github.com/fiendish/aardwolfclientpackage/commit/e2d79fcb3d064904619d86781a2b664b5d069044

https://github.com/fiendish/aardwolfclientpackage
Top

Posted by Nick Gammon   Australia  (23,070 posts)  Bio   Forum Administrator
Date Reply #3 on Sat 29 Oct 2022 08:43 PM (UTC)
Message
Maybe we should make it larger as an interim solution?

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Fiendish   USA  (2,533 posts)  Bio   Global Moderator
Date Reply #4 on Sat 29 Oct 2022 11:20 PM (UTC)
Message
We could. What's a good size though in 2022? 100? 200?

https://github.com/fiendish/aardwolfclientpackage
Top

Posted by Fiendish   USA  (2,533 posts)  Bio   Global Moderator
Date Reply #5 on Sat 29 Oct 2022 11:24 PM (UTC)

Amended on Sun 30 Oct 2022 12:41 AM (UTC) by Fiendish

Message
Ok, I increased it in https://github.com/nickgammon/mushclient/pull/75
But it seems so wasteful. The plugin should probably just be updated to write to plain files instead of the state file.

https://github.com/fiendish/aardwolfclientpackage
Top

Posted by Fiendish   USA  (2,533 posts)  Bio   Global Moderator
Date Reply #6 on Sun 30 Oct 2022 12:48 AM (UTC)
Message
Anyway, Faedara, as an interim stopgap solution until Nick updates how the plugin saves its data, this MUSHclient executable should work around the issue at the expense of a bit more RAM used: https://github.com/nickgammon/mushclient/releases/download/latest_commit/MUSHclient.exe

Just replace your current exe with it.

https://github.com/fiendish/aardwolfclientpackage
Top

Posted by Faedara   (106 posts)  Bio
Date Reply #7 on Sun 30 Oct 2022 04:42 AM (UTC)
Message
Wow, ya'll are an incredible team. I was expecting something along the lines of, "You forgot to update how much allocation is applied by this setting", not "I updated the client within a day to solve your problem, and we're rewriting the plugin to prevent similar future problems."

Absolutely astounding~ <3 Thank you.

The eternally clueless <3
Currently looking for a mentor/tutor to help me learn everything about Lua from the ground up. I can't understand 28*41 until I learn what causes 2+2.
Top

Posted by Nick Gammon   Australia  (23,070 posts)  Bio   Forum Administrator
Date Reply #8 on Sun 30 Oct 2022 06:00 AM (UTC)
Message
Fiendish's change is to allow slightly more than 20 times as much when reading an XML file. 100 MB to be precise.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,070 posts)  Bio   Forum Administrator
Date Reply #9 on Sun 30 Oct 2022 06:02 AM (UTC)
Message
Fiendish said:

But it seems so wasteful. The plugin should probably just be updated to write to plain files instead of the state file.


I want to do it with a minimal amount of extra work changing the plugin (because I am lazy). Perhaps serializing the plugin data (to a disk file) rather than putting it into a state file would be the simplest solution.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,070 posts)  Bio   Forum Administrator
Date Reply #10 on Sun 30 Oct 2022 06:53 AM (UTC)
Message
I don't think it is particularly wasteful. That line that you changed is just a sanity check to not load too large files. The memory isn't actually allocated until the file is read in.

And if it is serialized a different way it will still take the same amount of memory. Unless it is rewritten to use the SQL database, which would be quite a lot of work.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


12,678 views.

It is now over 60 days since the last post. This thread is closed.     Refresh page

Go to topic:           Search the forum


[Go to top] top

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.