Problem about auto-wrap in non-English MUD

Posted by Zhenzh on Tue 20 Mar 2018 06:29 AM — 18 posts, 64,743 views.

Zhenzh China Tue 20 Mar 2018 06:29 AM #0

I'm playing a Chinese MUD in which both Chinese and English may exist in one line.

Each Chinese word occupies 2 characters while English word occupies only 1 character.

Which means I can not predict the exact position of a separator when the length of a line exceeds the specified wrap width as fixed position may divide a Chinese word into two half.

I have an idea in my mind to resolve such problem:
1. Capturing all lines from MUD server.
2. Directly print lines within wrap width.
3. For lines out of limit, using string.byte() to count all characters and get the position of the separator avoiding dividing Chinese word.
4. Loop print separated lines

It should be able to resolve the auto-wrap problem. But I'm worrying about the performance of such method as it may cost huge of additional computing resource and may impact on the speed of the whole MUD.

Does anyone has similar experience and any better method to resolving the problem?

Nick Gammon Australia Forum Administrator Tue 20 Mar 2018 06:43 AM #1

What do you mean by a separator? The program currently wraps at spaces, so it shouldn't split a word into two.

Nick Gammon Australia Forum Administrator Tue 20 Mar 2018 06:44 AM #2

Do you mean that it wraps too early?

Zhenzh China Tue 20 Mar 2018 07:51 AM #3

Yes, it may wrap too early so that Chinese word will be divided into two half.

For example, the auto-wrap width is set to 100 characters. The 101st character will be printed in a new line. While the 100th word is a Chinese word which occupies 2 characters(100th and 101st). Auto-wrap at the end of 100th character may divide the Chinese.

Nick Gammon said:

The program currently wraps at spaces

Different from English, Chinese text don't use space separating two words. If a line doesn't contain any space character, what will happen?

Zhenzh China Thu 22 Mar 2018 05:37 AM #4

I guess wrap will separate the specified column when no space characters in the line.

It do be a problem for all non-alphabet languages as space will not be the separator between two words.

Nick Gammon Australia Forum Administrator Thu 22 Mar 2018 06:41 AM #5

This plugin may help:

To save and install the Split_long_Big5_lines plugin do this:

Copy the code below (in the code box) to the Clipboard
Open a text editor (such as Notepad) and paste the plugin code into it
Save to disk on your PC, preferably in your plugins directory, as Split_long_Big5_lines.xml
- The "plugins" directory is usually under the "worlds" directory inside where you installed MUSHclient.
Go to the MUSHclient File menu -> Plugins
Click "Add"
Choose the file Split_long_Big5_lines.xml (which you just saved in step 3) as a plugin
Click "Close"
Save your world file, so that the plugin loads next time you open it.


<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE muclient>

<muclient>
<plugin
   name="Split_long_Big5_lines"
   author="Nick Gammon"
   id="19e492fbb1ac8429ad1f9b1b"
   language="Lua"
   purpose="Splits long Big5 lines"
   date_written="2018-03-22 17:11:23"
   requires="4.97"
   version="1.0"
   >
<description trim="y">
<![CDATA[
Breaks Big5 chinese lines to not be inbetween characters.
]]>
</description>

</plugin>


<!--  Script  -->


<script>
<![CDATA[

local LINE_LIMIT = 40

local big5 = "[\129-\254][\064-\126\161-\254]"
-- handle incoming packet
function OnPluginPacketReceived (sText)
  return (string.gsub (sText, "(" .. string.rep (big5, LINE_LIMIT) .. ")", "%1 "))
end -- function
]]>
</script>


</muclient>

What that does is look for 40 lots of "Big5" characters (0x81 to 0xFE then 0x40 to 0x7E or 0xA1 to 0xFE) and inserts a space after them.

I'm not sure how well that will work for you, if you have non-Chinese characters in the middle then the count will be out. However by inserting a space that should let MUSHclient wrap at the end of a character and not in the middle of it.

Change the "40" in the plugin to be some other number if you want to tweak the splitting point.

Nick Gammon Australia Forum Administrator Thu 22 Mar 2018 10:14 AM #6

It might be possible to make the client handle this better, if the wrapping algorithm was improved. At present it wraps at the closest space to the end of the line (when the limit is reached). Conceivably, if no space was found, it could backtrack to the most recent double-byte character.

Would I be correct in thinking:

The lines in question have no spaces at all.
You are using Big5 or GB2132 encoding?
You are not using Unicode? (see the UTF-8 checkbox in the output window configuration)

Nick Gammon Australia Forum Administrator Thu 22 Mar 2018 10:00 PM #7

I think I may have fixed this problem. Don't use the plugin, however download the latest pre-release version of MUSHclient as described here:

http://www.gammon.com.au/forum/?id=13903

Replace your MUSHclient.exe file with the newer one. This causes a line-wrap to occur at the start of a Big5 or GB2132 character rather than in the middle of it.

Zhenzh China Fri 23 Mar 2018 04:40 AM #8

So cool. I have tried the per-release. It do works well for my GB2132 characters.

One comment, seems the space character has higher priority to separate a line which may get an English/non-English mixed line wrap earlier.

I recommend get space and double-byte character using same priority to get a mixed line wrapped at the closest column of line limit in big5/gb2312 environment.

Nick Gammon Australia Forum Administrator Fri 23 Mar 2018 04:52 AM #9

Is this a real problem? You say it "may" happen. It is happening enough to be annoying?

Zhenzh China Fri 23 Mar 2018 05:15 AM #10

Yes, it happened in my environment.
Not a big problem. At least, avoiding separate double-byte characters is much more better then before.

Zhenzh China Fri 23 Mar 2018 05:34 AM #11

Suppose I have an mixed line:

Quote:

English1 English2 English3 Chinese1Chinese2Chinese3

The line limit is specified at the column of "Chinese2".
Note: EnglishX stands for an English word, ChineseX for a Chinese word

The expected result will be:

Quote:

English1 English2 English3 Chinese1Chinese2
Chinese3

The current real result is:

Quote:

English1 English2 English3
Chinese1Chinese2Chinese3

Nick Gammon Australia Forum Administrator Sat 24 Mar 2018 12:33 AM #12

OK, reworked the line splitting somewhat so it should split at the last Big5 character or a space, whichever comes last.

Grab the latest pre-release version as described earlier.

Zhenzh China Sat 24 Mar 2018 06:20 PM #13

The new version works very well. Thank you for your support.

Nick Gammon Australia Forum Administrator Sun 25 Mar 2018 05:59 AM #14

Just out of curiosity, did you (or anyone) every translate the menus and dialog boxes into Chinese? In particular see this thread:

http://gammon.com.au/forum/?id=7953

Zhenzh China Sun 25 Mar 2018 08:03 AM #15

Yes, someone has translated mush to simplified Chinese. But it's based on an older version 4.x.

I have updated the ch.lua to 5.05 and I'm now working on updating ch.dll to 5.05.

BTW, there are two .dll files under locale directory of 5.05, en and en_small. Which one should I use as the base of translation?

Nick Gammon Australia Forum Administrator Sun 25 Mar 2018 08:05 PM #16

The small one is for people with very small monitors (eg. old laptops) so for most users I would use the ch.dll file.

I have deliberately not changed the resources file since releasing it as a separate file so it should work fine with the latest version of MUSHclient. If not, there should only be minor changes required.

Zhenzh China Mon 26 Mar 2018 06:05 AM #17

The translated ch.dll for older version do works with the latest version and I'm using it as a temporary solution.

But the size of the older ch.dll is 451k while the en.dll in 5.05 is 1.4M. I doubt whether the older .dll can fully satisfy the new version.

Comparing and updating .dll files do be difficult. Why not using a dictionary file like lua table to afford language translation and get the .dll reading dictionary? Text based files will be much more easier for comparing and upgrading with mush version.