I'm using a plugin to modify packets, and basically I'm trying to pull information from telnet subnegotiations (\255\250\200...\255\240) and then cut them out of the packet. Once in a while (usually when the packets are coming very fast), the server will split a subnegotiation into two separate packets so I get a packet like [packet text]\255\250\200[portion of subnegotiation] followed immediately by a second packet: [rest of subnegotiation]\255\240[packet text]. Here's what I've coded to work around this with the relevant parts bolded.
function OnPluginPacketReceived(x)
Note("packet: " .. x)
x = string.gsub(x, "\255\251.", "")
if packet and not blackout then
subnegotiation(packet .. tostring(string.match(string.gsub(x, "\255\250\200.-\255\240", ""), "^.-\255\240")))
end
packet = ""
blackout = false
if string.find(x, "\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249") then
blackout = true
c = {}
for a in string.gmatch(x, "\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249") do
c[table.getn(c) + 1] = a
end
end
if not blackout then while string.find(x, "\255\250\200.-\255\240") do
subnegotiation(string.match(x, "\255\250\200.-\255\240")) x = string.gsub(x, "\255\250\200.-\255\240", "", 1) end
packet = string.match(x, "\255\250\200.-$") x = string.gsub(x, "^.*\255\240", "")
x = string.gsub(x, "\255\250\200.*$", "")
elseif blackout then
b = x
while string.find(b, "\255\250\200.-\255\240") do
subnegotiation(string.match(b, "\255\250\200.-\255\240"))
b = string.gsub(b, "\255\250\200.-\255\240", "", 1)
end
x = string.gsub(x, "\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249", "\201")
x = string.gsub(x, "\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240","")
for _,d in ipairs(c) do
x = string.gsub(x, "\201", d, 1)
end
end
if GetEntity("timestamp") == '1' then
x = string.gsub(x, "\255\249", " {" .. os.clock() - math.floor(os.clock() / 100) * 100 .. "}\255\249", 1)
end
return x
end
The problem is that sometimes a strange newline will appear in the output like so:
(The packets are visible because of the first line of my function: Note("packet: " .. x); also, in case it isn't clear, ÿúÈ = \255\250\200 and ÿð = \255\240)
See how the last prompt in the first packet was split up? Somehow, either the newline after "Char.Vitals" wasn't taken out with the rest of the subnegotiation (which definitely was taken out because they're normally visible in the output) or a new newline was mysteriously inserted. Can anyone see how this happens?
I almost forgot: I set the option so that Mushclient converts ÿù to a newline, which is why the prompts are on separate lines in the output but not in the packet, in case anyone was wondering.
Without fully analyzing what seems to be some rather complex code, I get the impression you are causing it by removing the subnegotiation from the first packet, so that when MUSHclient sees the newline it literally displays it.
I wouldn't do it like you have done. It seems to me that without writing a proper state engine, you are relying upon the subnegotiation sequences to appear intact. How about this? You need some sort of marker that uniquely identifies when you can reliably hand off the packet to MUSHclient, and it seems to me that IAC GA (ÿù) is quite probably it.
Try this sequence:
Accept incoming text (the argument to OnPluginPacketReceived). Append this to any previously received text (ie. a variable).
Look for the sequence (something)IAC GA. That is, you have found enough text to warrant a full prompt to the player. If not found, do nothing (exit returning an empty string).
In other words you are searching for: ^.-\255\249
This lets you handle negotiation sequences over a packet boundary. The part sequence is simply stored.
Once you match on ^.-\255\249 you pass the matching text to your subnegotiation handler. This then scans for, and removes, the extra subnegotiation stuff (\255\250\200...\255\240). You repeatedly do this, building up an output packet, with the subnegotiations removed. Anything over is stored for next time. Then you return the fixed-up packet.
If this doesn't make sense I can do a more detailed description, but it shows the general idea.
In that I have a partial line (partial) to which each new packet is appended. Then it looks for prompts and adds a newline. Conceptually it is a similar thing. I batch up packets until they are long enough to be analyzed.
I do it pretty much how Nick described in his first reply. Here's the main function along with helper ones. Note that there's a subtle bug with how parsed data is stored - if there are more than one ATCP message in the packet than the data in the "parsed" table will be repeatedly overwritten, but that's a simple fix (which I've been too lazy to make).
function parseATCP(packet)
local packet = leftovers .. packet
leftovers = ""
local s,e,prev,msg,nxt,asplit
local atcp, parsed = {}, {}
local new_packet = ""
new_packet = string.gsub(packet, PAT, function(cap)
local ret = get_room(cap, parsed)
if ret then
parsed = ret
else
parsed = get_vitals(cap, parsed)
end
table.insert(atcp, cap)
return ""
end)
s,e,msg = string.find(new_packet, "(\255[^\255]-\255?)$")
if s then
if (#msg == 1) or ( (#msg > 2) and string.find(msg, "^\255\250\200") )
or( (#msg > 1) and string.find(msg, "^\255\250") ) then
leftovers = msg
new_packet = string.sub(new_packet, 1,s-1)
end
end
packet = new_packet
return packet, atcp, parsed
end
function get_room(msg, tab)
local s,e,what,cont = string.find(msg, "^(Room%.%w+) (.-)$")
if s then
tab
Please help us by showing:
A copy of the trigger, alias or timer you were using (see
Copying XML)
The error message, if any, that you got
= cont
return tab
else
return nil
end
end
function get_vitals(msg, tab)
local asplit = utils.split(msg, "\10")
tab[asplit[1]] = asplit[2] or ""
return tab
end
codes = {
IAC_WILL_ATCP = "\255\251\200",
IAC_WONT_ATCP = "\255\252\200",
IAC_DO_ATCP = "\255\253\200",
IAC_DONT_ATCP = "\255\254\200",
IAC_SB_ATCP = "\255\250\200",
IAC_SE = "\255\240",
IAC_DO_EOR = "\255\253\025",
IAC_WILL_EOR = "\255\251\025",
IAC_GA = "\255\249"
}
leftovers = ""
PAT = codes.IAC_SB_ATCP .. "(.-)" .. codes.IAC_SE
The complete plugin is here: http://www.freewebs.com/keldar/atcp.htm
I don't see any reason why this shouldn't remove the incomplete subnegotiation at the end of the packet. Mushclient converts the second line of a subnegotiation to output (I don't know why, but it doesn't really matter). If the above code wasn't in the plugin, I would have seen "
NL:0/100 H:6000/6000 M:6300/6300 E:6300/5700 P:10/10 N:" in my output; the gsub successfully took it out. Yet somehow a newline is persisting, and I can't see how that's possible because the regex takes everything from the \255\250\200 to the end of the packet into account.
I studied Ked's plugin before I made my own, and the only major difference in the way we modify the packet is that Ked truncates it with string.sub and I use string.gsub. Maybe that's some sort of clue.
Also, I did consider doing something like storing up packets until I received a packet that ends with IAC GA. However, I know that IRE Muds (or at least Lusternia) don't add an IAC GA when they prompt you for your password, so there could be other, possibly unforeseen situations where there isn't an IAC GA and my output just goes blank while my plugin waits. Scary.
Mushclient converts the second line of a subnegotiation to output (I don't know why, but it doesn't really matter).
Well I think it matters because if you are trying to just process each packet at a time, you need to flag "incomplete negotiation" (is that the blackout flag?) - so that the next packet effectively adds that incomplete negotiation back to the start, as if it was there all along.
If you can solve why that isn't happening you should solve the whole thing.
The blackout flag is so I can pass the subnegotiation as output during a condition in the MUD called blackout that doesn't allow you to see your prompt. For the purposes of this examination, it can pretty much be disregarded (assume blackout = false).
This is how I handle the split packets:
while string.find(x, "\255\250\200.-\255\240") do
x = string.gsub(x, "\255\250\200.-\255\240", "", 1)
end
That loop removes all complete subnegotiations.
packet = string.match(x, "\255\250\200.-$")
That line stores the incomplete subnegotiation in the "packet" variable. Then look back up near the beginning of the function:
That combines the stored subnegotiation fragment with the fragment at the beginning of the next packet and sends it to the function I use to pull information from subnegotations. Finally, after the complete subnegotiations are removed from the new packet, the fragment at the beginning is removed here:
x = string.gsub(x, "^.*\255\240", "")
However, I don't see why any of this matters. I could disregard the need to piece together the packets, not add the end of the last to the beginning of the next, and this...
x = string.gsub(x, "\255\250\200.*$", "")
...should still cut out the incomplete subnegotiation at the end of the packet, along with any newlines that may be contained in it. If I'm not understanding something please point it out.
OK, I have taken the liberty of laying out your code in a bit more readable way, and adding comments to the end of the "end" statements.
function OnPluginPacketReceived(x)
Note("packet: " .. x)
x = string.gsub(x, "\255\251.", "")
if packet and not blackout then
subnegotiation(packet .. tostring(string.match(string.gsub(x, "\255\250\200.-\255\240", ""), "^.-\255\240")))
end -- if partial packet
packet = ""
blackout = false
if string.find(x,
"\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249") then
blackout = true
c = {}
for a in string.gmatch(x,
"\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249") do
c[table.getn(c) + 1] = a
end -- for loop
end -- if now in blackout
if not blackout then
while string.find(x, "\255\250\200.-\255\240") do
subnegotiation(string.match(x, "\255\250\200.-\255\240"))
x = string.gsub(x, "\255\250\200.-\255\240", "", 1)
end -- while loop
packet = string.match(x, "\255\250\200.-$")
x = string.gsub(x, "^.*\255\240", "")
x = string.gsub(x, "\255\250\200.*$", "")
elseif blackout then
b = x
while string.find(b, "\255\250\200.-\255\240") do
subnegotiation(string.match(b, "\255\250\200.-\255\240"))
b = string.gsub(b, "\255\250\200.-\255\240", "", 1)
end -- while loop
x = string.gsub(x,
"\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240%-\255\249", "\201")
x = string.gsub(x,
"\255\250\200Char.Vitals\10NL:%d+/%d+ H:%d+/%d+ M:%d+/%d+ E:%d+/%d+ P:%d+/%d+ N:%d+/%d+ W:%d+/%d+ \255\240","")
for _,d in ipairs(c) do
x = string.gsub(x, "\201", d, 1)
end -- for loop
end -- if
if GetEntity("timestamp") == '1' then
x = string.gsub(x, "\255\249", " {" .. os.clock() - math.floor(os.clock() / 100) * 100 .. "}\255\249", 1)
end -- if
return x -- return amended packet
end -- function OnPluginPacketReceived
First thing, while it won't cause any problems, this is confusing:
if not blackout then
-- some stuff here
elseif blackout then
-- other stuff
end -- if
The "elseif blackout" is redundant, why not just use else? ie.
if not blackout then
-- some stuff here
else
-- other stuff
end -- if
Now I'll take a closer look at your problem code, which I think you have correctly identified as the few lines at the start, where the packet is reassembled.
I don't intend to sound rude, but I still don't understand why that matters. I should be able to chop off the end of a packet regardless of whether I plan on attaching it to the next one, right?
That happens to be the line that only gets used if you have this split packet situation.
However I now see another answer to your problem. After carefully looking at your debug output I see an interesting transition:
6000h, 6300m, 6300e, 10p, 27900en, 29400w <--- MUD output
packet: 27900/27900 W:29400/29400 ÿð <--- your debugging
... and so on ...
End of packet
elrx- <--- MUD output
Also a paragraph is constructed when the output transitions from one type to another (eg. user input to MUD output, or script note to MUD output).
In other words, your own debugging is triggering off the newline you notice - once you transition from MUD output to a "print" it forces a newline. Take the debugging out and the newline will go away.
You can avoid this issue by using SendToNotepad instead - this will not cause newlines in your main window.
First, what is tostring for? Converting what to a string? The output from string.match will either be a string anyway, or nil. If it is nil your packet will become (something)nil which is probably not what you want.
Now in the middle of that line is:
string.gsub(x, "\255\250\200.-\255\240", "")
That is simply replacing any other subnegotiations (not the partial one, but another one further on) with an empty string. Is that really right? You are throwing one away.
I don't really see how the string.match will work either.
Wouldn't this be a whole lot simpler? ...
if packet and not blackout then
x = packet .. x
end -- if partial packet
That is adding the extra bit that you saved previously back into the packet. Then just fall through and process it as if it happened to be at the start of the packet anyway.
You probably don't even need the "if", what does that achieve?
It didn't seem right that the debugging would be causing the problem since I started the debugging because of the problem in the first place, but looking back you're absolutely right. At every instance where my prompt was split, there was a command echo in between, like so:
The simplest solution I can see would be turn off command echos, but it would be very saddening if that turned out to be the only solution.
As for my code, yes, it is likely to be unnecessarily complicated at parts and has room for improvement. The "if packet ... then" was added in hopes of increasing speed by having to evaluate one less Lua regex most of the time. In fact, since Lua regexes seem so much slower than Mushclient regexes, I plan on allowing some of the subnegotiations to pass to Mushclient for evaluation in the final version of my plugin.
Ah that explains it. You might consider simply doing a DoAfterNote with a very short interval (eg. 0.1 second) to try to get the note to appear after the prompt fully arrives.
As for the regexp, the Lua ones are very fast, however you aren't helping by doing multiple ones where one would do.
For example:
while string.find(x, "\255\250\200.-\255\240") do
subnegotiation(string.match(x, "\255\250\200.-\255\240"))
x = string.gsub(x, "\255\250\200.-\255\240", "", 1)
end -- while loop
You are basically doing exactly the same search 3 times, when one would do.
string.gsub can take a function as the third argument, so rather than doing what you are doing:
Is there a subnegotiation?
If so, refind it and pass to subnegotiation function.
Make sure the subnegotiation function returns an empty string.
And by putting the .- inside brackets, the subnegotiation does not have to - once again - find the part in the middle. The argument would simply be the middle bit.
This replaces a while loop, and 3 regexps, called repeatedly, with a single regexp. I think you will find that much faster.
Wouldn't DoAfterNote give just as much chance of splitting a prompt 0.1 seconds later? I think I may end up doing something like using OnPluginSend to store my commands and then Note them after the next prompt.
At the time I wrote the plugin, I was using gsub for the first time and didn't understand how you could use functions as your replacement, so, yes, that is another thing I could improve. :)
You could probably also simplify the stuff that stores the trailing bit. Instead of:
packet = string.match(x, "\255\250\200.-$")
x = string.gsub(x, "^.*\255\240", "")
How about:
a, b = string.match(x, "^(.*)(\255\250\200.-)$")
if a then -- if trailing subnegotiation
x = a
packet = b
end -- if
This sees if you have some final stuff and if so make the x variable (which you are returning) everything before it, and the packet variable everything after.
I think you are asking for trouble a bit here though. What if the packet ends in \255 and not \255\250\200? Or what if it ends in \255\250 only?
You should probably do this:
a, b = string.match(x, "^(.*)(\255.-)$")
if a then -- if trailing subnegotiation
x = a
packet = b
end -- if
Then when the packet gets prepended next time through, we assume the rest (from \255 onwards) will be there somewhere. Either in the old packet or the new one.
Wouldn't DoAfterNote give just as much chance of splitting a prompt 0.1 seconds later?
Well that is true, but in situations of high throughput we are assuming the server is pumping stuff out pretty quickly.
If you were worried you could probably batch your comments up (in a simple table) and then in OnPluginLineReceived display any outstanding ones, and clear the table.
OnPluginLineReceived will be called when a complete line has been received, so anything you display should not appear in the middle of a prompt.
Well that might work but I can forsee problems. What is going to force the command out exactly? I thought what I saw above was a note, but you are saying it is a command, is that right?
If you store a command for later, to be sent when you receive something (like a line), imagine what happens if there is no MUD output at all, and you type something like "inventory".
Now your OnPluginSend puts that in a table, and waits for a line to arrive from the MUD. So, it seems to you that your command has done nothing.
Maybe if you only did that if "packet" was not empty, then this would work, as you would assume another packet will arrive shortly.
Ah, no, I guess I wasn't clear. What split the prompt was a command echo. My idea was to uncheck the "Echo My Input In" option in the commands configuration, then, while still allowing commands to pass, keep track of what commands did pass (using information from OnPluginSend) and then Note them when I receive a complete line to simulate a command echo without a risk of it splitting prompts.
It turns out that won't be necessary after all. I think I mentioned earlier that I planned on allowing the subnegotiations that create a second prompt to pass to output. The problem before was I was trying to make a single line prompt from text that could possibly arrive divided in two packets, creating the possibility of a well timed note or command echo splitting up the prompt.
It turned out that creating a two line prompt by allowing the subnegotiation prompts to pass, in conjunction with combining incomplete subnegotiations with the next packet, eliminated the possibility of splitting the subnegotiation prompt, instead putting any would be splitters between the two prompts. It's still possible to split the first line of the prompt since I don't specifically check for incompleteness on it, but it's much rarer and of comparatively little consequence.
So, with that in mind I simply gag the two prompts and construct one with a ColourNote, and thanks to Mr. Gammon's code advice it's all lightning fast. Thanks for the help.
