Register forum user name Search FAQ

Gammon Forum

Notice: Any messages purporting to come from this site telling you that your password has expired, or that you need to verify your details, confirm your email, resolve issues, making threats, or asking for money, are spam. We do not email users with any such messages. If you have lost your password you can obtain a new one by using the password reset link.

Due to spam on this forum, all posts now need moderator approval.

 Entire forum ➜ MUDs ➜ General ➜ Suggested protocol for server to client "out of band" messages

Suggested protocol for server to client "out of band" messages

It is now over 60 days since the last post. This thread is closed.     Refresh page


Pages: 1  2 3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #15 on Wed 03 Feb 2010 07:24 PM (UTC)
Message
Twisol said:

I have to agree with David here, I don't see the benefits of this over ZMP in particular. If you wanted to use Lua as part of the protocol, you could probably define a subpackage in ZMP for it quite easily.


I am having trouble finding the full spec for ZMP (not a good sign in itself). I found this page, which seems fairly light on detail:

http://zmp.sourcemud.org/spec.shtml

Another link I found just led to an advertising page (ie. the original had shut down).

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #16 on Wed 03 Feb 2010 07:28 PM (UTC)
Message
Worstje said:

Awesome, Nick. Great to see such an option implemented.

The only thing I am sad about is that your implementation limits two things:

1) People are bound to use TELOPT 102. Which some muds don't use. I know I like my ATCP plugin (or Twisol's, or the other dozen varieties out there people wrote for themselves) but it could be rewritten so much simpler and more efficient if it didn't have to re-parse all input for telnet codes and such. Simply giving codes that weren't implemented by MUSHclient proper to a OnPluginTelnetOption(num, telopt) handler seems like a way more solid idea, since existing games aren't going to change their codes just for MUSHclient.


Thanks for all the positive feedback everyone.

For a start, the whole project is just a suggestion to kick along the idea of sending more data from MUD servers to clients. The code 102 doesn't have to be used for example, and the limit on message sizes could be removed.

I was thinking of adding support for the ATCP codes (except that for internal reasons MUSHclient doesn't like newlines inside subnegotiation but that can probably be fixed).

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #17 on Wed 03 Feb 2010 07:39 PM (UTC)
Message
David Haley said:

I'll fess up straight away that I don't really see the advantage of this over, say, ZMP. ... Were there specific issues with other protocols that you wanted to address?


I think I looked at ZMP a while back. The thing that threw me was the design decision to use the null byte (0x00) as an internal terminator. Straight away, for a language-neutral implementation, that made it hard to see how I could capture the incoming text, and pass it to any script language, when many languages use zero-terminated strings to pass strings, and indeed so does MUSHclient in many places.

Now, while MUSHclient has it at a low level (incoming data processing) it is able to handle the 0x00 bytes, but then it has to somehow place an indefinite number of parameters into a table in such a way that any script language could process it. I suppose it could be a VBscript array (like the way it does wildcards) so the problem is not really insurmountable.

Had he chosen *any other character* (eg. 0x01) then it would have been simple to just collect the data as a string and pass it to a plugin (like I do with the OnPluginTelnetOption). Then the plugin (in any language) could break the string down at the delimiter.

I also looked at MCP and thought while I read that "why does everything have to be so complicated?".

Basically my proposed system is *simple*. You don't need to use some new data protocol, you don't need to build up strings with internal 0x00 bytes (you must admit, that in C this rules out using strcat, for example), you don't need to learn a whole new set of rules for transferring data.

Plus, the Lua tables can, when required, be used to simply and easily store things like an entire player's inventory (including stuff in bags), or a room with exits, where you store the attributes of each exit).

However having said that, I don't want to be pig-headed about it. The next version of MUSHclient could:


  • Allow unlimited message lengths

  • Have a callback for ATCP messages

  • Have a callback for ZMP messages


Then you can choose protocol you want. A particular MUD doesn't have to convince the whole community. Provided a couple of mainstream clients support it, players who want the extra features can use them if they want to.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by David Haley   USA  (3,881 posts)  Bio
Date Reply #18 on Wed 03 Feb 2010 07:43 PM (UTC)
Message
ZMP, as far as I can tell, lives on MB:
http://www.mudbytes.net/index.php?a=forum&f=41
Elanthis (the guy who wrote the spec) frequents MB quite a bit, so it's probably the best place to get in touch with him.

AFAIK, ZMP didn't "take off", which is why it's hard to find references to it.

But in all honesty, I think that it -- or some variation of it -- is a better suited protocol for data transfer, due to the fact that semantics are encoded to some extent in the commands themselves. Having things segregated into packages (i.e., namespaces) also makes it fairly easy to provide "ZMP plugins".

Furthermore, data transmission in ZMP is more than just assignment; this can be more natural for commands that are indeed commands or functions, as opposed to simply passing bits of information. I guess what I'm saying is that it is useful to have data transmission in richer forms than assignment.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
Top

Posted by David Haley   USA  (3,881 posts)  Bio
Date Reply #19 on Wed 03 Feb 2010 07:47 PM (UTC)
Message
I made my post while you made yours, apparently. :)

Nick Gammon said:
The thing that threw me was the design decision to use the null byte (0x00) as an internal terminator. Straight away, for a language-neutral implementation, that made it hard to see how I could capture the incoming text, and pass it to any script language, when many languages use zero-terminated strings to pass strings, and indeed so does MUSHclient in many places.

I don't really see the issue here. When you get a message, you can split it on the zero string (although admittedly, yes, not using normal string library functions in zero-terminated languages). Then you just pass the arguments as an array of parameters as you would in any other case.

That said, using any other character means you need to escape it; a null byte is extraordinarily unlikely to be needed anywhere. (Then again you might say the same about 0x01, too...)



To be honest I'm not wedded to ZMP, I haven't implemented it in my MUD, etc. But I like the fact that it's not just about passing data, but also passing true commands. To hammer out what we really need, we'd need to go to good old use case analysis. :P

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #20 on Wed 03 Feb 2010 08:16 PM (UTC)
Message
David Haley said:

ZMP, as far as I can tell, lives on MB:
http://www.mudbytes.net/index.php?a=forum&f=41


Well I had to laugh. There is a lengthy thread there about the maximum message size. Ah well, we go over the same ground again eh? And I note that some of the arguments used are similar to mine (eg. an inventory even, shouldn't be more than 5Kb, but what happens if it is 5Kb + 1?)

elanthis said:

Even a well-written client is still going to have a buffer size limitation, too. Otherwise a server could send IAC SB followed by 2GB of data with no IAC SE and cause bad things to happen (which has nothing to do with ZMP -- it could be _any_ subrequest).


Now, I have for the moment removed any limit - after all, we have a limit or we don't. Now perhaps you and Worstje can fight it out ...

Worstje said:

Look at Zmud and MXP: it was constrained from the very beginning by the limitations of that client, effectively crippling it from becoming as good as it could have been. Taking away the size limitation is a trivial matter

...

please, don't put artificial and arbitrary limits of 5000 bytes or whatever on it.


So, do we want a limit (eg. 1 Mb?) or no limit at all?

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #21 on Wed 03 Feb 2010 08:26 PM (UTC)
Message
David Haley said:

ZMP, as far as I can tell, lives on MB:
http://www.mudbytes.net/index.php?a=forum&f=41


I still can't see a spec anywhere, apart from what looks like an overview. For example, this is about all I can see about packages:

ZMP spec said:

The package system for ZMP is similar to the idea of namespaces found in popular programming languages, such as C++ and Java. Every command sent over ZMP should belong to a package. A package is simply a name, and a specification of the commands that are available in the package.


That doesn't tell me a heap.

David Haley said:

To be honest I'm not wedded to ZMP, I haven't implemented it in my MUD, etc. But I like the fact that it's not just about passing data, but also passing true commands. To hammer out what we really need, we'd need to go to good old use case analysis. :P


Ah yes, well I was trying to preempt the design phase and leap straight into implementation, like your students would. ;)

I'm not totally sure I see the distinction. After all, the data is "passed" from server to client (or vice-versa). Now I suggested that part of the data be a transaction type (the command, if you like). It is semantics really. We are passing X from A to B. I was deliberately trying to not impose lots of rules about what X is, and if you want to build a command, or many commands (whatever a command is) into X, then you can.

In fact, the mention of namespaces is one of the reasons Lua tables are so attractive. My example on page 1 illustrates that:


"victim":
  "level"=1
  "maxhp"=11
  "name"="the kobold"
  "hp"=11
"maxhp"=43
"hp"=40
"level"=2


Effectively here victim.maxhp is in the victim namespace. What is the difference? The format is so flexible you can use tables as namespaces if you want to.

My major point is that once you start having a lengthy spec and saying you should, or must, use certain things like "packages" then you are imposing a learning curve on developers. Perhaps having a complex system is why ZMP doesn't seem to be widely known.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #22 on Wed 03 Feb 2010 08:30 PM (UTC)
Message
David Haley said:

But in all honesty, I think that it -- or some variation of it -- is a better suited protocol for data transfer, due to the fact that semantics are encoded to some extent in the commands themselves. Having things segregated into packages (i.e., namespaces) also makes it fairly easy to provide "ZMP plugins".


Once again, my suggestion of tt field effectively make it what you seem to be calling a package or namespace. A particular message, identified by a tt field, effectively places the rest of the fields into that namespace. So in a "combat" message a "victim" field might mean "the person you are fighting" whereas in a "duel" message the "victim" field might mean "the person you are duelling", or in a "trade" message the "victim" field might mean "the person you are trading with".

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by David Haley   USA  (3,881 posts)  Bio
Date Reply #23 on Wed 03 Feb 2010 09:37 PM (UTC)
Message
I'll have to think more about namespaces vs. tables. Something in my gut tells me that the nice thing about the ZMP notion of packages is that it defines namespaces not only for one game, but across games as well. So I can trust that the command "subwindow.print" will have universal meaning; it is the command "print" inside the (well-defined) package "subwindow".
But then again, you can achieve the same thing by having the message be a table called 'print' inside a table called 'subwindow', with the parameters to print inside the 'print' subtable...

I do like the fact that the Lua syntax is used as it makes non-primitive data much easier to pass around. ZMP makes no provision whatsoever for data types, as far as I can remember. This means that you must define your own interpretation for what it means to have subfields, etc. So the protocol here solves that problem, although it only solves it easily for Lua (everybody else has work to do).


Perhaps ZMP was indeed trying to accomplish too much, in defining not only a transport layer but also trying to enforce package specifications shared across all MUDs. I think the idea was that certain things simply make sense for all MUDs, like font colors, sound effects, and so forth, (in essence, replacing MXP, MSP, etc.), and that these should be encoded in a "standard library" if you will so that any MUD can implement ZMP, emit the expected commands, and the client will do the Right Thing.

Your protocol, by only being a data transport layer, makes no provisions for things like this. But I'm not sure if it's the problem you're trying to solve in the first place.


Regarding limits, I think that it's quite difficult to pick a limit and say that it's a hard and fast limit. I think it might be nice to state, before the data, how long the data is going to be, and let the client choose to drop it or not. (You might have some message saying "I ignored your message because it's too long", or something.)

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
Top

Posted by Twisol   USA  (2,257 posts)  Bio
Date Reply #24 on Wed 03 Feb 2010 09:49 PM (UTC)
Message
On the topic of limits, doesn't HTTP itself state exactly how much will be sent as a header message? That might be a good route to take.

On a personal note, I very much like the namespacing that ATCP/ZMP enforce. Pairing commands with modules/namespaces, and inherently requiring the command to be sent as part of the protocol (as opposed to passing it as part of the data a la your 'tt' parameter), seems to make things less of a bag of data and more of a structured message. What I also like about ZMP in particular is that every parameter is separated by NUL, and it's not hard at all to split by NUL bytes.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #25 on Wed 03 Feb 2010 09:53 PM (UTC)

Amended on Wed 03 Feb 2010 10:01 PM (UTC) by Nick Gammon

Message
Well I am looking at totally rewriting the IAC subnegotiation code in MUSHclient, to try to keep everyone happy.

Now I have a question about Telnet I can't see to find the answer to...

The subnegotiation goes along these lines:


IAC SB c <some data> IAC SE


Where 'c' it the one-byte protocol code (eg. 102 for the option described earlier, 85 or 86 for MCCP, 90 for MSP and 91 for MXP).

Also, IAC IAC is supposed to be converted to a single IAC. For example:


IAC SB 0xAA "hello" IAC IAC "there" IAC SE


OK, so this means protocol 0xAA is going to be negotiating the message "hello" IAC "there" (as the two IACs turn into one).

Now let's assume we have a state machine, and we are processing one byte at a time (they aren't necessarily in the same packet). This is the problem point here ...

We receive:


IAC SB c <some data> IAC x


Now x has two valid meanings:


  • SE - in which case the subnegotiation is over, and we can process <some data> in the context of type 'c'.

  • IAC - in which case the subnegotiation continues, and we replace IAC IAC by a single IAC


But what about the other 254 cases? What do they mean? For example:


IAC SB c <some data> IAC x   (where x is not SE or IAC)


We have neither a second IAC, or a SE, so this seems to me to be undefined. I can see a few ways this might be handled:


  • Terminate the subnegotiation, assuming we really got:

    
    IAC SB c <some data> IAC SE x
    


    Thus we process <some data> in context "c" and then also process the x byte.

  • Terminate the subnegotiation, assuming we really got:

    
    IAC SB c <some data> IAC SE
    


    Thus we process <some data> in context "c" and then discard the x byte (effectively assuming the x was really SE).

  • Keep the subnegotiation open, assuming we really got:

    
    IAC SB c <some data> IAC IAC x
    


    Thus we are still looking for IAC SE (which may possibly never arrive).

  • Terminate the subnegotiation, but not process <some data> on the grounds that it is an incorrect sequence and deserves to be ignored. However, process the extra byte. This assumes we really got:

    
    x
    


  • Terminate the subnegotiation, but not process <some data> on the grounds that it is an incorrect sequence and deserves to be ignored. In addition, discard the extra byte. This assumes we really got:

    
    <nothing>
    


  • Terminate the subnegotiation, assuming we really got:

    
    IAC SB c <some data> IAC SE IAC x
    


    In other words, in this case we open up a new negotiation (eg. x might be WILL, so we processed "IAC SB c <some data> IAC SE" and now might be getting "IAC WILL y".



So, any telnet experts out there may be able to advise me on the correct approach. :-)

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by Nick Gammon   Australia  (23,165 posts)  Bio   Forum Administrator
Date Reply #26 on Wed 03 Feb 2010 10:00 PM (UTC)
Message
David Haley said:

Regarding limits, I think that it's quite difficult to pick a limit and say that it's a hard and fast limit. I think it might be nice to state, before the data, how long the data is going to be, and let the client choose to drop it or not. (You might have some message saying "I ignored your message because it's too long", or something.)


Now things are getting complex. Instead of a simple sprintf we now have to do the printf first, find out how long the results are, and build that number back into the *start* of the message you already have. And as for "I ignored your message" that assumes a two-way protocol which I wasn't insisting you use. And if you say "I ignored your message" (a response which might arrive a few seconds later) I would ask *which* message did you ignore? In which case you now need to add message sequence numbers. And what does the server do if the client says it ignored the message? Send "whatever" to it?

- Nick Gammon

www.gammon.com.au, www.mushclient.com
Top

Posted by David Haley   USA  (3,881 posts)  Bio
Date Reply #27 on Wed 03 Feb 2010 10:02 PM (UTC)
Message
My reading of the spec is that "IAC x" will be ignored unless 'x' is the code for a known command.

So my interpretation of

IAC SB c <some data> IAC x

is that we should keep on processing the subnegotiation, and pretend that we never received the 'IAC x' if 'x' is not a known command.

I'm not sure why 'x' couldn't be any of the several other defined commands, as given at the bottom of http://www.faqs.org/rfcs/rfc854.html. It might be weird, but I guess it's consistent.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
Top

Posted by David Haley   USA  (3,881 posts)  Bio
Date Reply #28 on Wed 03 Feb 2010 10:06 PM (UTC)
Message
Nick Gammon said:
Now things are getting complex. Instead of a simple sprintf we now have to do the printf first, find out how long the results are, and build that number back into the *start* of the message you already have. And as for "I ignored your message" that assumes a two-way protocol which I wasn't insisting you use. And if you say "I ignored your message" (a response which might arrive a few seconds later) I would ask *which* message did you ignore? In which case you now need to add message sequence numbers. And what does the server do if the client says it ignored the message? Send "whatever" to it?

Yes, it's complex, and yes, you'd need sequence numbers of some kind. Still, I'm not sure how else you can allow unlimited messages, while still giving the client some control over how much memory it wants to use. As soon as you fix some limit, you run into that good old limit+1 problem. There's little reason to believe why you wouldn't want to send binary image data around, at which point your protocol limit is essentially defining that you can't send images that are "too big". So what do you pick? 500k, 1mb, 3mb, 10mb, 300mb? etc.

The server would probably ignore "your subneg was too long" messages, or perhaps send a (non-subneg) message to the player saying so.

I don't really like the "content length" message in this case, but I don't view it as too terrible, either. It's not really different from what HTTP does.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
Top

Posted by Worstje   Netherlands  (899 posts)  Bio
Date Reply #29 on Wed 03 Feb 2010 10:13 PM (UTC)

Amended on Wed 03 Feb 2010 10:15 PM (UTC) by Worstje

Message
I'm too lazy to piece together all kinds of different quotes, so I'll just reply based on what I remember reading:

Maximum length:
Awesome, no more limit. Maybe it will never be surpassed, but at least we are not stopping any possible use cases, nor complicating those cases where we indeed have 5k+1 characters to transmit.

Seperator byte:
My C implementation of the telnet protocol has zero issues with \0 bytes and allocates memory for telnet sub-negotiation sequences in decent steps thus reasonably optimizing for the smaller cases, but never limiting them. As I said before, if you want it, I can upload it somewhere. (I have optimized the telnet parser as a whole for speed as well so it could save some time in that sense.)

OnPluginTelnetOption callback:
Nick, you spoke of different callbacks for different protocols if I understood you right. I am talking about a single subnegotiation callback for all of the telnet options MUSHclient does not handle itself. Or even better, I'd love it if we could actually take up the choice for the primary negotiation ourselves for such cases. (Not to fall into a repeat affair with my mudserver codebase, but I also implemented a lua API for telnet sub-negotiation in there, keeping proper track of negotiation states and all that jazz.)

The protocol(s) itself:
Let's not fuss about what protocol we are using precisely. Not in the slightest because most plugins will still be quite mud specific, since all muds tend to differ in their gameplay mechanics.

I don't feel ZMP is the perfect solution, since from what I see it has considerable flaws. First of all, it keeps mentioning commands, and I don't see anywhere about plain notifications that do not require a response. I am also afraid it brings along too much of an urge to start doing all the scripted actions in subnegotiation by both mud coders and plugin writers alike, thus moving the actual 'playing' away from the primary channel which really already is a command-response kind of medium. Second, it has limitations on the content-bytes, no NUL bytes and so forth. The telnet protocol has no such limits, although it does ask IACs to be doubled as a form of escaping. Last of all, it seems to have packages and parameters, but beyond that it is all quite unstructured. Essentially there are only string parameters (unless I misunderstand). Nothing like arrays/lists/dictionaries unless you go and hack around a bit. In essence, really simple commands can easily be implemented, once you have anything with a bit more structure you'll be hacking away outside of the spec.

The new Lasher-Nick-Lua protocol also has its issues. Most of all that it IS quite Lua centric. I would prefer to see a straight table be submitted rather than the current different assignments into a setfenv()ed context, because this way, we can gently guide mud authors into writing more compatible messages with other languages. If you allow free-form Lua, you don't only need to worry about sandboxing and such (ok, that seems to be taken care of in your example) but also about the kind of commands you end up allowing. For example, I can imagine some weirdo scripters use for loops, while loops and the like to achieve what could be sent over in a simple table with said loops being done server side - all of which really complicates a Python implementation for example.

However, simply supporting a simplified table syntax would be simpler to implement in such languages. The only limitations that I can think of right away would be 1) no inline function declarations, and preferably 2) no mixing of lists and dictionaries in such tables as that is quite Lua unique. Then you'd get something like (extra whitespace for clarity, and way too specific inventory for the sake of being precise):

<IAC> <SB> <102> {
move=110,
poisoned=false,
mana=94,
maxhp=43,
hp=42,
tt="status",
combat=false,
level=2,
maxxp=7084,
gold=900,
xp=2116,
maxmove=110,
maxmana=94,
inventory = {
   [1] = {
      name = "book",
      desc = "a magical book of Lua wizardry",
      pages = {
          [1] = "First page",
          [2] = "Second page",
          [3] = "Last page already."
      }
   },
   [2] = {
      name = "glasses",
      desc = "some glasses"
   }
},
dead=false
}


Which could also be loaded directly into a variable in Lua (_, blah = loadstring("return "..telopt_data) if I recall the syntax etc correctly). And for Python and other languages, writing a simple parser would be quite elementary (syntax is simple, C parser is available freely also, as are other languages I'm quite sure).

And in the case Lua isn't good enough, there's still other options. XML for the ones with suicidal tendencies (not my choice at all for live data like that, too verbose), but another choice might be JSON. My personal choice would likely be bencoded data (the format used in .torrent files). These last few are already established formats with the kinks worked out of them, and interpreting might be easier for clients due to existing libraries and such.

Either way, my point is... ZMP has flaws. The new LNL (Lasher-Nick-Lua) has flaws. XML has flaws. JSON has flaws, as does Python, VBScript, as well as bencoding. The proper protocol completely depends on the use case as well as the person implementing it.

Did I forget anything? :D Edit: Apparently yes, a small dozen new posts showed up while I was typing. *goes to read those now*
Top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


928,612 views.

This is page 2, subject is 23 pages long:  [Previous page]  1  2 3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  [Next page]

It is now over 60 days since the last post. This thread is closed.     Refresh page

Go to topic:           Search the forum


[Go to top] top

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.