Hello monkeys,
So my dull day job is developing high-performance low latency trading system (you can yawn now!). I’ve just started a project using Apache MQ and have been truly amazed by the throughput we’re getting, even on our modest spec servers. We’re producing and consuming around 2,000 messages a second atm, and that’s with no load-balancing considerations.
So on a whim, I started to play around with using it to create a multiplayer “room” in JME just to see how far I could get. I’ve created a really basic framework where each “player” posts their location to the server (via a JMS topic). Other players then consume these broadcast messages and update accordingly. It’s working nicely.
So my question is: Would anybody like me to document what I’ve done? I don’t want to tread on the toes of Spider Monkey, but not sure exactly what stage that’s at. It seems like Darkstar/Red-Dwarf is pretty much dead and buried as well, so it could be something useful for those considering it. It’s actively being developed, and open source too.
Let me know if you think it’s worthwhile and I might even put some youtube videos up on the subject!
Cheers
Richard
I for one love documentation on anything! (I still have yet to read up on networking, so this could be pretty helpful!)
hi, i’m also interested in this topic… I have tried to set up an Apache MQ for some other web projects in the past, and was thinking about trying it in my game. I’m just about to start the networking parts, having done some stuff in spidermonkey already, have tested red dwarf, and I still would like to try more approaches before choosing the final implementation. 
Curious… is this using TCP or UDP?
Also, can you tell me more about the 2,000 messages per second throughput… is that all local to the machine or from one host to another? And is that just one endpoint to another endpoint or do you see similar throughput to multiple clients, ie: when there is actual broadcast dispatch happening?
I want to draw a proper comparison to Spider Monkey (which is pretty solid, by the way) but I want to make sure to compare apples to apples.
Hi guys, thanks for the interest.
I’m using TCP (I can’t risk message loss). AMQ supports TCP, UDP, SSL, NIO etc as well should you need them. With regards to the 2,000 messages, this is based on my business logic throughput running AMQ on a pretty old dual-core processer server (E7200-2GB RAM) with another pc posting the transactions, and (for my testing) 2 pcs consuming. Default AMQ persistence is enabled. I posted 10,000 messages to the server per test, which took on average 4800ms each time (2,083 messages per second).
I am posting to a topic (aka broadcast), so yes, that would be 2,000 messages a second to each individual client so long as the network pipe is fat enough. It sounds like this is pretty poor compared to modern servers (or even modern pcs by the sounds of things!) that can handle 20K+ messages a seccond (CHECK THIS LINK HERE). FYI, I believe network latency was < 1ms per message, so I was satisfied that this was negligible enough to be discarded from the test results. No need to get netperf out just yet!
My SLA is 100-200 messages per second, so I pretty much stopped the performance testing based on my results ( 1000% faster means it should scale for the next few months 
).
Like I mentioned pspeed, I don’t want to step on the work you’ve already been doing. Of course, using the AMQ approach is very much a “roll-your-own” style, whereas the stuff you’ve done has a solid framework in place.
Let me put together what I’ve done and you can take a look.
Cheers
Richard
It was funny… running the SpiderMonkey throughput test to check some numbers I finally found a bug that had been reported once but never actually seen in a real application that I know of. It is the only known network bug in Spider Monkey and I just fixed it.
For real time games, UDP for some kinds of messages is required. It was a huge glaring weakness of Red Dwarf and why that pretty much has to be limited to turn-based games. The issue is with messages that are made obsolete by the next message… like world state updates, etc… position information that becomes stale with the next position update. TCP requires that all of these messages are received and it requires that they all get received in order. This is usually ok but a single packet problem can cause long delays as the pipe tries to catch up and receive all of those old stale messages.
I had an app once that used TCP to synchronize camera movement between several views. We already had TCP messages so we just reused it. Every now and then a client would get a 1+ second pause and then suddenly the camera would zoom through all of the data once it caught up. And this was on a local LAN with all machines in the same switch. Over the internet, these situations can compound to the point where you get regular hiccups.
The bottom line: non-turn based games need to consider this issue if they don’t want their clients freezing for seconds at a time and/or warping all over the place. (Because the other option is to do predictive movement but then you end up warping the player when the real data comes in and making them retrace their steps.)
Anyway, Spider Monkey supports both UDP and TCP over the same channel. If you want to try it sometime it’s much more geared towards this kind of application versus a general enterprise message architecture. (For example, you get things like server side per connection session data.)
On my 6 core AMD running the throughput tests (client and server run in the same process but still use the network layer, non-loopback) with TCP I get around 34,000 messages per second (and that’s client->server->back to client) and 43,000 or so with UDP. And the CPUs are only moderately busy.
For Mythruna, even with 10 players on at once from various parts of the world, I’ve never seen network usage go above 1% and CPU usage generally never above 3%… on a 6 core system.
In my opinion, the only “weak” part of Spider Monkey is the Serializer stuff… and that’s mostly because getting the hang of setting up the messages the first time can be tricky (always register all @Serializable classes on both client and server in the same order). It does have the benefit of making really really tight data. It’s as compact as you can get without resorting to something like bit streams. For example, a message with 3 floats in it takes 16 bytes. A message with a Vector3f in it takes 17 bytes. Very tight.
Don’t get me wrong… Apache MQ and Mina, etc. are great tech but they are really trying to solve a different problem. So it might be worth looking at something like Spider Monkey just to see.
Cheers for the update. 34,000 messages a second sounds outstanding enough to be worth a look.
What is also clear is that you’re still actively developing on this. My initial searches of the JME website lead me to believe it was still in development to some degree. I’ll definitely take a look.
Incidentally (having not evaluted the SpiderMonkey API yet), do you have any guidelines for handling out-of-sequence UDP location messages?
I do actually disagree somewhat with your last comment about solving different problems seeing as AMQ does support UDP. The JMS header overhead is a valid point, and I would never have considered JMS 2 years ago, but these days with high-speed broadband, that’s pretty much moot.
Also, that 1 second tcp lag you speak of is probably due to congestion control (ie Nagles Algorithm). I see this a lot, and there have been times when it’s been necessary to play around with the TCP_NODELAY option, which will eradicate this. World of Warcraft, Guild Wars and (I think) Final Fantasy XI adopt this approach.
- Richard
Yes, you can drop nagle’s (and Spider Monkey does) but latencies vary and add up. One lost packet can be pretty expensive latency wise on a TCP socket… and the rest of the data has to be held before that packet gets there.
I still actively develop on Spider Monkey in the sense that Mythruna uses it (that’s why I rewrote it actually) and if I find or hear of something wrong then I fix it. The API is stable and the implementation is stable other than bugs found. Just stay away from the deprecated packages. The only reason there aren’t better docs is because the wiki swallows line feeds when you re-edit something and that makes me want to break things.
UDP location messages need to be timestamped, really. If a real-time display is what you are going for then there is going to be some level of interpolation anyway… so those times will be necessary. (If you don’t interpolate then a rolling message index value can work, too.)
Some articles worth reading on that subject: http://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking
and: http://developer.valvesoftware.com/wiki/Latency_Compensating_Methods_in_Client/Server_In-game_Protocol_Design_and_Optimization
I read those three of four times before I implemented Mythruna’s networking. It also talks about bandwidth and how that breaks down. Quite humbling. And since I know several people who cannot get broadband and still play games like WoW over a modem… it was a hard decision to mostly ignore that user when I implemented my network layer. (Though as it turns out, it is possible… just not pleasant.)
Regarding AMQ, I generally think of it in context of Apache’s other service bus style offerings or as a JMS underpinning. It felt like that’s the kind of thing it was geared towards. And JMS over UDP makes me shudder for some reason. So maybe I unfairly categorize it but maybe it’s also not strange that other gaming network APIs don’t use it.
The rewriting of Spider Monkey actually took some of these other architectures into account. It’s why I separated the raw byte[] network dispatch/receipt from the more game-like API layer. All of that is hidden from the typical Spider Monkey user but the base Client and Server implementations are written on top of a much simpler kernel layer that simply routes bytes to endpoints.
But for most users, they get a nice set of Client and Server+HostedConnection interfaces that does the magic of managing the TCP and UDP connections, connection setup, etc. for them… along with listener dispatch, session data, and so on.
If you play with it and find problems… let us know.
OK, seeing as I’ve got the devlopers attention I am already falling at the first hurdle (sort of!).
I’ve tried the following:
[java]
Server server = Network.createServer(12345);
server.start();
[/java]
No errors, but thre thread ends straight away. I’ve tried attaching a message listener and a connection listener to see if that helps… No good.
I’m using a 3-day old version from the nightly downloads. I’ve only included the JMonkeyEngine.jar in my project.
Tell you what mate, if/when I get this working, I’ll stick some docs on for you OK?
- Richard
Also meant to ask. When calling a “broadcast(…)” method, how do you define whether it’s UDP or TCP?
Thanks
Richard
I just want to note that UDP is not a requirement for real-time games. As an example, World of Warcraft only uses UDP for one thing: voice chat. Everything else the game does goes over TCP[1]. The game has some very heavy client-side prediction logic, and surprisingly enough over the years I played the game I came to realize that they mostly trust character position information coming from clients (ala, you almost never experience rubber-banding on your character, except during the worst lag). To compensate they have a number of tools on the backend that do quite verifications of sudden jumps of player movement and report you to a GM if something is amiss. So yes it is quite possible (and I’ve seen plenty of videos) to write a speed hack or a teleport bot for WoW, but you aren’t going to get away with it for long.
So it really comes down to “can your game afford to lose a packet of information, or worse, get packets in the wrong order?” If that answer is no, then use TCP, because working to build a reliability framework on top of UDP (which, lets be honest, is simply re-implementing TCP) is, in most cases, way too much work for the benefit you get.
[1] http://us.blizzard.com/support/article.xml?locale=en_US&articleId=21015
Cheers for the input James. I confess I would NEVER have considered using JMS for any sort of truly high-performance network applications a couple of years ago. But we live in ever changing and faster processing times. Hence the reason for this thread really. AMQ scales well (as I’ve been finding out over the last couple of days) with the simple introduction of more servers (across multiple servers if required) and comes with a Web administration console that gives you control over queues/topics etc.
I’m still going to give SpiderMonkey a run for its money and see how far I get.
- Richard
richard_hawkes said:
OK, seeing as I've got the devlopers attention :) I am already falling at the first hurdle (sort of!).
I've tried the following:
[java]
Server server = Network.createServer(12345);
server.start();
[/java]
No errors, but thre thread ends straight away. I've tried attaching a message listener and a connection listener to see if that helps... No good.
I'm using a 3-day old version from the nightly downloads. I've only included the JMonkeyEngine.jar in my project.
Tell you what mate, if/when I get this working, I'll stick some docs on for you OK?
- Richard
With no connections and nothing else holding the app open the server will shut down. Usually a server is started and then some console runs or is started from a UI of some kind and these would hold the application open.
So, for example, after starting the server you could enter into a loop watching System.in and waiting for some kind of exit command.
richard_hawkes said:
Also meant to ask. When calling a "broadcast(...)" method, how do you define whether it's UDP or TCP?
Thanks
Richard
On a message, the "reliable" flag determines if it should go out on UDP or TCP... true for TCP, false for UDP. Or on a received message will tell you how it was received.
jameskilton said:
I just want to note that UDP is not a requirement for real-time games. As an example, World of Warcraft only uses UDP for one thing: voice chat. Everything else the game does goes over TCP[1]. The game has some very heavy client-side prediction logic, and surprisingly enough over the years I played the game I came to realize that they mostly trust character position information coming from clients (ala, you almost *never* experience rubber-banding on your character, except during the worst lag). To compensate they have a number of tools on the backend that do quite verifications of sudden jumps of player movement and report you to a GM if something is amiss. So yes it is quite possible (and I've seen plenty of videos) to write a speed hack or a teleport bot for WoW, but you aren't going to get away with it for long.
So it really comes down to "can your game afford to lose a packet of information, or worse, get packets in the wrong order?" If that answer is no, then use TCP, because working to build a reliability framework on top of UDP (which, lets be honest, is simply re-implementing TCP) is, in most cases, way too much work for the benefit you get.
[1] http://us.blizzard.com/support/article.xml?locale=en_US&articleId=21015
It's also interesting that they seem to open six different TCP channels... which I guess is one way around worrying about one channel getting clogged. In another discussion we sort of concluded that most of these RPGs actually have cleverly disguised turn-based combat. So I guess it works out pretty well.
I would never write a game the _exclusively_ used UDP. UDP is good for the stuff that is time sensitive... like world state. Mythruna uses both TCP and UDP. TCP is for communication of user issued "commands" and setting blocks as well as reliable communication of the entity system state, events, etc.. UDP is used for transient object position and orientation... the stuff that is useless after a few hundred milliseconds. And it seems like using a system to work around the limitations of TCP is as bad as trying to write your own TCP on top of UDP. My systems are simpler for not having to worry about that level of prediction.
I sort of have to do it this way, though, since "killing" something in Mythruna may be the result of a series of physics events that would be hard to stick into any kind of turn/initiative queue. And prediction may be "really hard" given the physics elements I want to employ.