I am beginning to design the network framework for an online multiplayer game which is to have it graphics portion implemented in jME. This next bit may sound a little crazy but bear with me. I've decided to abstract all decision-making to the server, such as how Counterstrike works, for a number of reasons. Firstly, it will make hacking the client worthless as the client would essentially be just a graphics engine and would have no control over the state of play beyond standard input. Secondly, assuming the number of players on the server is small enough to handle easilly, it will leave more CPU-time free to the client for fancy effects and other CPU-intensive things. The last big reason is that updating the game would be discreet - I could just edit a file on the server when it's down and nobody would know I had updated it or have to download any new content unless it were a change to the client or the introduction of new models and textures. I've also always favoured the "the server is always right" idea, that whatever happens in an online game occurs at the server end
I've started wondering about how I will handle client-server-communication. The basic idea I've been throwing around is that the client runs on a main game loop of 60FPS and each loop sends any keyboard input or commands generated by the client (such as to pick up an object) to the server. The game will act on the input and move the character rather than wating for a response from the server, to give the illusion of zero lag and keep the game responsive. The server recieves the information on keyboard input etc and decides what happens in the game, changing the data accordingly. The server moves all of the players according to the keyboard input they sent and updates their position in the its own data. Every 100 milliseconds (I could change that but 100 sounds like a good tick rate), the server sends every client a package of data containing the position and state of every object which has changed since sending the last package. The state refers to the orientation of their model and any special effects to apply (such as starting to cast a spell, an item being destroyed in the world due to being picked up. On top of that, anything relevant only to each player will be sent only to them.
So the client will recieve the position of the objects every 100 milliseconds if they change. In between, it will use a dead reckoning method to predict the position of the person based on their last known trajectory. Upon recieving a package, all object positions and states will be set to the data in the package (and special commands activated on them where appropriate) except the data for your player. Each client will only update the data it holds with the data it recieves from the server if the distance between the two is above a certain value (likely a function of the player's ping). That way, the player can move their character and see it moving completely normally but the reality on the server is that their actions are happening with a slight lag due to internet latency. Of course, it would also have to resynch the player's position when standing still if it were off at all. That should be no problem at all and I'm sure people wouldn't notice their player sliding a few pixels to the side now and then ^^.
People are going to reply and tell me to have the client report its position to the server periodically rather than using real-time input and I WILL consider this if the lag turns out not to be handlable with the first method but there's a good reason I'm not happy with doing that. Speed hacks. Every game has speed hacsk because they give the client control over its own position within the world it knows. People have hacks in some games to teleport about the place using that. I would very much like to avoid giving the client too much control over the gamestate.
So any ideas or comments, people? I know some of you are real network buffs and can probably see a hundred flaws in the idea already so I'd like to hear a few potential problems before I spend time on the coding.
You should definitely synchronize positions between the server and the client (after all, what does it take? 16 bytes?). But this doesn't have to originate from the client. Of course this generates more lag for the player than doing everything client. You could take a look at this topic: http://www.jmonkeyengine.com/jmeforum/index.php?topic=2238.0
And of course darkfrog has done work on this, even made a lib out of that. You can see his topics on this forum. I'd take a look at that first.
The problem with just synchronising position between server and client is that either the client is in control and is sending information on its position to the server (in which case other players might see your movement as jerky if you lag) or the server repeatedly tells the client where they are and the client feels unresponsive due to lag. It's about giving the server absolute authority on positional information while letting the client feel responsive.
What I've described would actually be synchronising the position between the client and the server but the server would have absolute authority on the data to keep it nice and secure. The only difference is that the client wouldn't show the synchronisation exactly, instead allowing you to appear to be moving without lag and then catching up from the server. It wouldn't let you move a certain distance past where the server thinks you are to limit the amount you can become desynched with the server and I will work out a method of tending toward the real position on the fly.
I've seen darkfrog's library but there isn't much documentation or any examples of use to it. To be blunt, I'm not good with other people's code unless it's as superbly documented as the jME docs are. Without that, it's almost as much work to read through and understand what the code is doing as it would be to write it myself.
Hey Nyphur, your ideas sound quite similar to what I would choose to implement for a network layer for a game. And I think it is similar to darkfrogs actual plans and implementation, too. So go ahead it can't be that wrong 
I would expect it to reduce the felt lag alot for computer controlled entities. Player controlled entities might teleport around a bit when actual lag is high (as behaviour prediction will be less acurate) - this could be possibly improved by interpolating between current position and received / computed positions…
Nyphur,
Forgive the lack of documentation, but the project is not completed yet and in fact I've almost completely re-worked the ODENetworking stuff I started out with. If you are willing to wait a couple days I would appreciate some good feedback as I will be releasing the first version of my API. In fact, I've pretty much thrown away ODENetworking entirely in favor of a jME-Physics Networking API. This has a lot of good JavaDocs and I'm currently working on a test-case to show simple networking between a client and server.
The idea is very similar to yours. The server is authoritative about all objects and pushes updates out to the clients. However, the clients are responsible for sending updates down for their own objects as well. The server is still authoritative as it has to make decisions on whether or not an action sent from the client is allowed (to keep client hacks from occurring), but this solves the problem of dealing with characters, weapon fire, etc. that is controlled explicitly by the client. The whole concept here is the synchronization of physics objects. There is a duplicate physics world running on the server to that which is running on the clients (less the visual aspects). This way the clients can run perfectly smooth on their own and just send updates down to the server and receive updates back from the server for synchronization. This solves the problem of lag making your game extremely choppy or having to send more information than is necessary to keep the game in sync.
darkfrog
I don't know if this will help but read the documentation for Torque's networking, OpenTNL. It might be at http://www.opentnl.org, but I can't remember.
irrisor said:
Hey Nyphur, your ideas sound quite similar to what I would choose to implement for a network layer for a game. And I think it is similar to darkfrogs actual plans and implementation, too. So go ahead it can't be that wrong :wink:
I would expect it to reduce the felt lag alot for computer controlled entities. Player controlled entities might teleport around a bit when actual lag is high (as behaviour prediction will be less acurate) - this could be possibly improved by interpolating between current position and received / computed positions...
I was thinking of using a system not for "teleporting" a player by synching their co-ordinates but to slow players down if they're getting too far ahead of the server. Guild Wars uses a similar system where you keep moving unti you get a certain distance from the last server reference for your position, then you slow down markedly. And yeah, I was thinking the dead reckoning would make entities controlled by other players jump around a lot. Any fancy methods of making it look natural would just eat up CPU and it should only happen when the lag is stupidly high.
darkfrog said:
Nyphur,
Forgive the lack of documentation, but the project is not completed yet and in fact I've almost completely re-worked the ODENetworking stuff I started out with. If you are willing to wait a couple days I would appreciate some good feedback as I will be releasing the first version of my API. In fact, I've pretty much thrown away ODENetworking entirely in favor of a jME-Physics Networking API. This has a lot of good JavaDocs and I'm currently working on a test-case to show simple networking between a client and server.
The idea is very similar to yours. The server is authoritative about all objects and pushes updates out to the clients. However, the clients are responsible for sending updates down for their own objects as well. The server is still authoritative as it has to make decisions on whether or not an action sent from the client is allowed (to keep client hacks from occurring), but this solves the problem of dealing with characters, weapon fire, etc. that is controlled explicitly by the client. The whole concept here is the synchronization of physics objects. There is a duplicate physics world running on the server to that which is running on the clients (less the visual aspects). This way the clients can run perfectly smooth on their own and just send updates down to the server and receive updates back from the server for synchronization. This solves the problem of lag making your game extremely choppy or having to send more information than is necessary to keep the game in sync.
darkfrog
I think there is one problem with the idea of the client acting alone and just requesting confirmation/synchronisation and Everquest 2 is seeing it right now. Speed hacks still work. The server can determine the validity of someone's movement quickly by checking if it's over the maximum speed attainable or it can make a few calls and find the character's speed and everything modifying it. In an FPS, this is fine since there's usually just one speed modifier (the weapon carried). For a game like EQ2, they need to call every piece of equipment he has on, check if it gets a speed bonus, add it to the total and multiply the speed by that to determine the character's speed, plus a little to account for lag perhaps. To reduce server load, I think EQ2 just has a single value of maximum speed attainable and to allow for expansions without editing that code, that number is quite high. So high that people can speed-hack and run extremely fast but the server says it's fine.
So there's definitely a problem with having the client track it's own object and simply request acknowledgement from the server, especially since calculating if the move is valid would be just as expensive as calculating the move. The difference would, of course, be that instead of sending co-ordinates, it would just effectively send an "OK". That might be OK but to be honest it wouldn't save much since you have to send the I was thinking of effectively using a feedback loop. Client moves, client sends moves to server, server moves the client's character, server sends new position back to the client. It also saves the headache of what to do when the client receives an "invalid" message and it'll be sending the position of all objects that have moved since the last frame sent anyway, one more won't hurt. The client's job is to make sure it runs smooth with the info the server gives it, not to actually control anything or send its position to the server at all. Although I'm thinking that will be possible too with your API.
I'll give you some feedback when you document it but I've been leaning toward writing an object for a frame of data and just using an ObjectInputStream/ObjectOutputStream. The frame could be a dynamic data structure containing data on each of the objects the server is tracking which are in range of the client. An easy way to determine which are nearby is to use a grid system. The world is divided into a grid. The server sends info on objects from your current square of the grid and those around that. There would be a small spike when moving across grid lines but if loading this was kept to a separate thread and connection-usage for loading was capped, that wouldn't be too bad and the draw distance can be lowered enough to prevent objects appearing out of nowhere.
Of course I'm just throwing some ideas around at the moment.
Nyphur, perhaps you should tell a bit more what type of game you are making.
If I get it right, so far you suggest a system where the server is completly authorative, however the players local position can vary to prevent the experience of lag, and updates are frame based. That's fine if in your game:
- there's no aiming like in most FPS
- the response time to events happening is not critical.
So it could be good, eg. for a game where you're wandering around, and there's turn based combat between players.
If you do include the elements mentioned, this is why your approach would lead to bad experience:
Aiming is impossible because the client is completly non-authorative, there is no way to correct betweent the "client" position, and the "server" position. Also, server-to-client lag is felt as real lag to the player. So more lag means aiming gets a lot harder (think Quake1 and 2). This is only enhanced by the non-authorative client. Client to server lag also suffers from this, however if you chooise to use a frame based approach, that can be reduced, however it opens the way for very easy cheating. You can literarly "slow down" the world by reducing the frame update rate (and blame it on lag). Also aim bots become very easy.
Even without aiming, if response to events is important (eg. two players see each other and the one to click the other first has some kind of advantage) you still have the same problems.
This is why modern games with FPS related elements no longer use these approaches. That's why they use prediction on the client, to try and predict what the position on the server is, and try to compensate for lag clientside, and the server usually allows them some "authority" to play around with that's hard to cheat with.
Nyphur,
Actually, I agree with your problem and have already dealt with it in the networking synchronization system. I don't send a "denial" back to the client if their move is outside of the valid moves. The server simply ignores the request but it triggers the server to send back a synchronization event to the client ONLY if they send invalid synchronization information down. This solves the problem of consistently synchronizing back and forth. The client can be responsible for its objects and send down synchronization messages and the server has a method that gets called asking if this is a valid request. If it is then it applies it in the server world and goes on, if it is not then it doesn't get applied and the server then sends a synchronization message back up to the client to re-synchronize it. This way the server can always maintain authority, but does not micro-manage what the client is doing. This also reduces the effects of lag.
darkfrog
llama said:
Nyphur, perhaps you should tell a bit more what type of game you are making.
If I get it right, so far you suggest a system where the server is completly authorative, however the players local position can vary to prevent the experience of lag, and updates are frame based. That's fine if in your game:
- there's no aiming like in most FPS
- the response time to events happening is not critical.
So it could be good, eg. for a game where you're wandering around, and there's turn based combat between players.
That's exactly the type of game I'm going for. There'll be no aiming, instead you'll have to target a player by selecting them on your client and it simply sends the action and the name of the person you're doing it on rather than trying to caculate vectors, which is very messy. It's not an FPS, it'll be your standard fantasy MMO. Since the draw distance and thus the selection distance will be bigger than the range of any ability in the game, even if pvp were implemented it wouldn't be an advantage to be ahead of where the server sees you at all. Effects on other players wouldn't take effect until they do so on the server anyway. Slowing the game down by skipping or slowing frames would only be a disadvantage.
darkfrog said:
Nyphur,
Actually, I agree with your problem and have already dealt with it in the networking synchronization system. I don't send a "denial" back to the client if their move is outside of the valid moves. The server simply ignores the request but it triggers the server to send back a synchronization event to the client ONLY if they send invalid synchronization information down. This solves the problem of consistently synchronizing back and forth. The client can be responsible for its objects and send down synchronization messages and the server has a method that gets called asking if this is a valid request. If it is then it applies it in the server world and goes on, if it is not then it doesn't get applied and the server then sends a synchronization message back up to the client to re-synchronize it. This way the server can always maintain authority, but does not micro-manage what the client is doing. This also reduces the effects of lag.
darkfrog
So that means the server is only resynching with the client when it makes a bad request and assumes it is correct otherwise? That makes sense, I could definitely use that to determine if the client is too far ahead of the server (lots of lag) and slow the character down. Since the client can pretty much do what it likes with the synch message, that would work fine. It would save on uneccecary calculation and badwidth usage, too. I can see you've put a lot of thought and effort into this! There is still the small problem of speed hacks but in this case it would be because of lazy programming. I've got a system in mind to stop that without having to put too much effort into calculating the client's possible speed in any given instance. I'd basically have an event when equipping any gear or using any ability which modifies things like speed which alters a sngle value for the character's speed so only a single call is made when checking character speed rather than checking all the gear. But all that is beside the point. Your system sounds useful, I'll definitely give it a shot when you're done. I'm still in the design stage so it's perfect timing for me to help with feedback.
What do you think about separating Input, graphics rendering and loading into their own threads? I'm not even sure if that's possible, I've never worked with threading before, but I am a little sick of games stalling while they load models and textures as they come into view and input being disrupted and slowed by low FPS, which usually means I can't type capital letters with the shift key since I let go of the shift key before the letter appears on screen.
Well, keeping things in their own logical threads makes a lot of sense. Definitely you should keep as much out of the graphics thread as is possible, but there are some things that just can't be moved out of there. The networking API actually maintains two threads that deal with the different aspects of the networking (1. Updater - sends messages to the client or server respectively, 2. Listener - waits for incoming synchronization messages). An AI system would probably want to run in its own thread as well. Anything that would cause any amount of processing time to execute should be outside of the graphics thread if possible in my opinion. This is the same ideology for graphical applications as well. The same potential problems exist in Swing apps. You'll often see the bad thread programming by developers in Swing apps when you see the graphics thread get stuck on something and causes the screen to not refresh. Obviously this creates a bad UI experience for the user.
Anyway, I have got my first test finished that utilizes my physics networking API and have just started on a simple game that will utilize this as a proof of technology. Once this is finished I will be going back to the drawing board of this API and re-thinking the entire design as I've already found some serious flaws. This was moderately a hack to just verify functionality and I need more than just myself when I get to the point of creating a useful physics networking API for various game types. This may or may not be something your interested in, but either way I still appreciate the feedback.
Thanks,
darkfrog