Sorry you keep bumping into problems with the code.
I’m not sure what’s happening but that limit comes from:
long bitsRemaining = (bufferSize * 8) - estimatedSize;
I haven’t drilled farther than that… but I guess estimatedSize is larger than buffer size. Which without looking deeper implies that something didn’t split when it should have.
But you know just glancing through this code, I see a couple of things that trouble me:
bitsRemaining is never recalculated in the loop… which seems wrong to me but it may be there is a reason and I just didn’t leave a comment.
probably harmless, but FrameState’s getHeaderBitSize() seems 64 bits too small.
I’m about to go to bed. Sorry you are having to debug my code for me. Hopefully it’s a net win in the long run. Thanks for doing the leg work.
I have no problem helping you harden the code. I am using it in an opened world true scale space game after all lol. I spent all morning on it and this is how far I got. It looks like it’s a bug in the split code. Everything is pretty stable with frames that get spit only a few times but things to wacky with very large frames on the split code. I confirmed this by increasing the mtu ten times and stability was returned. I will create a simple test server and simple test client that will reproduce things for you and will give you a few throttling parameter to play with. Once it becomes stable I can try it over the internet. I have monster servers on a 10 gig backbone in Texas and servers in Australian for a good latency test that we can run it on in full throttle.
By the way, I will definitely have to add distance culling based on the size of the spacial in the camera’s frustum because in my scene I VERY LARGE object (planets) and VERY SMALL objects (ships) in the scene and as you can imagine, there are thousands of ships for each planet in a grid
a solar system can be seen up do 5k units away … of course you are plety much seeing the sun and planets are small dots … but the sun is big … This is 2500 units away from the sun.
You aren’t sending the planets using the networking, though, are you? That seems like it would be wasteful given that their paths are completely predictable.
…so then your zone grid only needs to worry about ship visibility. There is a performance balance between grid cell size versus grid radius (ie: how many cells around the player you show.)
Only the rotation of the parent orbit node is transmitted … stars get culled around 10K units … ships will fill up all of space … I still need to transmit rotation of orbits 10k units away. Planets MUST be synced because … some are land able. I know … so I went overboard with the game lol
Now you know why I will be culling ships at the server when you can not see them anymore … that alone will drop my entities down by at least one order of magnitude. Are frames transmitted per zone or per viewable radius? Another alternative is to have two parallel instances of your server. One that transmits planets at the 5000 units zone scale, and one that transmits ships at a much smaller zone scale. Either way I cut it I am still looking at thousands of entities view-able in the worst case due to clumping
Per zone. Network visibility is managed by the zones you have around you.
At any given instant of game time, a planet’s location and rotation is completely predictable from some parameters. There is no reason to network sync these and it totally screws up the zone sizes to do so. Even if you didn’t want to do it that way, planets could be interpolated with one second, five second, 10 second, or even larger update sizes and you’d never lose accuracy. The only reason you’d have to sync planets is if landing on them caused them to move.
Yes… and then just don’t do the planet one. As I said, planets are not your normal physics object since their paths are 100% predictable years in advance. You could sync them using regular networking waaay less often and way less efficiently (and steal the current game time from what SimEthereal thinks it is)… then just interpolate the curves from there.
You are correct. When the player enters the game I can give them back the current position of one planet and that is enough information for the client to position the entire universe from the games previous saved state. Genius!
Now if I only knew how to take the 200k+ stars in my database and create a relative real time skybox at runtime that would be killer! Any ideas??
P.S. I still want to fix your split bug but I can now do it next week
You aren’t doing this at all, now, right? Sounds like something that needs offline processing and heavy compression. Else without knowing more about the data, I can’t say.
Cool. Yeah, there is definitely something dodgy there so I look forward to your test case. I think I know at least one thing wrong but I’m cautious about touching code without fully loading it into my brain.
Be glad that you don’t really have a real true scale game - otherwise the outer space would be an empty and uninteresting place (planets look like stars when seen from each other - and even the sun is very small unless you are really close - in which case it is so huge that it covers the full screen while your ship is like a star or little pebble in relation to that).
For the 200k+ stars and the skybox: divide those stars into “near to the player” and “far far away” (where the Orion ogres live together with their red haired mother).
The near to the player you tread as a user-controlled particle system and update quite often and also when the camera moves.
The far far away stars you render to a cube texture or double-dome of highest possible resolution and reuse them for like 10000 frames or so - and every 1000 frames you render one side of the texture and do not move these far far away stars. Every frame you calculate 20+ stars, but don’t update their position. You calculate all of the stars with the same time span (i.e. the approx time it takes to render 10000 frames).
That’s how I would do it.
That or the new OpenCL binding that just got released.
Or both - to let them compete.
Btw, a very nice topic. I often thought about “how will I simulate 10,000 objects with 100+ constantly moving?”. I’m still a networking noob and have to learn a lot from others or time-consuming experiments. So, thanks for the material found here.
I solved my star problem today while I switched my brain away from @pspeed’s code. I got the idea from @pspeed’s network code and how he deals with zones and it dovetails PERFECTLY and solves both problems at once.
Here is the breakdown:
Coordinate system:
Keep in mind it’s just the current scale … I can now go VERY big or VERY small if I wantt.
1 unit is 500 kilometer
1 zone = 500 y units, 500 x units, 500 z units
User’s view-able radius is 10 zones in all directions (the network layer will transmit cords for entities in that area.)
NOTE: You can only see stars 10 zones away. Planets are too small.
The Rendering:
Objects within 10 zones of the player are rendered normally.
Objects 11 - 100 zones away … basically ONLY stars … are rendered on an off screen buffer cube map so that I keep all the details
Objects 101 - 1000 zones away are rendered are particles on the same off screen buffer cube map
Objects over 1000 zones away are culled
My entity management system is double precision even on the client … So I can do cool stuff like highlight a star on the hud even though it is on the cube map.
@pspeed I have a noob question … Only because I have never had to do this before. Your PositionTransition has world based rotation and translation data. How the heck am I so-posed to convert it to my client’s spatial’s local translation and rotation? Before you say “Your client spatial should be attached directly to the root node then it’s no issue” then is a valid reason why they are not.
EDIT: Never mind! I had a blond moment because it’s 1am … I just have to send the local translation and local rotation from the server. I AM AN IDIOT!
@pspeed ok so I have been hammering at your network code all week non stop getting it ready to release to my use community in my game. Fixed a few bugs and found out why my network traffic was so high. The split bug comes up ALLOT! So I temporarily coded around it so that when it can’t split a frame it just sends the whole frame instead of dumping out. I will post the details later on but the patched version passes ALL my load/stress/ and non happy path tests now. I have been up debugging for 30 hours straight … time for some sleep now.
@pspeed the code has been in production for 2 days. About 800 simultaneous users at peak times. It has not blown up yet and performs well under both tcp and udp. This is what I changed / fixed so far.
FrameState.java split() method hack to temporally get around the split bug.
long size = getHeaderBitSize();
int split = 0;
while(split < states.size())
{
ObjectState s = states.get(split);
int bits = protocol.getEstimatedBitSize(s);
if(size + bits > limit)
break;
size += bits;
split++;
}
if(split == 0 || split == states.size())
{
// throw new RuntimeException( "Error splitting message. split:" +
// split );
// Zissis HACK!!! Somehow estimatedBitSize turnes to negative with
// really large frames.
return null;
}
long leftOverBits = estimatedBitSize - size;
StateWriter.java startMessage() method to fix a problem when the client has a temporarily very slow outbound connection and when it goes back to normal it floods ack packets.
protected void startMessage()
{
if(outbound != null)
{
return;
}
// Just in case we'll put a watchdog in here. The reason this
// is unlikely to trigger is because the receivedAcks set only
// grows when we've received a message from the client. And at
// that point we get to remove every receivedAck the client
// confirms. So one new ID should always nearly empty the set.
int size = receivedAcks.size();
// Zissis commented this out because it blows up the state collector
// when a client is on a bad outbound connection
// if( size >= 128 ) {
// throw new RuntimeException( "Very bad things have happened in the
// receivedAcks set." );
// }
// Build the ACKs array
if(receivedAcksArray == null)
{
receivedAcksArray = new int[size];
int index = 0;
for(Integer s : receivedAcks)
{
receivedAcksArray[index++] = s;
}
}
this.outbound = new SentState(-1, receivedAcksArray, new ArrayList<FrameState>());
this.headerBits = outbound.getEstimatedHeaderSize();
this.estimatedSize = headerBits;
}
There is one bug that I have to spend more time on. When I set the view-able extends past 1 … for example 3,3,3 … If an entity is located at 0,1,0 the frame is tagged as if it’s at 0,3,0. For now I have a HUGE grid and set the view-able extends to be 1,1,1
I will keep you posted if anything new comes up.
By the way, here’s your code in action with over 600 players in the game at the same time. I am looking at a solar system from over a kilometers away
Do you mind if I add a few upgrades to the code? This is what I am thinking:
When translation AND rotation doesn’t change in a frame just send a super compressed ObjectState message with just 1 bit
Viewing extend is set on a per connection basis this way I can have a connection for object groups that cull FAR away like nebulas etc, and a connection for objects that cull at close range like ships
Extend the Protocol so that you can configure rotation to be sent as Quaternions or a 3 float angle on a per connection basis. This will reduce bandwidth and in some cases increase precision for certain entities. You would of course configure the Vec3Bits used for angles separately.
Allow a connection to override the default protocol in terms of bit size upon connecting. This will further optimize bandwidth. For example, since my ships will work off of a very small grid size I can use a small amount of bits and the huge grid size entities that are on another connection can use large bit sizes.
Extend the Protocol do be configured to only send rotation or only send translation.
1 bit would only save two or three bits, I guess. You’d still need the IDs and there is only one bit per non-changing field after that.
But in this case the far away objects do not require networking synching, yes? In Mythruna, I will need two grid sizes and I will have to figure out how to support that. I guess there is nothing wrong with allowing per connection extents but I worry it starts to complicate some code that can make some constant assumptions. Maybe not.
It works out to be the same, I think. I think for euler angles you will require a higher precision… I wouldn’t do less than 16 bits, personally. Which works out to be the same number of bits because a quaternion only needs 12 bits each. This makes sense since euler angles have a bunch of redundant ranges.
I always saw this as two separate grids, personally. In Mythruna I will have large objects like water and air ships… these will require both a larger grid size and different extents… versus the items which are using a 32 meter grid with only radius 1 extents.
Again, if rotation or location are not changing then those are only one bit each. This doesn’t really save you much, I guess.
The multi zone feature is not just for far away objects … remember, my game is simulated true scale. A small fighter can’t be seen part 100 kilometers but a massive battleship can be see well over 300,000 kilometers. Battleships will have a different zone size than fighters. Otherwise I will be transmitting tens if not hundreds of thousands on entities just to make sure battleships can be seen far away. You do not have that problem in Mythruna.
10,000 entities in a zone in my game … saving 1 bit per entity and 20 ticks = 10,000 bits per tick … or 195.3125 kbps … That’s a considerable amount of bandwidth. It all depends on your game mechanics and use cases. In my case, sending 1 bit instead of 2 for stationary objects is a huge savings in bandwidth.
Yes they are two separate grids but you want larger repression for your large grid versus your small grid so two different protocols for optimal optimization.
