Question to pspeed about his bit stream class

@pspeed do you have an updated version of your bit stream class that includes floats with user defined precision?

Just to be sure we are talking about the same class, you mean this one:

In SimMath, there are some ‘bits’ classes that allow you to turn floats to/from bits based on a range and desired bit size. For example:

The ObjectStateProtocol class has an example using them to marshal floats:

I opted for this approach versus a ‘precision’ because a) that’s tough to guess what the bit size will be and b) often you want a specific range like only positive values or only values between -1 and 1 (for quats). There are similarly Vec3d and Quat bits classes for setting up FloatBits for the different components. I found it was frequently desirable to have different precision in x,z versus y, for example.

Hope that answers the question.

1 Like

You’re the best @pspeed! Thank you! saved me having to reinvent the wheel!!

You’re welcome. And you’ll have to take my word for it that getting this particular wheel consistently right took lots of testing. :slight_smile:

2 Likes

OK @pspeed I looked over your code and incorporated all the cool stuff for bit compression. REALLY NICE … but then, I saw SimEthereal. Read the code, got the general idea of how it works and was wondering if you ever got around to putting together a basic tutorial or some basic docs. If not, how much would it cost to bribe you to do it? Some very nice stuff in that library that is perfectly suited for my needs and the design concept of zones works perfectly with what I have at the moment.

Unfortunately, no. I have me test ‘game’ that I wrote to develop it but never got time to swing back and make it publishable.

Well, cost is not really the issue but time. I will see if I can write up a high level how-to some time today that might help an explorer such as yourself figure out the rest. I think there may be only a handful of important bits. (pun intended as per topic)

Ok, I have a few trapped minutes with my laptop and my source code so I will try to explain the ethereal fundamentals.

At a high level, an application needs to do a few things to set sim-ethereal up and make it work:
-run the server-side service
-run the client-side service
-server: send object updates
-client: receive object updates

Setting up the server-side service:

Basically, you need to add the EtherealHost to your SpiderMonkey network services on the server.

In my espace game, I have some things defined as constants to setting up the service looks like this:

        // Create the server-side service that will manage network sync
        EtherealHost ethereal = new EtherealHost(ESpaceConstants.OBJECT_PROTOCOL, 
                                                 ESpaceConstants.ZONE_GRID,
                                                 ESpaceConstants.ZONE_RADIUS);
        server.getServices().addService(ethereal);

Getting those values right can be tricky in that the bit sizes you specify in the protocol need to be big enough to support the resolution of your grid size.

Here are mine at least as an example:

    private static final int gridSize = 64;
    
    public static final ZoneGrid ZONE_GRID = new ZoneGrid(gridSize, gridSize, 0);
    public static final ObjectStateProtocol OBJECT_PROTOCOL 
                = new ObjectStateProtocol(8, 64, new Vec3Bits(-12, gridSize + 12, 16), new QuatBits(12));

Setting up the client-side service:

Similarly, to the server-side, you have to add the EtherealClient to SpiderMonkey’s client-side services.

In ESpace, mine looks like this (ignore the other services I’ve added):

        this.client = Network.connectToServer(ESpaceConstants.GAME_NAME,
                                              ESpaceConstants.PROTOCOL_VERSION,
                                              host, port);
                                              
        client.getServices().addServices(new RpcClientService(),
                                         new RmiClientService(),
                                         new EntityClientService(),
                                         new AccountClientService(),
                                         new ArenaClient(),                   
                                         new ChatClient((short)-2),
                                         new EtherealClient(ESpaceConstants.OBJECT_PROTOCOL,
                                                            ESpaceConstants.ZONE_GRID,
                                                            ESpaceConstants.ZONE_RADIUS));

Note that it uses the same constants… this is important. Both the EtherealHost and EtherealClient must share the same setup or they will get very confused.

It’s possible that in the future I may make client setup automatic.

Sending the object updates:

Once a physics frame, whatever physics engine you are using, you need to call some methods on the ZoneManager (accessible from EtherealHost)
-beginUpdate() with your game time. This locks the frame time of all of the updates you are about to add.
-updateEntity() called once per object to update its position and orientation.
-endUpdate() when all of the updates for that frame are complete.

Handling updates on the client:

Add a SharedObjectListener to the EtherealClient.

This object will be called with beginFrame(), updateObject(), removeObject(), etc. to update its local version of objects. The update is passed a SharedObject with the current state of the object and this can be transferred to your local objects however. I personally use time buffers that auto-interpolate and support lock-free multithreaded access. updateObject() is called from a different thread so you will have to make sure to handle that. I can provide an example of using a time buffer if you are interested.

I think those are all of the important bits. Let me know if you have any specific questions after playing with it and we can drill in further.

…my trapped-with-the-laptop time is just about up. Hope it helps. :slight_smile:

@pspeed you where referring to the PosittionBuffer for interpolation. I saw that and it’s cool enough for me to use. I have decided to integrate your code into my project. This will give you the ability to know how it works in the real world. I have 1500 beta testers across the globe. I will also be upgrading it to include some features that my current net code does like the server knowing about the camera cull distance and camera position so it does not send updates to clients if spatial are culled because they are too far and the ability to dynamically adjust the tick rate on a per entity basis based on clien’t camera distance. Oh and the ability to have multiple channels so that weapon updates do not impede ship updates. I am sure I will have a few questions about your code after I integrate it this weekend. I will keep my extensions componentized so that you can merge them into your library if you want to later on.

Keep in mind that this code only sends deltas and only for objects that actually move. It’s using a UDP protocol that properly handles all of that and makes sure the proper latest data makes it through so that deltas work.

So just be careful when skipping stuff on a per entity basis as it may actually defeat some of the other stuff that’s going on.

The goal of the library was to get as many object updates under the size of a typical MTU as possible so that the UDP packets are unlikely to be broken apart (as that affects their reliability adversely). Consequently, with the compression that is in place, I can fit (from my fuzzy memory) 70-80 object updates in a single 1400 byte data set. Even more if they aren’t moving. (My original test had x number of bouncing, rolling, cubes)

So it could be that you don’t need to do any optimization beyond that.

Also note, if you are trying to optimize at a low level. It does an internal buffering of frames and I think (fuzzy memory again) only sends real updates 20 times a second. So a particular message will end up with 2-3 frames of data for some objects… though it automatically breaks up frames and messages as needed. So for example, if you really did have 80 objects all rolling around then it might actually be sending one message per frame because the internal max message size was reached.

I look forward to seeing how it works in someone else’s game. My own testing was limited to be LAN and some fake performance degradation to test packet loss and stuff.

Edit: just in case, it seems like I should drop this link here until I get official docs up and can hide it there. :slight_smile:

My thoughts where to break up the scene into multiple server connections. When you connected you would specify the types of entities you where interested in. Planets, ships, weapons etc. I would have a server stack on your library running for each entity type and feed it appropriately. As for the distance culling, that would require a bit of tweaking of your library. Since my space game is in true scale, and since you can not really see the potentially thousands of ships and space stations scattered around just one solar system from any one vantage point, distance culling is critical for my needs.

I contributed to your patreon by the way. You have helped me many times in the past and this library is worth it as a great jump start. Keep up the great work!

@pspeed OK I started to reverse engineer the code since there is no documentation … it’s pretty easy to read. I have a questions … I understand how zones are defined and how I can tell the server what zone(s) I can view on the initial connection, and I also see how I can get the world translation from a shared object in a zone. The question I have is as the player navigates space in my game, I don’t see a way to update the server on my new location so the server can switch my position to the next zone once my viewing position crosses the initial zone. Am I missing something here?

That’s what the zones provide. You only see stuff in the zones around you.

Ahah! A very important part that I missed! And it’s a bit counterintuitive maybe because one might think “where on the client do I set this thing?” And the answer is “You don’t.” Because it’s on the server…

I initially looked all over my client code to find this before remembering that it’s inefficient to set it there so I must have done something on the server. Sure enough:

    // Now setup our listener for streaming data
    getService(EtherealHost.class).startHostingOnConnection(conn);
    getService(EtherealHost.class).getStateListener(conn).setSelf(ship.getId(), new Vec3d()); 

So, on the server I do that when the specific player is ready to actually enter the game. ship.getId() returns the object ID that represents the player’s location (it’s the entity ID in my case because I use Zay-ES). The second parameter new Vec3d() is the initial location.

Actually, I think what you are seeing is how you tell the server what relative zones that the player will see. Vec3i clientZoneExtents defines the range of zones around the player that the client will receive events for.

Hope that clears some things up. My sample game is a 2D top-down space shooter and I very much tested it with small zones and limited range just to see objects clipping in and out as I flew around. :slight_smile:

Thanks!.

Explains why my clients got ZERO updates even though the server was processing TONS of changes :slight_smile:

So basically if I understand corectly, after telling the server the initial cam location the server moves your camera for you and that’s how it knows where you are at all times? In my case the camera is not a real entity (I have an entity system as well) … The camera is a client controlled view. Is there any way for me to set the camera location initially without requiring an entity id and … can I update it grammatically on the server?

It might be easier just to create a non-visible entity for your camera view and then move that entity on the server manually instead of through physics.

I’m not sure what side effects you will have otherwise as the changing of the center zone has to be carefully interleaved with the other events that are happening since all of the zone IDs are relative to that center zone… and zone IDs are built into every object’s state. Get it wrong and everything will get very confused.

(The state messages do everything they can to reduce size and one way is to use zone IDs that are relative to the center. It does mean that as soon as the ‘main’ entity crosses a zone boundary that all of the zone IDs now mean something different to that client.)

I can set up a non visual entity id for each client’s camera … in fact, I use a collision shape on the client anyways. The last question them is does the entity ID for “self” on client a get updated on client b? That would be a waist.

Yeah, but it’s a really tiny waste. If it’s an object that moves and doesn’t rotate then it’s mostly only 9 bytes I guess.

True. Oh and before I forget … In my server update loop I call updateEntity() … the second argument “active” is not used in the code. What gives??

public void updateEntity(Long id, boolean active, Vec3d p, Quatd orientation, AaBBox bounds)

Probably a hold-over from the prototype stage. My physics engines tend to send a flag to tell me of objects are asleep or not. I may have transliterated that into the interface when making the zone manager and then forgot about it. It should probably be removed when I release for real.

1 Like

@pspeed if I read the code correctly, if I have a ZoneGrid of 5000, 5000, 5000 and the largest radius for any aabbox in my game is say, 100 … then the position bits for my ObjectStateProtocol can just be a range from -100 to 5100 correct? My assumption is that you transmit local coordinated to the grid and not world coordinate and local grid coordinated on all axis start at 0 and end at grid size for that axis … the -100 +100 is an extra sanity pad. Are all these assumptions correct?

Coordinates transmitted are relative to the grid, yes. You need some buffer/margin because an object is ‘in the zone’ if its radius intersects the zone at all. (ie: an object can be in more than one zone at a time.)

Edit: yes, basically your assumptions seem correct.

Edit 2: you haven’t said how big your grid cells are so I’ll have to assume you’ve done the rest of the math properly. :slight_smile: