[SOLVED] Spider Monkey message size limit?


#1

I am trying to send a large message over spider monkey, and just ran into an interesting error.
I am wondering if there is a message size limit?

java.nio.BufferOverflowException
        at java.nio.Buffer.nextPutIndex(Unknown Source)
        at java.nio.HeapByteBuffer.put(Unknown Source)
        at com.jme3.network.serializing.serializers.ByteSerializer.writeObject(ByteSerializer.java:51)
        at com.jme3.network.serializing.serializers.ArraySerializer.writeArray(ArraySerializer.java:124)
        at com.jme3.network.serializing.serializers.ArraySerializer.writeObject(ArraySerializer.java:109)
        at com.jme3.network.serializing.serializers.FieldSerializer.writeObject(FieldSerializer.java:202)
        at com.jme3.network.serializing.Serializer.writeClassAndObject(Serializer.java:457)
        at com.jme3.network.base.MessageProtocol.messageToBuffer(MessageProtocol.java:73)
        at com.jme3.network.base.DefaultClient.send(DefaultClient.java:282)
        at com.jme3.network.base.DefaultClient.send(DefaultClient.java:238)

My message simply contains a String and a byte[], but the byte[] contains 271KB of data. I would like to be able to send much larger messages if possible.


#2

You need to split the messages up. There is a 32k limit.


#3

Ah, that sucks. Is there any way to bypass or increase the limit?


#4

The limit comes from the way Spidermonkey sizes its internal buffers. To my knowledge there’s no external API for changing that.

API aside, you don’t want to just toss large messages at the wire - over UDP, they will need to be split into 64k packets anyway, and larger packets are more likely to be dropped. Over TCP, you’ll have large messages doing head-of-line blocking of following messages, which is undesirable for many circumstances. I’d implementing a lightweight message splitting mechanism.


#5

No. It’s a limitation in the design of the serializer. We have to always preallocate for the biggest message possible no matter what size the actual message is. So if you are sending a 5 byte message, we create a 32k buffer for it. If you’re sending a 31k byte message, we allocate a 32k buffer for it.

It’s a compromise.

One of my biggest mistakes in life was not rewriting the irresponsibly designed serializer when I rewrote the rest of SpiderMonkey way back when. I was so strongly assured that it was “very efficient” that I was afraid to mess with it.

Anyway, splitting messages is better overall for the general throughput of your channels… and pretty simple to implement.

Also, depending on the kind of large data you are streaming you should consider whether a game networking API is really appropriate. For example, an HTTP server may be more appropriate for certain kinds of content and Java can trivially stream content from HTTP.


#6

Alright, thank you! I wrote a system for breaking up my large messages before they are sent out. I probably need to do it over http as I already have a http server as part of the server system, but it is not a common thing for the client to send large files to the server, so I will stick with this work around for now.