[Solved] BufferOverflow while sending ~26368 bytes

Hi,



I would like to send my gameworld (Mythruna block style) from my server to my client.

So I have a chunk containing a 1616128 array of bytes (containing the block-type) + 2 booleans + 2 integers that I would like to send in one message.

That should be 32778 bytes.



In the SocketConnector class from the jme there is the following declaration:

private byte[] buffer = new byte[65535];

So I thought that sending 32778 bytes would be possible.

But I get every time a BufferOverflowException…



So I tried to find out where the map limit is that I can send in one message and I found that this is a 1616102 array. That would be approximately 26122 bytes. When I try to send a 1616103 (26378 bytes) array, I get the exception.



Now I have three questions:

  • Is the size limit of a message still 32kb like I read in an older thread?
  • If that is true, is there really about 6kb overhead in my message that I can’t localize?
  • Do I need to split the message into two messages, or is there another already implemented way to solve this (like compress the message)?



    I hope you can help me :slight_smile:



    Greetings, Daniel

Compression of messages “live” is easy from Java. For example here’s a class I wrote a few years ago that compresses any payload passed into it.



[java]

public class CompressedMessage implements Serializable {



private static final long serialVersionUID = 1L;



private final byte[] payload;



public CompressedMessage(Serializable ob) throws IOException {

if (ob == null)

payload = null;

else {

ByteArrayOutputStream baos = null;

OutputStream os = null;

ObjectOutputStream oos = null;



try {

baos = new ByteArrayOutputStream();

os = new DeflaterOutputStream( baos );

oos = new ObjectOutputStream( os );

oos.writeObject( ob );

oos.close();

payload = baos.toByteArray();

} finally {

if (baos != null)

baos.close();

if (os != null)

os.close();

if (oos != null)

oos.close();

}

}

}



public Serializable getPayload() throws IOException, ClassNotFoundException {

if (payload == null)

return null;

else {

ByteArrayInputStream bais = null;

InputStream is = null;

ObjectInputStream ois = null;



try {

bais = new ByteArrayInputStream(payload);

is = new InflaterInputStream(bais);

ois = new ObjectInputStream(is);

return (Serializable)ois.readObject();



} finally {

if (ois != null)

ois.close();

if (is != null)

is.close();

if (bais != null)

bais.close();

}

}

}



public int getPayloadSize() {

return payload.length;

}



}

[/java]

1 Like

The limit is 32k. You can see that in MessageProtocol. I don’t remember what SocketConnector allocates 65536 but it’s one buffer per connector so it’s not to big of a deal.



If you are sending a 3D array, like bye[][][] then you are forgetting the data associated with each array entry. If you are sending byte[16][16][128] then there are [16][16] byte[] arrays that could be any length or null. And 16 byte[][] that could also be any length or null, and so on.



The array serializer tries to be as compact as possible but it wouldn’t surprise me if there is that much overhead in a 3D array of that size. If you flattened into a 1D array for transport then there would be less overhead. Run length encoding also helps.



Mythruna flattens, run length encodes, AND gzips the data. An most of my 32x32x32 “chunks” fit in a message… and I split them into two when they don’t.

2 Likes

Thank you for your replies.

I have already implemented a basic compressed message class (very much like yours @zarch). I just wondered if this is really necessary.



Flatting the array is a good idea. That should do it for now.



Thanks,

Daniel

1 Like

hi,
i also try to send an object of 3 arrays (2x byte, 1x short all flatten). the max size is 97KB. i don’t know how to compress a message (gzip) and decompress. can you tell me a short code example? or how to split a message?

Tell you a short example? I posted one 2 posts above yours…!

i dont know how to use the Serializable object in that script and what about decompression?

the constructor compresses it, getPayload() decompresses it. Do what you want with the serializable…

…and the fact that you called it a script (combined with what seems like a general lack of java knowledge) prompts me to ask how much java experience you have? You will struggle to get far with jme3 without learning java first (although once you do know java it works very well indeed).

how would you call it? code, class, snipped … script seems to be the same to me
my java skills are basic and i try to understand how to use your class. you have a constructor and one method for decompression… but if the message is compressed, how do i get the payload? because its private

the payload is a byte array but what should i do with it?
please give me a short example how to use your class with my message

this is the message class I use:
[java]
import com.jme3.network.AbstractMessage;
import com.jme3.network.serializing.Serializable;

@Serializable
public class ChunkMessage extends AbstractMessage
{
private byte[] blocks;
private short[] lights;
private byte[] topLevelBlocks;
private int x;
private int z;
private Type type;

public ChunkMessage() // empty constructor required
{
}

public ChunkMessage(byte[] blocks, short[] lights, byte[] topLevelBlocks)
{
	this.blocks = blocks;
	this.lights = lights;
	this.topLevelBlocks = topLevelBlocks;
	type = Type.ResponseChunk;
}

public ChunkMessage(int x, int z)
{
	this.x = x;
	this.z = z;
	type = Type.RequestChunk;
}

public byte[] getBlocks()
{
	return blocks;
}

public short[] getLights()
{
	return lights;
}

public byte[] getTopLevelBlocks()
{
	return topLevelBlocks;
}

public int getX()
{
	return x;
}

public int getZ()
{
	return z;
}

public Type getType()
{
	return type;
}

public enum Type
{
	RequestChunk, ResponseChunk, RequestUpdate, ResponseUpdate, NotifyUpdate;
}

}
[/java]

Code works, in this case its a full class though so most people would just call it a class. Generally script refers to run-time parsed languages (i.e. javascript, etc).

The class is really simple to use:
[java]
ChunkMessage cm = …blah;
CompressedMessage compressed = new CompressedMessage(cm);
// Send compressed

// at other end
CompressedMessage compressed = //however you read it
ChunkMessage cm = compressed.getPayload();
[/java]

The protocol I was using these with I added some code at each end so when it received a message of type CompressedMessage it automatically retrieved the payload and then proceeded as though the payload is what had originally been received.

That way you could decide to compress or not compress whatever objects you like in order to send them and it was completely transparent on the other end.

It’s possible to use SpiderMonkey’s serializer to do compressed messages that are much more compact. But I already feel like the OP is in over their head. Networking a game is easily 100 times harder than compressing a message.

For what its worth I agree with @pspeed…

@alric - networking games is hard. Start with something simple and then work up from there.

And by simple I mean something like tic-tac-toe…even that involves a surprising amount of complexity.

thank you for your example :). but eclipse says:

[java]
public class CompressedMessage implements Serializable // The annotation type Serializable should not be used as a superinterface for CompressedMessage
[/java]
the implementation of an interface means that i have to implement all methods from the interface so i have some empty methods.

furthermore
[java]
ChunkMessage cm = new ChunkMessage(…);
CompressedMessage compressed = new CompressedMessage(cm); // The constructor CompressedMessage(ChunkMessage) is undefined
source.send(compressed); // The method send(Message) in the type MessageConnection is not applicable for the arguments (CompressedMessage)
[/java]

if i simply say:
[java]
source.send(new GZIPCompressedMessage(cm));
[/java]
it seems that nothing was sent to the client side :confused:

@zarch said: For what its worth I agree with @pspeed...

@alric - networking games is hard. Start with something simple and then work up from there.

And by simple I mean something like tic-tac-toe…even that involves a surprising amount of complexity.

it is my first try to send some objects over the network. what is simplier than that? if i never trying it, how do i get the knowledge to programm a simple network game?

The problem is that there is no such thing as a “simple” networked game. That’s why we are saying start off single player - then once you understand that you can move up to networked.

You are probably importing the wrong Serializable or something. It’s just a marker interface, no methods implemented.

I don’t use the jME3 networking so can’t help you about sending the message. What you see is what you get - you need to take it from there…but if you are just starting out why are you trying to compress the message anyway? That’s an optimisation and premature optimisation makes for sad monkeys…

i want to compress the message because of my first post. i am unable to send one of these arrays. the smallest is 32KB (to big for jme network)

You could also split them into multiple messages… and you may have to anyway. Even compressed they may come out bigger than the 32k limit.

problem solved, I encoded the array data by myself. the new size is ~3 KB + overhead + message container => far from 32 kb :wink:

1 Like