Hi, lately i’ve been addressing some serious memory concerns for the demo of my game, and its been a pain. Namely the memory handling hygiene leaves a lot to desire, but the main problem is:
for context, i spawn 15k crates with fields:
public class Crate{
Long id,
Node node,
Item[] items, //empty for the sake of testing
String name;
}
now when i remove the reference to the crates they are not garbage collected (i’ve overriden the finalize method to print when they are collected)
so the stats are as follows:
a) on game startup (15k live crates) ~600MB memory taken
b) after removing references (~600MB, as expected)
c) after some time after removing references, when gc is naturally invoked, it doesnt perform a Major gc and leaves the “dead” crates in the memory (uses Minor collection and only cleans the latest trash).
When i use System.gc() after removing the references, the memory goes back to 50MB which i’ve confirmed is the minimum the game data needs.
this causes my game to consume about ~5-10x more memory than it actually needs.
I am completely lost and would really appreciate any help from you guys
The JDK will generally not collect garbage urgently unless it needs to (e.g. if it doesn’t need the memory yet). Do you apply any memory pressure or just let it sit with nothing to do?
Are you experiencing issues or just looking at these stats and being worried by them?
the thing is that when another big allocation happens, the jvm requests even more memory from the system rather than cleaning all those dead objects from the pool, and it causes both:
a) insane memory consumption
b) heating (im using a weak laptop for the development and i am optimizing the game to consistently need less than 1GB memory)
edit: so from what you say (and from what i noticed from the profiling) i have to live with jvm taking up a lot of memory at will? For now it seems that the JVM prefers taking memory from the OS rather than cleaning it
The JVM will (by default) not ask for more than a quater of the machines total memory. What happens then? Do you run out of memory and crash or does it reclaim it? (That will answer if this is truly a memory leak).
Using lots of memory in and of itself shouldn’t generate heat (initially allocating it would, but just letting it sit there won’t). Sounds like your application is using a lot of CPU (or gpu). Using a profiler might answer what its doing if its the cpu
P.s. these 15 thousand crates arent seperate geometries in your scene right? GPU is going to have a real hard time rendering that
a) ran a test with 15k crates (each with 3 items inside)
the heat was caused by serialization of delete object messages (not batched yet)
the JVM allocated 900MB at the beggining, deleting references (and sending them over the network caused a spike to 1.5GB/1.9GB which is in fact 25% my memory)
at 1.7/1.9GB Major gc was performed returning to 0.5GB/1.54GB
But now that the maximum memory is 1.54GB after first garbage collection, the garbage collector waits until about ~1.4GB worth of trash is there in the memory rather than giving the unnecessary memory back to OS and just performing the collection more often. When i invoke System.gc() the memory is 50MB/140MB and the garbage collection happens at ~120MB (thats the behavior i want to achieve)
So to sum up this issue:
causes long garbage collections
Makes my game take up waaaay too much memory
and as of what i understand the issue is:
a) when a Major GC happens, it doesnt get all the dead objects
b) then in a normal game loop the major gc doesnt happen ever again, only Minor GCs that clear young garbage – meaning that old garbage pollutes the memory and there is never a chance for it to get collected apart from me invoking System.gc()
The JVM has to trade off performance and memory usage. It isn’t going to balloon the memory for no reason (it will GC before filling up all available memory if memory usage is gradually building) but if it is available and it makes sense to use it rather than GCing it will.
If you truly are memory constrained and other processes need that memory then tell the JVM that with JVM arguments. But if the memory is just going to be left empty it is better to let the JVM have it so that it can better manage a large amount of memory
causes long garbage collections
I strongly suspect your problem isn’t hanging on to too much memory here but that you are creating way too much garbage. Why is your process generating so much garbage; investigate that first.
Makes my game take up waaaay too much memory
If your game wasn’t using the memory would some other process be using it, or would it be resting empty. Empty memory is wasted memory
b) then in a normal game loop the major gc doesnt happen ever again, only Minor GCs that clear young garbage – meaning that old garbage pollutes the memory and there is never a chance for it to get collected apart from me invoking System.gc()
Is this causing you an actual problem? The JVM usually knows what it is doing when it comes to memory/cpu time trade offs
What is your game actually doing? Why are there 15 thousand crates?
If your game wasn’t using the memory would some other process be using it, or would it be resting empty. Empty memory is wasted memory
assuming that we cant really assume anything about the machine of the player, the less memory it’d take the better
Is this causing you an actual problem? The JVM usually knows what it is doing when it comes to memory/cpu time trade offs
Yes, thats the cause - large “batch” allocations expand the memory and the gc doesnt care to clear old dead objects - from this point onwards it just uses the pool it was given rather than the JVM shrinking it.
What is your game actually doing? Why are there 15 thousand crates?
It was both
a stress test (in a real scenario there would be at most 5 thousand objects, most of which need less memory than a crate, thats why i picked it for testing)
a test of memory handling on both the client and the server during clearing level data (what gets kept, to spot some references preventing gc in case there were any etc)
but now that i did some more profiling, and from your answer i figured i should
avoid massive allocations that would overly expand the pool (in case the players computer has enough memory - it will simply eat more, in case not there will be multiple garbage collections mid execution)
reuse short-lived objects, so that the JVM has some time to decide to shrink the memory pool it has taken
as a last line of defense, i could invoke system.gc() to perform a full gc during level loading
The JVM sees a lot of free ram in your computer so it prefers to use it rather than running the garbage collector.
If you want to control this investigate the parameters, there are a lot you can use.
For instance limit the heap to 1gb with: -Xmx1g
But this is only the heap, there are other areas the jvm allocates memory, so look for more params in try them out.
Another param you can set is -Xss:256k to make smaller thread stacks (the default is 1MB and that usually not needed).
You can also change the GC if you don’t like the default, for instance experiment with the newer ZGC with -XX:+UseZGC.
This will backfire. The shorter lived the object, the easier it is for the GC to free it. Extremely young objects get freed almost instantly and essentially for free. Caching objects just to fill them in with completely different data later is often detrimental.
Note: if you are looking at the Windows task manager to get “how much RAM is my app using?” then the picture is quite a bit more complicated than the default RAM field that task manager is showing. (The default task manager fields are routinely showing my computer using 10x the amount of physical RAM that is in the system.)
Only a Java profiler will give you real valid data… or turn on maxmimum fields on the task manager and do a lot of reading. Java profiler is easier, though.
If these objects are all JME geometries (or contain JME geometries) then you’re probably going to have a bad time. That’s a LOT of scene objects.
You can also change the GC if you don’t like the default, for instance experiment with the newer ZGC with -XX:+UseZGC.
Something we ran into at work with our JVM microservices, was the Humongous Objects problem with G1GC. It may/may not be at play here.
We have a common pattern of storing maps/lookups in memory for runtime performance, and reloading when the underlying data changes. However, the old objects weren’t getting garbage collected, and we were blowing the heap.
As others have said, the JVM will generally prefer to take more O/S memory for the heap, rather than do a costly GC.
I wrote a test script to loop creating large objects and observed the results in VisualVM.
The default G1GC GC would keep requesting more RAM from the O/S and growing the heap as new objectes were created. But once the heap had reached the max allowable size, instead of garbage collecting the humungous objects, the JVM blew its heap.
With ZGC, JVM also grow the heap rather than GC as new objects created, but once it reached max heap size, would run a GC to create space for the next allocation. It could quite happily run this loop forever.
This has made me wonder if G1GC is the best default GC for JME?
That’s probably useful information. Sounds like you guys went through the appropriate steps to find the actual problem and then determined a solution.
I feel like OP might still be in the “find the actual problem” phase if they haven’t been in a profiler yet.
As an opposite anecdotal case, Mythruna creates and destroys a “shed load” of objects/meshes/etc. frequently and GC seems to keep up and I never run out of RAM. So it may depend on other factors.
I feel like OP might still be in the “find the actual problem” phase if they haven’t been in a profiler yet.
most likely, yes, though i’ve been profiling it for 2 days now with java runtime api
GC seems to keep up and I never run out of RAM.
Its not the fact that the game ever runs out of RAM as it doesnt, but i dont like the fact that when the game needs only 60MB of space after being cleared it still clings on to the memory taken from the OS
As an opposite anecdotal case, Mythruna creates and destroys a “shed load” of objects/meshes/etc. frequently and
could you give an example of what you mean in numbers? What i mean by “bad memory hygiene” is that on avg my app allocates ~1.5MB per second (90% of allocations are new network messages). Also, upon further profiling i’ve noticed that JVM gradually (about every 5 minutes) gives back the memory (after around 15 mintues the game is back to 50MB/120MB).
Also, how do you reuse message instances or do you instantiate new ones whenever needed? I know i have some issues to be resolved, such as actually checking if a change occured before updating the client with new state.
But it was for the person to which I was responding… and I was just pointing out that ‘running out’ may be contextual. It was my of pointing out that their solution might still not be appropriate for your case.
Every time I place or remove a block, I regenerate the mesh+geometry data. The old one gets GC’ed naturally.
Also, for networking there are literally thousands and thousands of objects created per second that get GC’ed almost immediately.
If you mean network messages then I instantiate new ones whenever needed. Holding onto objects you don’t need just so you can fill them in with new data later is not a good practice (frowned upon since Java 1.2 actually… but especially with modern garbage collectors). It messes up the ability for the JVM to manage its heap efficiently when you keep objects around to use for unrelated things. And there really isn’t any good reason to do it.
Cache objects that are expensive to create. Don’t cache anything else. (Even just the cache logic is likely to outweigh any gains in the latter case.)
If you are doing real time network syncing, then SimEthereal can take care of all of that mess for you… and in a super compact way. It only sends changes, has zone management built in, etc… all on top of a reliable UDP-based protocol (read: fast).
I’m late to the party here, but personally I view this as a feature. Requesting/releasing memory from/to the OS is quite expensive relative to anything else the JVM will do with a chunk of memory. The JVM assumes that if you used a chunk that large once, it’s fairly likely you will use it again soon so it hangs onto it. For example, if you use a 1GB chunk and then it frees it to the OS and then you use it again and then it frees it again repeatedly, this would result in making numerous system calls that are ultimately unneeded, consuming CPU (and on a mobile device, battery power) that could be avoided. It would also be significantly slower than letting the JVM hold the memory and manage it itself, so the JVM tries to optimize this by holding onto memory that it thinks it might use again soon. Personally I think this is the right tradeoff, and it’s not even one that’s ever going to be noticeable to users unless they’re trying to run a game alongside a bunch of other memory-hungry processes - in which case, the whole system will be slowed down if RAM is exhausted due to page swapping. I don’t personally have a problem with telling users that they might not be able to maintain performance of the game or their system if they max out their RAM with other processes while the game is running.
I’m late to the party here, but personally I view this as a feature. Requesting/releasing memory from/to the OS is quite expensive relative to anything else the JVM will do with a chunk of memory. The JVM assumes that if you used a chunk that large once, it’s fairly likely you will use it again soon so it hangs onto it. If you use a 1GB (for example) chunk and then it frees it to the OS and then you use it again and then it frees it again repeatedly.
^^ This right here.
Although, due to increasing popularity of cloud computing / containerized environments / kubernetes etc where people are paying for resources used there was a change to the default G1GC in JDK12 onward to make it more aggressive in releasing unused heap memory back to the O/S.
Shenandoah GC I believe is even more aggressive in releasing memory, but I’ve not yet experimented with it…
