[SOLVED] How to close a physics space to free up ram

Troubleshooting physics
1

new to libbullettjme, so sorry if my question is super basic

not using JmonkeyEngine, using the .jar and .so files from the github

I made a project that relies entirely on temporary physics spaces
codes functional, works way better then i expected.

but during stress testing i realized there was a memory leak as I’m unsure how to remove the physics spaces when I’m done with them to free up resources

its not a issue with removing rigid bodies, was already doing that
this is just from creating the physics space

1 Like
2

Physics spaces are Java objects. They get freed automatically (by the garbage collector) when they are no longer accessible. You might be able to speed up the process by invoking PhysicsSpace.destroy() and System.gc(). If that doesn’t work, look for persistent references to the spaces in your code. And if you’re using native physics (such as Minie or Libbulletjme) make sure the “Physics Cleaner” thread is running.

3

Bit of a weird/undocumented setup
the Libbulletjme code is being mirrored/reflected, so maybe its not acting 100% normal

I tried PhysicsSpace.destroy(), but I noticed its description is “Remove all collision objects and physics joints.” but i think its the physics space itself, not the objects in it.

from looping the start of my code the leak starts as soon as i create the local variable as a new PhysicsSpace(PhysicsSpace.BroadphaseType.DBVT)

Not sure if that means the cleaner thread isn’t being made to cleanup the physicsspace when it stops being used, or its being made and it fails to stop the physicsspace

4

Are you sure it isn’t be garbage collected? With the JRE garbage collector unused objects will be collected “at some point” but not necessarily immediately. Are you seeing a growing number of these PhysicsSpace objects over time

1 Like
5

So its running on a gameserver as a way to add physics without needing client mods
the way its currently setup it runs a gameworld and physics spaces separately

it creates a async thead that creates a new physics space every time there’s a trigger
the physics space adds the games local environment to the physics space as the rigid body moves
the game then maps the rigid bodies location/rotation to game assets

The physics spaces were planned to be disposable made and destroyed pretty regularly, existing for under 30 seconds and i didn’t care if the objects interacted with each other

while doing stability testing I realized that every physics space was using ram i didn’t get back, so over time it would grow until the server crashed.

CPU performance wise its great, it scales across cores, if there’s a explosion throwing 200 ragdolls in the air, the server didn’t seem to mind, the issue is 200 physics spaces worth of ram going POOF, i left the server running for awhile, and it didnt seem to clear up memory over time

If I cant clear up the memory leak i might have to try and make permanent physics spaces
First time dealing with Async code, so thread safety/limitations made it easier to just do everything i needed on the game thread ahead of time, then creating a Async thread for physics, and not have to send information back and forth

2 Likes
6

…because you loaded it into a profiler and checked it? Or because “after some time things didn’t clean up yet”? Or because “you started getting out of memory errors”?

Only the first and last are diagnostically significant. And the first would have given you lots of useful information for tracking the problem.

7

Going off of the pterodactyl Panel, servers running in a docker container
ram usage fluctuates as i join/leave/whatever

the profilers i have from the game side don’t show the ram usage going up, only the panel shows a increase.

I played with some of the java flags last night, and it looks like it might of helped
its taking a few minutes to do GC and its ram useage is still way higher (like GBs over before i started spawning physics spaces) but it does seem to drop down to a certain level of ram usage.

but yesterday it didn’t seem to drop atall after starting them, but maybe my ram usage wasn’t high enough

game profiler still thinks I’m not even using 3GB, but panels showing 11GB range

after a half hour its still 11.6GB
pushed it back up to 18.63 GB and it went down to 12.5gb

In terms of errors… it varies, sometimes theres out of memory errors, sometimes the CPU useage drops to sub 1% the server basicly freezes, but it happens when the ram runs out.

8

The cleaner thread was hand-coded to maintain compatibility with Java 1.8; java.lang.ref.Cleaner wasn’t added until Java 9.

The high-level documentation is here (for Minie) and here (for Libbulletjme).
Plus there’s javadoc, of course, but I see you’ve found that.

True. There’s no magic to explicitly free a physics space from Java. However, each object added to a space includes a hard reference to the space, so removing all objects from the space should make the GC’s job easier.

9

Backwards compatibility is important, so makes sense to keep it
if i can use java.lang.ref.Cleaner, then ill see what i can do with it
its running Java 21 currently

This is basically the first time I’ve had to use java in so long that i forgot java loves having periods everywhere, i have a bad habit of doing things above my skill level

just in case its part of the issue, is there a way to prevent the cleaner thread?
some of the cleanup is working, but part of it isnt

heres a crash error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

10

I’ve never tried using Cleaner for physics objects. I imagine it would involve forking the relevant physics library and building it locally. (I’m still unclear whether you’re working with Minie or Libbulletjme, by the way; Minie is for JMonkeyEngine and Libbulletjme is for apps that don’t use JMonkeyEngine.)

is there a way to prevent the cleaner thread?

Without modifying library sourcecode, I don’t know any way to disable the cleaner thread. And if you did, the native code would eventually exhaust its heap, and the app would crash.

java.lang.OutOfMemoryError: unable to create native thread

According to Stack Overflow, this indicates the app is hitting an OS limit on the number of threads per application. What OS are your app running on, and how many threads does your app create? (The physics library should only create one cleaner thread.)

11

Oh sorry, im using Libbulletjme

I dont think its run out of threads, it always lines up with when the server runs out of ram
I can watch it slowly rise

I think the java flags were impacting the normal garbage collection ontop of my original lack of cleanup efforts, its behaving better now
its still crashable, but should take long enough to be safe

Its running in docker inside of Ubuntu via pterodactyl/wings

after playing with the flags, and adding the PhysicsSpace.destroy()
it seems like as long as i don’t create them fast enough to overwhelm the garbage collector its stable.

its definitely still got a memory leak, im gonna let it run overnight spawning 7 physics spaces a second to see when it crashes, or how far along it is when i wakeup.

I think theres a better profiler if i edit the docker image that might help
so i might do a second test server just to try and figure out what its doing

1 Like
12

I would recommend connecting to your running server with VisualVM (you’ll just need to open some Java debug ports on your Docker container and maybe set a JVM parameter or two - I always run locally and it connects without any special config). That will give you a detailed view into exactly what your app is doing, along with the numbers of live instances of every class (which should almost directly tell you what your leak is). System RAM indicators are not at all reliable for determining how much RAM your app is using because the VM over-allocates and rarely releases back to the OS - the JVM might allocate 2GB from the OS even if your app is only steadily using 1GB. That said, you should only get the OOM if your app must allocate more RAM from the OS but cannot.

2 Likes
13

Libbulletjme has built-in tools for diagnosing native-memory leaks. They require a custom-built native library:

There are even a few documented native-memory leaks in the physics-space creation code:

I doubt you’re exercising those paths, but of course there may be other leaks I’m unaware of.

These being native-memory leaks, they might not show up in JVM-oriented tools.

I haven’t spent much time trying to fix these leaks because they seem minor, and the production apps I’m familiar with only create one physics space. So I’m concerned by talk of allocating 7 spaces per second overnight: hundreds of thousands of physics spaces.

Will you really need that many? Is there a good reason not to re-use physics spaces?

1 Like
14

I just remembered there’s a ThreadLocal data structure that maps threads to physics spaces:

It seems to me that, if you keep creating threads that create physics spaces, the spaces will never be freed.

It’s unfortunate physicsSpaceTL is a private field. Perhaps you could use reflection to null out entries when threads terminate.

15

Thanks, I atleast know where the issue is, and that its base code level so GC wont matter
main reason im not reusing physics spaces are

1: based on a proof of concept someone made with a plane and boxes/ragdoll and learned as i went, could also go with a general laziness

2: first time doing Async, so not super well versed in sending data between threads, i honestly make near 0 use of global variables.

3: Dont need physics objects to interact between the spaces, so i didnt see much reason to put them in the same space, also avoids it spawning physics objects in the middle of other physics objects

4: multicore, if I have ~7 cores free, then the physics spaces can be spreadout between them, doing a non worst case but still bad case physics space it can add ~80 static objects, and this prevented me from needing to do culling

5: jank, wasn’t sure how to do water, part of my placeholder is that it just runs the physics space at a lower speed

6: thread safety wise some things need to be done from the game thread, and some need to be done on the Async physics thread, so I made it self contained, it runs all the game thread things ahead of time, then switches to Async and runs all the physics needed before closing.

So maybe what i need to do is figure out what i can do to a physics space without being on its physics thread, maybe i can have a thread just looping physics updates while other threads do some of the logic of adding/removing physics objects.

And have a few physics spaces for different triggers to spread out the load

1 Like
16

think im on the right track
i was able to get these string outputs

CollisionSpace.physicsSpaceTL as a string is “java.lang.ThreadLocal@57c6ce85”

CollisionSpace.physicsSpaceTL.get() as a string is “PhysicsSpace#7fb8c40043e0”

this will reset CollisionSpace.physicsSpaceTL.get()
CollisionSpace.physicsSpaceTL.remove()

1 Like
17

600K physics spaces made, manageable leakage
Thanks Sgold, your a gentleman and a scholar

3 Likes
18

In the next release of Libbulletjme, setLocalThreadPhysicsSpace() will be public, so one will be able to clear the local reference more easily.

19

your the best, i will 100% be using that
now the only limitation i have is being bad at coding :slight_smile:

1 Like
20

lost some java flags i made prior to this, looks like this might of been only half of my solution :smiley:
so the memory leaks back!

I’m actually not sure what setLocalThreadPhysicsSpace() does
when i do a google it seems pretty undocumented for someone dumb like me