I think there must be something on the above setup, I don’t know much information about the NativeObjectManager that exists on jme, so I cannot help now on this, I might need some time to spot the problem, but I am truly sure, it’s not related to jme-alloc; the caller-stack haven’t reached the target java code.
1 Like
I suspect the location of this line is not true.
If jme overrides the default allocator, then you have to use this after the original jMonkeyEngine, we don’t know if this property is getting overridden elsewhere on the simpleInit or the update, I see you do print the property at the start of simpleInit, but you may try to do this too on the update to exclude this point.
It should not. It will be set in LwjglContext (only in lwjgl 3) only if nothing is previously specified.
To be certain, I added a System.out.println("Buffer allocated!") inside the TestAllocator.allocate() method and I see the message is printed in the console.
Edit:
In my test, I kept spawning and clearing the model repeatedly until I run out of memory.
1 Like
What is the result when using Lwjgl-3 allocator on this same test?
1 Like
jme3-alloc-v1.0.0-beta is now available:
2 Likes
In my test, the destroyDirectBuffer is not called with Lwjgl 3 allocator too, but I do not run out of memory (memory usage never went beyond 1.9 GB in my test), GC does automatically free the memory.
With jme-alloc memory usage continues to grow.
As stated by the documentation as far as I remember, the direct buffers are allocated on the native heap which are not freed by the GC, so this test is revealing the fundamentals of direct buffers, but idk why the lwjgl-3 buffers are freed…
Btw, jme-alloc has a very straightforward code, it brings the stdlib dynamic memory allocation to java with no extra interfaces or extra java code, so it is utilized the same as if you using the GNU standard library dynamic memory allocation API.
1 Like
I notice LWJGLBufferAllocator uses a PhantomReference to keep track of allocated ByteBuffers so when the ByteBuffer object GCed it frees the native memory. To free the memory it uses the Long address of the ByteBuffer because the reference to the ByteBuffer object is no more available once GCed.
It does this from a separate thread
1 Like
So basically it bypasses JME NativeObjectManager which seems to only clear GL objects from GPU but does not clear native buffers from RAM even if the UNSAFE flag is set to true.
1 Like
I also tried with PrimitiveAllocator and in my test, the destroyDirectBuffer() method was never called by JME but memory usage did not go beyond 2GB because it was auto-cleaned by GC.
Edit:
Based on these experiments our AndroidNativeBufferAllocator is also broken (has Memory Leak) because it does not deallocate buffers like jme3-lwjgl3 does using PhantomReference, and NativeObjectManager will never call destroyDirectBuffer on native buffers, and because ByteBuffers are generated from native C code and not by ByteBuffer.allocateDirect(size), java garbage collector won’t free them automatically.
1 Like
I don’t have much background about the NativeObjectManager and PhantomReferences, hooking direct buffers to the JVM GC seems to kinda violate the direct buffer rules, if jme directly calls the NativeBufferAllocator#release(Buffer) after finishing from the buffer as intended by the current design of the API, this won’t be a problem, otherwise, the current jme3-alloc isn’t yet compatible with jme…
If your testcase isn’t a part of the engine and was intended for enhancements/features, you could open an issue and provide this techdemo and provide enough information about NativeObjectManager and PhantomReferences.
1 Like
I tested TestInstanceNode backthen with android, and I confirmed the destruction of buffers by visualizing the debug logs, no memory leaks will result if you call release when you finish using the buffer, these are the rules of direct buffers, and ByteBuffer#allocateDirect() is the same as its JNI counter-part NewDirectByteBuffer, but hooking to JVM GC is not considered as essential:
If you want to test the node instancing with the current jme-alloc, I will consider having a concomitant debug version for each release, we can do this CI/CD work starting from jme3-alloc-v1.0.0-gamma for example, that will give you a better way to always visualize your buffers.
1 Like
Your quoted passage does not support this incorrect assertion.
Direct buffers should be GC’ed. They are not (necessarily) a part of the “regular GC-managed heap”. These two statements are not at odds.
Traditionally, this was a problem because you could allocate gigs of direct memory and the GC wouldn’t know you are running out of RAM… but direct memory WAS GC’ed. It just wasn’t involved in heap-size calculations.
This seems to have been fixed on modern Java and in fact I’m no longer able to query the size of direct memory because it’s the same as the heap.
Either way, it’s kind of anti-Java to force applications to free Buffers manually. Nice to provide but not required.
2 Likes
Interesting, I haven’t faced these internals while studying the JNI specification guide and java nio packages, may be I missed some lines, it’s worth considering on later releases, if you have official resources from java on this, could you please kindly send them?
I currently don’t care about other applications, jme3-alloc project was based entirely on this interface BufferAllocator, and so, one last question to help me decide whether to continue to the stable release, does jme rely on calling the BufferAllocator#release(Buffer) manually or freeing direct buffers indirectly via GC? The answer should be simple and should determine what should I do.
1 Like
Manual freeing by the application has never been required. JME providing the ability to manually release things was added later for applications that want to tightly control their resources (I know because I was one of the original ones that requested this feature for Mythruna… and now I don’t even need it.)
Whatever JME/libaries/whatever need to do to make that happen, applications should not be suddenly required to self-manager all of their buffers.
3 Likes
I thought jme takes care of releasing the memory manually on a state or something as a part of the update.
I guess the current API now is not yet compatible with jme, so I will stall the 1.0.0-stable release for now until I better understand what happens under the hood, kindly, please, I would appreciate it if someone opens an issue to describe this, and points me to where should I look for more info on the source code.
For regular Java-managed direct buffers, JME does not need to clean them up as the GC will clean them up.
For any memory that Java is not allocating, special support (like PhantomReference) will have to be used to make sure to free the native memory.
2 Likes
I think LWJGLBufferAllocator is a good place to look at how you can implement this with PhantomReference
1 Like
Great, I started to understand a bit after looking into the PhantomReference documentation, so a kind of creating CollectableDirectBuffer may be the solution.
Note that for this your API should be able to release buffer by its address (a long value). See LWJGLBufferAllocator.
1 Like