Jme-alloc project

Well, apparently there were problems with the classifiers (javadoc, sources) upload documentation on sonatype, somehow using the code there generated the same API incompatiilities that the maven-3.8 binaries generate!

I finally produced a prototype version 1.0-pre-alpha-10 from the current PR branch, I know the tag is silly, but that is just to test against the Sonatype requirements.

I will make sure an alpha version 1.0-alpha will be available from the master branch soon.

I see, thank you, I will submit patches for the deployment script in future for sure, I will take a look if this is useful, for now, I managed to sign the files with a public key and send them to the remote repository as a part of GitHub releases.

Finally, after some prototyping, I announce release 1.0.0-alpha on the maven-central repository io.github.software-hardware-codesign:

The building and releasing are automated via GitHub runner images:

@Ali_RS Please try (you know it may need some time to appear on the maven-central servers):

implementation 'io.github.software-hardware-codesign:jme3-alloc-desktop:1.0.0-alpha'

I have exams next week so I might not be available beyond today to integrate it into the engine, so feel free to add it to the engine and solve the Reflection Allocator issue.

So, feel free to bing me here to release the final 1.0.0 when everything is OK!

This tutorial saves some time, thanks! Maybe we should have another tutorial for publishing artifacts to maven, something specific to trivial builds which is not already on documentation…

I realized this when I was surprised that each jar artifact should have its own POM file, the original project is sub-divided into a “jme3-alloc” sub-project which is a java code for both desktop and android and a “jme3-alloc-native” sub-project for native-code, so this was incompatible with what the Gradle does as standard, as the sub-project “jme3-alloc” outputs both the artifacts and “jme3-alloc-native” doesn’t output artifacts, it only produces the native binaries, I know there must be a way to do this via the Gradle plugins, but this may need a lot of structuring that I am not aware of so far…

Anyway, I found a simple workaround which is dynamically generating the POM file based on the artifactId :wink::

4 Likes

Thanks, going to try it.

FYI, from my observation when managing jme releases, it usually becomes available in the central repo (repo.maven.apache.org/maven2) in less than 20 minutes but it takes some time to appear in the maven search.

Central Repository: io/github/software-hardware-codesign/jme3-alloc-desktop/1.0.0-alpha

2 Likes

Just tried it, I am getting this exception:

Mar 24, 2023 5:16:29 PM com.jme3.system.JmeDesktopSystem initialize
INFO: Running on jMonkeyEngine 3.6.0-stable
 * Branch: HEAD
 * Git Hash: 53f2a49
 * Build Date: 2023-03-20
Mar 24, 2023 5:16:31 PM com.jme3.system.lwjgl.LwjglContext printContextInitInfo
INFO: LWJGL 2.9.5 context running on thread jME3 Main
 * Graphics Adapter: null
 * Driver Version: null
 * Scaling Factor: 1
Mar 24, 2023 5:16:31 PM com.jme3.renderer.opengl.GLRenderer loadCapabilitiesCommon
INFO: OpenGL Renderer Information
 * Vendor: Intel Open Source Technology Center
 * Renderer: Mesa DRI Intel(R) HD Graphics 3000 (SNB GT2)
 * OpenGL Version: 3.3 (Core Profile) Mesa 20.0.8
 * GLSL Version: 3.30
 * Profile: Core
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio Renderer Information
 * Device: OpenAL Soft
 * Vendor: OpenAL Community
 * Renderer: OpenAL Soft
 * Version: 1.1 ALSOFT 1.15.1
 * Supported channels: 64
 * ALC extensions: ALC_ENUMERATE_ALL_EXT ALC_ENUMERATION_EXT ALC_EXT_CAPTURE ALC_EXT_DEDICATED ALC_EXT_disconnect ALC_EXT_EFX ALC_EXT_thread_local_context ALC_SOFT_loopback
 * AL extensions: AL_EXT_ALAW AL_EXT_DOUBLE AL_EXT_EXPONENT_DISTANCE AL_EXT_FLOAT32 AL_EXT_IMA4 AL_EXT_LINEAR_DISTANCE AL_EXT_MCFORMATS AL_EXT_MULAW AL_EXT_MULAW_MCFORMATS AL_EXT_OFFSET AL_EXT_source_distance_model AL_LOKI_quadriphonic AL_SOFT_buffer_samples AL_SOFT_buffer_sub_data AL_SOFTX_deferred_updates AL_SOFT_direct_channels AL_SOFT_loop_points AL_SOFT_source_latency
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
WARNING: Pausing audio device not supported.
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio effect extension version: 1.0
Mar 24, 2023 5:16:32 PM com.jme3.audio.openal.ALAudioRenderer initOpenAL
INFO: Audio max auxiliary sends: 4
Allocator=buffer.TestReleaseDirectMemory$JmeNativeAlloc
Mar 24, 2023 5:16:32 PM com.jme3.app.LegacyApplication handleError
SEVERE: Uncaught exception thrown in Thread[#33,jME3 Main,5,main]
java.lang.ClassCastException: class java.nio.DirectFloatBufferU cannot be cast to class java.nio.ByteBuffer (java.nio.DirectFloatBufferU and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
	at buffer.TestReleaseDirectMemory$JmeNativeAlloc.destroyDirectBuffer(TestReleaseDirectMemory.java:80)
	at com.jme3.util.BufferUtils.destroyDirectBuffer(BufferUtils.java:1292)
	at buffer.TestReleaseDirectMemory.simpleUpdate(TestReleaseDirectMemory.java:73)
	at com.jme3.app.SimpleApplication.update(SimpleApplication.java:261)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.runLoop(LwjglAbstractDisplay.java:160)
	at com.jme3.system.lwjgl.LwjglDisplay.runLoop(LwjglDisplay.java:224)
	at com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:242)
	at java.base/java.lang.Thread.run(Thread.java:1589)



Here is a test case:

public class TestReleaseDirectMemory extends SimpleApplication {

    public static void main(String[] args){
        System.setProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION, JmeNativeAlloc.class.getName());
        TestReleaseDirectMemory app = new TestReleaseDirectMemory();
        app.start();
    }

    @Override
    public void simpleInitApp() {
  
        System.out.println("Allocator=" + System.getProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION));
       
    }

    @Override
    public void simpleUpdate(float tpf) {
        ByteBuffer buf = BufferUtils.createByteBuffer(500000);
        BufferUtils.destroyDirectBuffer(buf);
        
        FloatBuffer buf2 = BufferUtils.createFloatBuffer(500000);
        BufferUtils.destroyDirectBuffer(buf2);
    }

    public static class JmeNativeAlloc implements BufferAllocator {

        @Override
        public void destroyDirectBuffer(Buffer toBeDestroyed) {
            NativeBufferAllocator.releaseDirectByteBuffer((ByteBuffer) toBeDestroyed);
        }

        @Override
        public ByteBuffer allocate(int size) {
            return NativeBufferAllocator.createDirectByteBuffer(size);
        }
    }
}
1 Like

Direct java buffers are only of type java.nio.ByteBuffer, that is stated by the NewDirectByteBuffer jni method.

I think if you want to create a FloatBuffer, you shouldn’t obviously cast, but you should use the method putFloat to put a float data inside the byte buffer.

The problem is at this line, you are utilizing a wrong method signature.

Luckily, the destroy method signature can be generalized to the abstract Buffer, based on GetDirectBufferAddress, the current jme-alloc method only supports ByteBuffer, so this could be done easily, let me know if this is update is needed.

I am not sure that is correct. DirectFloatBufferU implements DirectBuffer but it does not extend java.nio.ByteBuffer.

BufferUtils.createFloatBuffer(); is a JME method and is used internally by JME in lots of places. Internally it uses ByteBuffer.asFloatBuffer().

Yeah, I think it should accept a Buffer instead of ByteBuffer otherwise it won’t be compatible with JME BufferAllocator interface.

1 Like

Yes, and so, all direct buffers are literally only ByteBuffers, byte buffers are contiguous memory chunks, and each chunk resembles 1 byte of data aka. 8-bits, the data could be any primitive data.

The asFloatBuffer just creates a view reference of type FloatBuffer for a previously created byte buffer.

EDIT:
To keep things clear, here is the documentation:

1 Like

Yep, I totally agree, I was looking for a getter method on the FloatBuffer, so I can fetch the original direct ByteBuffer referenced by it, but I cannot find anything related…

So, the alternative is to accept the abstract java.nio.Buffer instead, the asFloatBuffer documentation states that the created object is a view (reference) of the specified ByteBuffer and changes taking place through it will be reflected on the Original ByteBuffer memory…

1 Like

I guess it should be the same for asShortBuffer, asDoubleBuffer(), and asIntBuffer() too, right?

Because JME uses all of them…

1 Like

Yes, yep sure, the ByteBuffer memory model can be re-oriented as you like! So, in case of FloatBuffer, a float is 4-bytes of contiguous memory and so on, that is why you multiply sizeof(type) by the capacity on the BufferUtils#createFloatBuffer(int) to obtain the total memory to allocate.

1 Like

By the way, where does the natives get extracted?

2 Likes

Natives are extracted on the user.dir.

EDIT:
If you would like to change this, open an issue to discuss this.
If you would like to use the jme NativeLibraryLoader and disable the stock loader of this library, please open an issue to discuss it too, I already thought of this, it could be simply disabled using a static flag, for instance.

2 Likes

Please, open an issue for this crash, I will try to work on it tonight!
Thanks for testing.

2 Likes

jme3-alloc-v1.0.0-alpha-1 is now available with the above-suggested modifications and some additional CI/CD enhancements:

It will be available soon, I have closed and released it now on the nexus-repo-manager.

2 Likes

Works fine now, thank you so much @Pavl_G :slightly_smiling_face:

2 Likes

This may be off-topic, but I was unable to see destroyDirectBuffer ever called by JME NativeObjectManager.

I made a test in case you want to try it.

I am using Lemur InputMapper.

You need to change

assetManager.loadModel(new ModelKey("Models/head/head.gltf"));

to use a different model.

Here is the code:


public class MemoryReleaseTest extends SimpleApplication {

    FunctionId F_SPAWN = new FunctionId("spawnModel");
    //FunctionId F_DEBUG_MEMORY = new FunctionId("debugDirectMemory");
    FunctionId F_FORCE_GC = new FunctionId("forceGC");

    public MemoryReleaseTest() {
        super();
    }

    public static void main(String... args) {
        System.setProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION, TestAllocator.class.getName());

        //BufferUtils.setTrackDirectMemoryEnabled(true);
        NativeObjectManager.UNSAFE = true;

        MemoryReleaseTest main = new MemoryReleaseTest();
        main.start();
    }

    @Override
    public void simpleInitApp() {
        GuiGlobals.initialize(this);

        System.out.println("BufferAllocator:" + System.getProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION));

        InputMapper inputMapper = GuiGlobals.getInstance().getInputMapper();
        inputMapper.map(F_SPAWN, KeyInput.KEY_S);
        //inputMapper.map(F_DEBUG_MEMORY, KeyInput.KEY_D);
        inputMapper.map(F_FORCE_GC, KeyInput.KEY_G);

        inputMapper.addDelegate(F_SPAWN, this, "spawnModel");
        //inputMapper.addDelegate(F_DEBUG_MEMORY, this, "debugDirectMemory");
        inputMapper.addDelegate(F_FORCE_GC, this, "forceGC");

        initLight();
        viewPort.setBackgroundColor(ColorRGBA.Blue.clone());
    }

    private void initLight() {
        AmbientLight ambient = new AmbientLight(ColorRGBA.White);
        rootNode.addLight(ambient);
    }

    public void spawnModel() {
        System.out.println("Spawning model...");
        Spatial spatial = assetManager.loadModel(new ModelKey("Models/head/head.gltf"));
        rootNode.attachChild(spatial);
        //debugDirectMemory();
    }

    /*public void debugDirectMemory() {
        BufferUtils.printCurrentDirectMemory(null);
    }*/

    public void forceGC() {
        rootNode.detachAllChildren();

        assetManager.clearCache();
        assetManager.clearAssetEventListeners();

        System.out.println("Forcing garbage collection...");
        System.runFinalization();
        System.gc();
        //debugDirectMemory();
    }

    public static class TestAllocator implements BufferAllocator {

        @Override
        public void destroyDirectBuffer(Buffer toBeDestroyed) {
            NativeBufferAllocator.release(toBeDestroyed);

            System.err.println("Buffer destroyed!");
           
        }

        @Override
        public ByteBuffer allocate(int size) {
            return NativeBufferAllocator.allocate(size);
        }
    }
}

Pressing the S key will spawn the model and G key will clear up rootNode and asset manager and trigger a GC.

It should print a “Buffer destroyed!” message when the destroyDirectBuffer is called, but in my case, it is never called!

1 Like

Those issues should be considered before going into beta:

This is something I noticed the last month when before working on the deployment CI/CD, it doesn’t lead to problems with the stock NativeBinaryLoader, but since you want to use the Engine loader instead, it should be standard:

As for the guava JRE issue, I don’t care very much apart from cleaning the unused dependencies, but guava libraries aren’t literally shipped with the output jars or even used on the build compilation bin folder, so they may be used internally by Gradle, anyway it’s not urgent.