Jme-alloc project

Pavl_G · February 2, 2023, 7:24pm

Idk, I haven’t glanced over lwjgl3, I have just studied the java.nio.ByteBuffer and simply ported the gnu libc to java, but the gnu memory allocation API is the only accessible cross platform memory allocation utility, there are other system specific API, for example kalloc used by the Linux kernel.

Pavl_G · February 8, 2023, 1:18am

I have added a logging debug api to the library, here is the PR:

In this PR, the following will be added:

A native logging api (supports logging to stdout, stderr and jme3-alloc-debug.log file).
Refactored the native build to use build classes from gradle buildSrc (utilizes org.gradle.api).
Compile debug binary tasks, that enables the logger api using the macros __ENABLE_LOGGER for stdout-stderr logging and __ENABLE_DEBUG_LOGGER for jme3-alloc-debug.log logging.

Here is a video for debugging the library, as you can see there seems to be no memory leaks so far till now:

Pavl_G · February 10, 2023, 10:26pm

Hi, i have added a simple memory profiling tutorial that utilizes both the jme3-alloc-native logging api and the GNOME system monitor and an example chooser (via gradle tasks).

github.com

Software-Hardware-Codesign/jme-alloc/blob/master/jme3-alloc-examples/README.md

# jme3-alloc-examples gradle module

This module holds a couple of simple techdemos to test the api.

## Here is the list of the available examples and how to run and build a runnable jar for stress testing: 
- [x] [TestNativeBufferUtils](https://github.com/Software-Hardware-Codesign/jme-alloc/blob/master/jme3-alloc-examples/src/main/java/com/jme3/alloc/examples/TestNativeBufferUtils.java):
```bash
└──╼ $ ./gradlew :jme3-alloc-examples:copyLibs && 
       ./gradlew :jme3-alloc-examples:TestNativeBufferUtils \
                 :jme3-alloc-examples:createJar \
                 :jme3-alloc-examples:run
```
- [x] [TestMemorySet](https://github.com/Software-Hardware-Codesign/jme-alloc/blob/master/jme3-alloc-examples/src/main/java/com/jme3/alloc/examples/TestMemorySet.java):
```bash
└──╼ $ ./gradlew :jme3-alloc-examples:copyLibs && 
       ./gradlew :jme3-alloc-examples:TestMemorySet \
                 :jme3-alloc-examples:createJar \
                 :jme3-alloc-examples:run
```
- [x] [TestMemoryCopy](https://github.com/Software-Hardware-Codesign/jme-alloc/blob/master/jme3-alloc-examples/src/main/java/com/jme3/alloc/examples/TestMemoryCopy.java):

This file has been truncated. show original

As always, if you have any suggestions or use a better tool for profiling native memory for java applications, i will happily wait for your PR, keep in mind that some tools as GDB may not work, because our library is not a valid native program, if we want to use GDB for instance, we will have to build another native c executable around the library that tests the library functions, that is another story …

Pavl_G · February 11, 2023, 12:31pm

Ali_RS:

I am not sure, maybe you should separate platform native jars using a “classifier” as lwjgl did. So they can be picked selectively.

For example:
jme3-alloc:version:natives-linux
jme3-alloc:version:natives-windows
jme3-alloc:version:natives-mac
jme3-alloc:version:natives-android

Please open an issue regarding this, giving the full instructions of what needs to be done, i will start working on a release after finishing the android build script…

Ali_RS · February 11, 2023, 12:54pm

I am not sure about this case TBH

I like your current approach of embedding all natives inside a single jar because the natives’ size is very small. So I would say if you are not going to have android support I suggest keeping the current approach. (i.e. let us keep jme3-alloc only for desktop )

but if you really want to include android builds then let me know so I will go ahead and open an issue for that on GitHub.

Pavl_G · February 11, 2023, 1:27pm

I am planning to migrate android allocation to the new allocation api, so if we added any new feature or a bug fix in future, it’s better to keep the build automated to all the supported systems in a single project.

I already opened an issue to track the android build .

As for the modular approach, i believe you meant the output would be modular and not having multiple module for each system…

So, one module jme3-alloc-native and different native modular outputs (for different binares) jme3-alloc-android-natives, jme3-alloc-desktop-natives or breaking the desktop further into mac, windows and linux.

So, they will look like this when implemented through gradle:

jme3-alloc-android-natives.jar: contains only the android native binaries and no java code.
jme3-alloc-linux-natives.jar: contains only the linux native binaries and no java code.
jme3-alloc-mac-natives.jar: contains only the mac native binaries and no java code.
jme3-alloc-windows-natives.jar: contains only the windows native binaries and no java code.
jme3-alloc.jar: contains only java code and no native binaries.

This may need a change in the NativeBinaryLoader class for desktop (an additional step to search for the jar file first before the binary).

There is another option tho, is to keep the current jar creating job that creates the jme3-alloc.jar for desktop variants and rename the jar to jme3-alloc-desktop.jar, and add a new job to output another jar for android jme3-alloc-android-natives.jar(only native binaries) and a java only jme3-alloc.jar.

The native android jar and the byte code jar are merged into a single apk by the AGP (android gradle plugin) when the user builds the jme game application.

So, we will have the following output jars with no change in code modules:

jme3-alloc-desktop.jar: is just a refactor rename of the current output.
jme3-alloc-android-natives.jar: contains only the android binaries in a lib folder.
jme3-alloc.jar: contains only the allocation API java code (a byte code jar).

By doing this, we are able to create a Cross-Platform desktop output and an android only output (with no desktop boilerplate binaries).

What do you think ? and we could further break the desktop jar into a mac and a linux and a windows ones in future…

EDIT:
Keep in mind, on android the NativeBinaryLoader will not need to extract the binaries, it will just load them as they are inside the lib folder inside their respective architectures and the android framework will append the absolute directory to them automatically based on the architecture, so having separate jars (one for natives and one for java) is the same as having a merged jar for android; because the AGP merges them together anyways in the mergeDex task, while on desktop merging is not done automatically…and i think extra steps will be needed, @sgold do you have any remarks that will be helpful, i think you have separated the Minie native binaries into multiple system dependent jars in a release, please let me know your opinion, thank you.

EDIT2:
I missed one thing, i haven’t tested it, but i think if the native binaries jars are included in the class-path of an output jar, they can be then loaded as if they are inside the same jar using the ClassLoader.getResourceAsStream(String)…

Ali_RS · February 11, 2023, 1:46pm

Yeah, that sounds good to me.

Ali_RS · February 11, 2023, 1:49pm

Yes, as long as all jars are on the classpath it does not matters I believe.

Ali_RS · February 11, 2023, 1:51pm

By the way, just in case this might be of interest to you

Pavl_G · February 11, 2023, 1:53pm

To keep things clear, for example in a user output game jar file, the MANIFEST.MF will have a Class-Path property:

Manifest-Version: 1.0
Class-Path: ./dependencies/jme3-alloc-natives-linux.jar
Created-By: 19 (Oracle Corporation)
Main-Class: com.jpluto.DesktopLauncher

In this case, the NativeBinaryLoader will treat the files inside the jme3-alloc-natives-linux.jar as a part of the game executable (as they are now on the same class path) and this may be a kind of merging the paths may be…as android do, but not really merging the real files…

EDIT:
And also other jMonkeyEngine jars downloaded from maven-central will need to be added as well to the class-path property and copied to the dependencies dir.

Pavl_G · February 11, 2023, 2:02pm

Thanks for pointing this, fortunately, my approach namely the “incremental binary loading” on PR#21, searches the binary with the specified name libjmealloc on the current working directory first, and tries to load it, if the binary is broken or cannot be loaded (i.e: an UnsatisfiedLinkError is raised) then the NativeBinaryLoader cleanly extracts the binary according to the current system variant from the jar file to the current working directory overwriting the broken or incompatible binary , then reloads it…

Ali_RS · February 11, 2023, 2:04pm

Cool

Pavl_G · February 13, 2023, 7:01pm

Okay, in this PR:

github.com/Software-Hardware-Codesign/jme-alloc

Android Basic Support

Software-Hardware-Codesign:master ← Software-Hardware-Codesign:android-buildscript

opened 05:13PM - 13 Feb 23 UTC

Scrappers-glitch

+387 -48

### This PR adds the following: - [x] Bash build script for android. - [x] An…droid compiling task. - [x] `UnSupportedSystemError`: that is thrown when an unsupported system has been detected. - [x] A github runner compile and upload jobs for android. ### And modifies the following: - [x] Gradle buildSrc: support script command args. - [x] `NativeVariant`: added the Android to the systems. - [x] `NativeBinaryLoader`: added a check for android and other unsupported systems. - [x] Github runner jobs to the new gradle tasks names.

I managed to:

Create an android script to compile to all android architectures.
Added a JarMetadata class at the buildSrc for the output jar name and version.
The android assemble task is the same as the desktop one, but with changing the jar file name through the JarMetadata class .
NativeBinaryLoader: Detecting unsupported systems (iOS for example…) and throwing an UnSupportedSystem business exception.

Here is the overall workflow (the workflow tree is composed of two branches, the android and the desktop branching from the compileJava task):

The current output releases are:

One more thing left is to try the android jar and create a jme3-alloc-android-examples module that sets up an android apk that implements the jme3-alloc-examples code…

EDIT:
If anyone would like to participate in testing this, just download the android release archive, extract the jar file and add it to android apk locally, then copy the NativeBufferUtils example for instance…

Pavl_G · February 14, 2023, 5:38am

Tested on android, works fine , here is some logs:

I/TestNativeBufferUtils: **************** com.jme3.testalloc.TestNativeBufferUtils****************
I/TestNativeBufferUtils: **************** com.jme3.testalloc.TestNativeBufferUtils****************
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
W/.jme3.testallo: Accessing hidden method Landroid/os/Trace;->asyncTraceBegin(JLjava/lang/String;I)V (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/os/Trace;->asyncTraceEnd(JLjava/lang/String;I)V (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/os/Trace;->traceCounter(JLjava/lang/String;I)V (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/FontFamily;-><init>()V (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/FontFamily;->addFontFromAssetManager(Landroid/content/res/AssetManager;Ljava/lang/String;IZIII[Landroid/graphics/fonts/FontVariationAxis;)Z (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/FontFamily;->addFontFromBuffer(Ljava/nio/ByteBuffer;I[Landroid/graphics/fonts/FontVariationAxis;II)Z (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/FontFamily;->freeze()Z (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/FontFamily;->abortCreation()V (light greylist, reflection)
W/.jme3.testallo: Accessing hidden method Landroid/graphics/Typeface;->createFromFamiliesWithDefault([Landroid/graphics/FontFamily;Ljava/lang/String;II)Landroid/graphics/Typeface; (light greylist, reflection)
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 1885423950
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/.jme3.testallo: ProcessProfilingInfo new_methods=1195 is saved saved_to_disk=1 resolve_classes_delay=8000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=90000 cap=90000]
I/System.out: Buffer Data: 323131

Code:

package com.jme3.testalloc;

import com.jme3.alloc.util.NativeBufferUtils;
import java.nio.ByteBuffer;
import java.util.logging.Logger;
import java.util.logging.Level;

/**
 * Tests the {@link com.jme3.alloc.util.NativeBufferUtils} basic functionalities including: clear memory allocation, destruction
 * and some logging stuff.
 * Note: to run type: └──╼ $./gradlew :jme3-alloc-examples:TestNativeBufferUtils :jme3-alloc-examples:run
 *
 * @author pavl_g
 */
public final class TestNativeBufferUtils {

    private static final Logger LOGGER = Logger.getLogger(TestNativeBufferUtils.class.getName());

    public static void main(String[] args) {
        LOGGER.log(Level.INFO, "**************** " + TestNativeBufferUtils.class.getName() + "****************");



        /* Note: data printed from here is not the buffer's anymore, this buffer's memory address has been destructed */
        /* Warning: the printed data is another buffers data ! */
        /* Note: writing on this buffer again will fire a jvm crash with a crash-log file; because the current buffer may not be thread-safe */
        new Thread(() -> {
            for (int i = 0; i < 100; i++) {
                final ByteBuffer buffer = NativeBufferUtils.clearAlloc(90000);
                buffer.putLong(323131L);
                printInfo(buffer);
                NativeBufferUtils.destroy(buffer);
                try {
                    Thread.sleep(5000);
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                }
                printInfo(buffer);

            }
        }).start();


        LOGGER.log(Level.INFO, "**************** " + TestNativeBufferUtils.class.getName() + "****************");
    }

    private static void printInfo(final ByteBuffer buffer) {
        System.out.println(buffer);
        System.out.println("Buffer Data: " + buffer.getLong(0));
    }
}

Notice, that the buffer destroy on android takes some time, i don’t know the internal threading architecture other than a choregrapher thread (for ui) and a Gl thread, there must be other native hardware threads internally bound to the application (and some memory organizing threads too), so it’s not something to worry about anyway, i am going to study it later and document the information when it’s available to me…

Ali_RS · February 14, 2023, 7:18am

Why do some logs show 0 for buffer data?

Pavl_G · February 14, 2023, 9:16am

It’s the print after the buffer destruction.

Ali_RS · February 14, 2023, 10:08am

Oops, sorry did not notice the second printInfo!

Pavl_G · February 15, 2023, 11:08pm

I have issued a new project on Sonatype, this is my first project on the mavencentral by the way :
https://issues.sonatype.org/browse/OSSRH-88874

During the response time, i will prepare java-docs and a basic website for the API.

Ali_RS · February 16, 2023, 6:55am

Great

Voted

Pavl_G · February 17, 2023, 3:08am

Thank you, i got the ticket now , expect a snapshot soon.