Hello fellows, following the talk on ReflectionAllocator is broken on JDK 16 · Issue #1674 · jMonkeyEngine/jmonkeyengine · GitHub, i am currently preparing a test-case project as a template for jme-alloc, i have chosen to model the project in a modular approach, a jvm-alloc module and a native-alloc side, so we can do everything in a step-by-step fashion, i have added a WIP todo-list in the readme to track the changes and here is generally what i will do:
Prepare the build environment (gradle).
Prepare CI/CD to build for different variants.
Packaging the shared object files within the jars.
What’s done ?
Separate native and jvm modules are ready,
Generating java headers to the native module ./include/. is ready.
Next copying native output and java byte classes to a single build directory and producing an active jar file.
EDIT:
The sequence of native compilation, packaging and linking will be:
Compile java code and generate jni headers → Copy [*.class] files inside their packages from the build folder to a shared output folder → Compile native code into shared object file → Copy [libjmealloc.so] files inside their variant packages from the build folder to the shared output folder → Package the output folder into a portable jar file → Extraction by java code and linking on runtime
Note that you do not need to create a new BufferUtils calss, JME already has it. You just need to create an implementation for BufferAllocator interface.
It’s my pleasure, i have chosen to learn new things with jme, i need to try this.
I know, thanks for pointing out, however this will force me to include jme-core as a dependency and i don’t want to do this, it’s better to keep it a separate project and then including it on jme will provide jme-core with additional raw buffer functionalities to implement the new buffer allocator, it is way much cleaner i think.
Long story short, i will be building a minimalistic api.
We can then add the published jar and use it inside jme as follows:
package com.jme3.util;
import com.jme3.alloc.NativeBufferUtils;
public final class DesktopBufferAllocator implements BufferAllocator {
/**
* De-allocate a direct buffer.
*
* @param toBeDestroyed the buffer to de-allocate (not null)
*/
void destroyDirectBuffer(Buffer toBeDestroyed) {
NativeBufferUtils.destroy(toBeDestroyed);
}
/**
* Allocate a direct ByteBuffer of the specified size.
*
* @param size in bytes (≥0)
* @return a new direct buffer
*/
ByteBuffer allocate(int size) {
return NativeBufferUtils.allocate(size);
}
}
Note, i haven’t started the project yet, since about 90% of it will be building for different variants, so that why i am now aligned to the building model, selecting a good clean building model from the start will keep this project maintained forever, so class and package names might not be the best for now…however they are easily refactorable later since this won’t be a user code.
Note2: I am not going to use cpp, i will use plain c, c++ complicates things with no need.
My original thought was when jme3-alloc.jar is included in the classpath it should be automatically detected and used by BufferAllocatorFactory this way we could make it switchable.
But I like your approach too, we can add a dependency to jme3-alloc inside jme3-lwjgl module and implement the LwjglBufferAllocator there (like the jme3-lwjgl3 did).
By the way just wanted to mention that XX:MaxDirectMemorySize settings will be ignored when using a native buffer allocator. So an OutOfMemoryError will never happen if it exceeds the MaxDirectMemorySize.
It makes sense, since the native buffer allocates memory on the native heap, but it must fire a jvm crash with stack logs that reflects out of resources state, i have collected a couple of jvm crash logs from my library Serial4j throughout the development phase to examine them later, however i cannot remember currently what is exactly the crash logs for this, but it will be something similar to native segmentation error or may be a native heap out of memory.
MacOS12 runner seems to have everything pre-installed.
I tested all github linux runners before on another project and they have everything pre-installed (GCC, gradle, and java).
Windows-server-2022 seems also to be complete and it has the bourne-again shell (BASH) which is great news for me.
EDIT3:
And this will be utilized between the compile jobs and the final assemble job, so basically gradle c++ plugin will compile a variant and upload it as artifact and there will be a matrix of jobs, each will point at a variant, the final assemble job will download all the artifacts and package them into a jar using the Jar gradle task.
[ Compile java (jdk-8) → upload byte code and native headers] → [ Download native-headers → Compile-variants job → upload libjmealloc.so artifacts ] → [ Download byte-code and libjmealloc.so artifacts → Assemble-job → upload assemble artifact] → [onRelease: Download assemble artifact → upload to maven ]
I am not sure i can support x86 for now, however i can support a workaround for intel instruction set linux_x86 on x64 systems by downloading the x86 toolchains and compiling the code against it or using x64 GCC options to compile for lower instruction sets (if ever exist), but i am not sure with other systems, i don’t develop natives on mac and windows.
EDIT:
github hosted runners donot support x86 out of the box, only x86_64.
Does lwjgl-2 support x86 systems though ? I need to be sure; because starting from lwjgl-3.2.3 x86 on mac and linux support with arm support (i got these info from lwjgl customize website).
I will try to apply my workaround , but it may involve some kind of custom script or bash scripting, i am already into it, knowing that windows also has bash installed is a good news, i will see what i can do, at least provide a linux-x86 image.
Yeah, i will consider that, thanks for pointing this out, however the root project name is jme-alloc so i am not sure, i may use other names like lib for the jvm code and native for the native code may be, let me know what do you think ?
If we are going to merge this into the main project, then we need to follow the naming conventions you stated from the start…we may consider the conventional module names in case this project is merged in future.
Note: this project is separated into 2 modules with no common gradle files so this won’t affect the engine if its merged later on.
, my fault, also check the artifact jar, you will find only the x86-64 folder.
, but it may involve some kind of custom script or bash scripting, i am already into it, knowing that windows also has bash installed is a good news, i will see what i can do, at least provide a linux-x86 image.