Jme-alloc project

Pavl_G · February 20, 2023, 1:05am

I restructured the project to include an android example application that implements the output jar jme3-alloc-examples and runs the StressLauncher example (tests all the memory capabilities), here are some logs:

I/TestNativeBufferUtils: **************** com.jme3.alloc.examples.TestNativeBufferUtils****************
I/System.out: java.nio.DirectByteBuffer[pos=4 lim=1000 cap=1000]
I/System.out: Buffer Data: 200
I/System.out: java.nio.DirectByteBuffer[pos=4 lim=1000 cap=1000]
I/System.out: Buffer Data: 200
I/TestNativeBufferUtils: **************** com.jme3.alloc.examples.TestNativeBufferUtils****************
I/TestDirtyMultithreading: **************** com.jme3.alloc.examples.TestDirtyMultithreading****************
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000 cap=2000000]
I/System.out: Buffer Data: 3000
I/TestDirtyMultithreading: **************** com.jme3.alloc.examples.TestDirtyMultithreading****************
I/android.exampl: ProcessProfilingInfo new_methods=0 is saved saved_to_disk=0 resolve_classes_delay=8000
I/TestMemoryCopy: **************** com.jme3.alloc.examples.TestMemoryCopy****************
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 200000
I/System.out: java.nio.DirectByteBuffer[pos=0 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=0 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 200000
I/System.out: java.nio.DirectByteBuffer[pos=0 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 200000
I/TestMemoryCopy: **************** com.jme3.alloc.examples.TestMemoryCopy****************
I/TestMemorySet: **************** com.jme3.alloc.examples.TestMemorySet****************
I/System.out: java.nio.DirectByteBuffer[pos=0 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 0
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 2500
I/System.out: java.nio.DirectByteBuffer[pos=8 lim=2000000000 cap=2000000000]
I/System.out: Buffer Data: 0
I/TestMemorySet: **************** com.jme3.alloc.examples.TestMemorySet****************
W/System.err: Stopped

And the application profiler shows a good performance…
The last W/System.err: Stopped is triggered from a Button to stop the thread running the examples (because backend memory allocations shouldn’t run on Ui thread).

github.com/Software-Hardware-Codesign/jme-alloc

Android examples: basic structure for an android example app module

Software-Hardware-Codesign:master ← Software-Hardware-Codesign:android-examples

opened 11:14PM - 19 Feb 23 UTC

Scrappers-glitch

+771 -14

### This PR modifies the following: - [x] The project root structure to includ…e 4 modules. - [x] Helper-scripts to include android-debug scripts/ ### This PR adds the following: - [x] `jme3-alloc-android-example` application module that implements both the `jme3-alloc` and `jme3-alloc-examples` jar outputs.

Pavl_G · March 20, 2023, 12:47am

I did a basic javadoc and nativedoc generators based on the javadoc.jmonkeyengine.org, but the generation is all dynamic through GitHub actions and will be based on the version of the release tag.

The current website is a prototype test for the core Gradle tasks, the Gradle tasks accept the output javadoc version to be deployed:

Pavl_G · March 20, 2023, 12:54am

I plan to extend the use of this API to more advanced concepts in the future, for example, editing the already created buffers by partitioning them and reusing some parts or mixing buffers, this could be used to edit a soundtrack or an image for example, What else?

This issue will be a future milestone for this extension:

github.com/Software-Hardware-Codesign/jme-alloc

[Feature-Request] Buffer Partitioning Utility

opened 04:43PM - 08 Mar 23 UTC

Scrappers-glitch

enhancement question

Provides an advanced buffer partitioning as a separate library. For example: … ```c #include<stdio.h> #include<stdlib.h> typedef struct { void* start_address; void* end_address; size_t offset; size_t size; size_t pointer_location; void (*invalidate)(void*); } MemoryPartition; static inline void invalidate(MemoryPartition* partition) { partition->start_address += partition->offset; partition->end_address = partition->start_address + partition->size; partition->pointer_location = partition->offset + partition->size; } static inline MemoryPartition create(void* buffer, size_t offset, size_t size) { MemoryPartition partition = { buffer + offset, (buffer + offset) + size, offset, size, offset + size, &invalidate }; return partition; } int main() { const size_t size = 4; void* buffer = (void*) malloc(size * 3); printf("Allocated buffer start address = %p\n", buffer); const MemoryPartition partition0 = create(buffer, 0, size); printf("Partition0 start address = %p\n", partition0.start_address); printf("Partition0 end address = %p\n", partition0.end_address); const MemoryPartition partition1 = create(buffer, partition0.pointer_location + 1, size); printf("Partition1 start address = %p\n", partition1.start_address); printf("Partition1 end address = %p\n", partition1.end_address); const MemoryPartition partition2 = create(buffer, partition1.pointer_location + 1, size); printf("Partition2 start address = %p\n", partition2.start_address); printf("Partition2 end address = %p\n", partition2.end_address); /* add some data to the partitions */ *((int*) partition0.start_address) = 44; *((int*) partition1.start_address) = 55; *((int*) partition2.start_address) = 255; printf("%s\n", "ls part: Print partitions data: "); printf("%i\n", *((int*) partition0.start_address)); printf("%i\n", *((int*) partition1.start_address)); printf("%i\n", *((int*) partition2.start_address)); return 0; } ```

Ali_RS · March 20, 2023, 6:07am

Hi @Pavl_G

Nice job!

When are you considering to release it on maven central?

Pavl_G · March 20, 2023, 7:05am

I am currently working on the maven-publish Gradle plugin, I already provided some of the requirements, such as the javadoc and source jars.

Do you plan to fix the issue related to this and integrate into the new engine release ?

Ali_RS · March 20, 2023, 7:17am

Sure for 3.7, but might be worth considering this for 3.6.1 too (I will do the 3.6.1 release when LWJGL 3.3.2 is out that should fix 1778) but I must consult this with other team members first.

Pavl_G · March 20, 2023, 7:22am

Brilliant, I will try to produce a release during days from now, but idk how much time it will take, I want to get the automation right, this is the most hardest part.

sgold · March 20, 2023, 5:50pm

I’ve done some release automation. Perhaps I can help?

Pavl_G · March 20, 2023, 7:15pm

Thank you, I am not familiar with the maven-publish Gradle plugin, I am studying the plugin parts to consider what to do next, but I am currently more deviated toward using Bash scripts and linking them to the UnixScriptRunner task, it’s already a part of the project buildSrc, however, If you have a better suggestion, let me know, I haven’t decided yet on a specific design for the publishing code.

pspeed · March 20, 2023, 7:41pm

SiO2 (and several of my other projects) use gradle to publish to maven central. The build is really straight-forward:

…using the buildSrc stuff. Properly setup, it’s a one-shot command to build and upload everything. Then I just need to go walk it through approval on the sonatype site.

Pavl_G · March 20, 2023, 9:25pm

Thanks @pspeed, I am willing to link the publishing to GitHub-Actions on-release event, I will take a look into your SiO2 building scripts for the maven-publish plugin utilization.

The documentation says I should log into nexus-manager, but it doesn’t accept my Jira credentials " Login Error Incorrect username, password or no permission to use the Nexus User Interface. Try again. Forbidden".

I saw some similar issues on the Jira platform, there were issues with the password, so I changed the password, but it didn’t work, Has anyone faced this issue before?

sgold · March 21, 2023, 4:35pm

I tend to agree with @pspeed: use Gradle’s ‘maven-publish’ and ‘signing’ plugins.

sgold · March 21, 2023, 4:37pm

Has anyone faced this issue before?

Occasionaly it shows me as logged in but won’t accept my commands. Then I have to log out and log back in. I think this is a recent issue.

Pavl_G · March 21, 2023, 8:59pm

Great, I will go with the maven-publish plugin, it looks straightforward, but one thing I am curious about, is where does it look for the actual artifacts to release on maven-central? Does it use the current latest GitHub release?

Pavl_G · March 21, 2023, 9:13pm

Hmm, seems it only publishes the primary jar, but only one jar? That might need some custom build.

Jme-alloc has 2 artifacts, an android jar, and a desktop jar.

sgold · March 21, 2023, 9:40pm

where does it look for the actual artifacts to release on maven-central?

When you run the “publishMavenPublicationToOSSRHRepository” task, Gradle uploads the artifacts you specify, typically from your build directory, to a SonaType server.

When you click on the “Close” button in the Sonatype web interface, the server copies the uploaded files to a “staging” repository.

When you click on the “Release” button in the Sonatype web interface, the server promotes artifacts from the staging repository to Maven Central. The process usually takes about half an hour. I don’t know any way to automate that part of the process or speed it up.

The JME Wiki includes a detailed description of these steps:

it only publishes the primary jar, but only one jar?

I suggest creating a Gradle sub-project for each artifact you plan to publish. For instance, the j-ogg-all project publishes 2 library artifacts, each of which has its own sub-project:

a “library” sub-project for artifacts with the “j-ogg-all” ID
a “vorbis” sub-project for artifacts with the “j-ogg-vorbis” ID

The “jmonkeyengine” repo is also structured this way.

An alternative is to pass a parameter to Gradle (using a “-P” command-line argument) to specify which build you want. That’s how the Minie project publishes different build flavors (“-droid”, “-debug”, and so on) without a bunch of subprojects. It proved a bit cumbersome, so I’ve gone and wrapped a shell script around it:

https://github.com/stephengold/Minie/blob/master/bash/release.sh

Pavl_G · March 21, 2023, 10:01pm

Thank you, I am already structuring the project into sub-projects “jme3-alloc” and “jme3-alloc-native” as main modules, but the build that is java compatible (jars) is produced at the “jme3-alloc”, so if it will end up writing a lot of publishing code that might not work at the end of the day, it will be better to use the maven sign-deploy-file command-line plugin, I wrote a code similar to it before on Serial4j-maven-publish for GitHub packages.

Pavl_G · March 22, 2023, 12:35am

I am satisfied with the manual Bash script, it seems easier to write and integrate with GitHub actions, thanks for the help, I studied the documentation and Paul’s script, but I found it hard to integrate the Gradle maven-publish plugin with GitHub actions…

Here is the PR:

As for my credentials error on the nexus-manager, I filed an issue, one maintainer replied that the problem may arise from using special characters within usernames and so I am now waiting for them to change my username to remove the special characters and grant the permissions of the current account to it.

Pavl_G · March 23, 2023, 6:45pm

I currently have an incompatibility error on the github-ubuntu-latest runner image:

It was happening also on my local Linux machine and I noticed something about this error in the release notes for maven-3.9.1:

Maven 2.x was auto-injecting an ancient version of plexus-utils dependency into the plugin classpath, and Maven 3.x continued doing this to preserve backward compatibility. Starting with Maven 3.9, it does not happen anymore. This change may lead to plugin breakage. The fix for affected plugin maintainers is to explicitly declare a dependency on plexus-utils. The workaround for affected plugin users is to add this dependency to plugin dependencies until issue is fixed by the affected plugin maintainer. See MNG-6965.

They say that this doesn’t happen with Maven 3.9! And the original maven shipped with ubuntu git image is Maven 3.8 which presumably affected!

I did a simple workaround by installing Maven-3.9 and using it directly, it works on my local machine and I can see the released packages on the Staging Repositories, but it didn’t for GitHub runner…

Totally unexpected!

xuan · March 23, 2023, 10:21pm

I recommend you to switch to maven wrapper: Apache Maven Wrapper – Maven Wrapper
So you the build doesn’t depend on the environment (as much).