Certainly the largest bit of work involved. Perhaps it would be better to consider this an opportunity to clarify the engine structure? What I am wondering is how many of the classes that would need to be moved are actually API points, and how many are typically only used internally. If this would actually move a lot of API points, we’d need a very good plan for migration, and getting the broader community on board would be a major necessity. In such a case, this would probably be the major blocker.
?? That doesn’t sound right. A protected class is always an inner class, and therefore a member of its enclosing class. It can be extended in classes that extend the enclosing class, which can be in different packages. (Ofc, the parent package would need to be exported from the module, but that is true of API extension points in general.)
Hmm… I’d say that those should each be a separate module. Perhaps something with provides... <Renderer plugpoints>
Well, gradle is supposed to be flexible. Whether a project re-layout is warranted or not is open to discussion, but I think that the basic workaround on the javadoc end is for the “combined javadoc” target to explicitly specify its combined sourcepaths.
From a “modules” perspective, JME is written as one big module… core, desktop, and effects probably. Networking should be a separate module (and probably already could be). Physics certainly (though there may be lingering issues with native physics and java physics sharing the same trees… for good reason). Without looking at the whole list right now, I consider everything else “not really JME”.
The fact that these are split up into separate subprojects is not really relevant to a discussion about modules.
I suspect by “protected classes”, he might have been referring to protected methods that get “friendly” access within the package. (JME might even have some lingering default ‘friendly’ access that has not been fixed to be protected.)
Many of these choices were made for API defense and/or performance. Performance is the number 1, 2, 3, and 4th goal in most of the engine.
It’s kind of unfortunate. Modules are really nice for giant enterprise web stacks… they kind of suck for a game engine.
@sailsman63 thanks for creating a thread for this discussion.
Yes, thank you for clarifying, that is exactly when I meant. If a class has default or protected access level on a member, that member will not be able to be accessed in another module that tries to have classes in the same classpath for the sake of performing the protected or default access. Both because this is unallowed in modules, and because no two modules can have the same classpaths.
Wow, I realize my explication on this before was extremely lacking, sorry about that, too early in the morning I guess.
I have to agree. While I have used modules in JavaEE (Oh sorry we are not allowed to say that any more) JakartaEE, enterprise application stacks already have loads more security for between jar/war/ear access than the JavaSE side, yet on the JavaSE side key implementation details in modules can make them very difficult to implement on any large project of scale. The largest issue, which is finally being resolved (many years too late) is tooling support. IDEA just added a fix in the latest EAP for mixing unnamed and automatic named module dependencies in a java module project. This is an issue I personally have been fighting for at least three years.
My ranting aside, key aspects of module based projects are separation of logic in different jars, which at first glance projects like JME look like they have that. Unfortunately, the boundary that modules place on that separation of logic is quite large, much larger than simple old school non-module based java.
Key places where issues will arise that will need to be sorted out.
Any place where classpaths overlap
Any place where we have multiple modules that can be swapped out (lwjgl2 vs lwjgl3 for example)
Android does not use modules, so no reliance on module technology can be used for loading modules at runtime. Yet, using a classpath loader bypasses the module system entirely making it a mute point. A hybrid solution will be needed for dynamically adding jars to the classpath. (I’m not sure how desktop natives are loaded these days for non-lwjgl natives. This may not even be an issue.)
For any resource that needs to be loaded between modules, explicit opens will need to be added for the location of those resources, otherwise they will be unavailable.
Loading user assets (as resources from the user module) with the jme asset loader (from jme module xyz) will require the user to add explicit opens to their module info for jme to open them. There will need to be some well documented information on getting a user project setup for jme modules.
OK, that list is long enough, and there will be many other challenges on top of the ones listed.
As stated, if someone (@sailsman63) was interested in creating a PR for pulling this off, I am happy to help with my knowledge on the subject, but I will not have the time to perform any of the conversion myself.
Note also that modules would allow deprecating our current native library extract-to-files-and-load mechanism with linked native libraries. To do this, we’d package and publish modules that use native libraries with the native libraries as jmod files, which are consumed by using jlink to create a custom JVM runtime image (essentially “baking in” our natives alongside JDK natives in the distributable JVM image). The advantage here is that those native libraries can then be found and loaded automatically by the System class as they’re on the library path.
I’m not sure if this is overall a benefit to the engine and users or not, but it’s an option if anyone’s interested in exploring it. I don’t know what this would imply for Android dev, in particular.
To me the biggest benefit is downstream modules - if someone is writing a jME library, they’re locked out of using the Java module system at all if jME isn’t itself modular. This has negative implications on a number of things, probably most notably on how you build your distributable image. It’s possible to build an image without the non-modular bits and then include them as jars and invoke the JVM image with appropriate arguments to load and run the non-modular bits, but it’s more complicated than building a single modular distribution.
Code that uses native libraries also benefits, as discussed above.
So for example, module-wise, can we consider jme3-lwjgl and jme3-lwjgl3 to be in the same module while keeping them separate subprojects, right? …then we should still be able to use split packages without issue, correct?
I wouldn’t consider those “split packages”, since they’re mutually exclusive. It’s more like incompatible versions of the same library. Since these are each compiled separately, they should be fine. Trying to load both simultaneously into an application will cause errors (and I suspect that jlink would balk as well) but that’s fine. I think that we would use different module names in this case, just for clarity.
Regarding the “friendly access” situation: I think there are probably multiple possible approaches. Off the top of my head, if we’ve got a type that needs to be accessible to other components, but not to end users, we can make it public and then the package can be exports... to the other modules involved. This would probably need to be evaluated case-by-case, though.
Ok, let’s draw a line in the sand and make a straw man because I think it will be illuminating.
Wrap all of JME in one giant module and call it a day.
What are the pros and cons?
I really worry because in the past with things like osgi and different tools, what folks want out of it and what they can actually have are two different things. So let’s work from the easiest and most ridiculous end of the spectrum.
If we end up with a lot of “We need this way because of X” and it turns out X is still impossible or the cost of X is too high… then we have something to talk about.
Okay, I’ll bite. to begin with, Modules have a 1:1 relationship with distribution jar artifacts. Each Module is contained in one, and only one jar. Each jar contains only one Module. To get this to work at all, we’d have to either
Interleave all of the sub-projects back into a single source tree, or
(More likely) have a custom build target that interleaves the class files into a “fat” jar with the module-info specified.
Note that we can’t include both lwjgl and lwjgl3 in the same artifact, so they would need their own modules anyway. Leaving them un-modularized means that they can’t be included in jlinked images without shenanigans from the end user side, which undercuts one of the major impetuses for modularizing in the first place.
Similar situation with jbullet: bundle by default, or keep it separate so that someone can more easily swap in minie if they prefer?
I suggest we may try to make good use of jdeps to generate module-infos for the already existing jar files, if jdeps is reliable, I think also it could be combined with GitHub CI/CD to generate modules after jarring that could be re-zipped again before deploying, I am not quite actively involved in using the java modular system, but I found jdeps could do this for my projects and I think it could for jMonkeyEngine as the engine is already organized into Gradle subprojects which are almost compatible with the java modular system, so in general, we could even generate modules without touching the current engine files and sources!
This was a good session back in 2017, where Mark Reinhold refactored the legacy jackson classpath library into a modular API using jdeps:
As for the Android system, I guess it’s okay, Android is using the DexFilePath and not the regular java class-path, and it should ignore the Java modular system, Android runs already on the modular java-11, anyway with jdeps we may have the chance to test that on a SNAPSHOT or something without corrupting the engine.
