JME - Future Development

[danielp]

JME’s future development is a topic we’ve all discussed multiple times before (here being the first that comes to mind), but in light of some new industry developments I think it bears revisiting.

I see two primary themes that have emerged over the years:

1. Will jME support Vulkan?
2. jME’s API could be improved (@RiccardoBlb’s work on Pipelines comes to mind here)

Regarding (1), the consensus here (which I concur with) seems to be that jME doesn’t stand to gain much (if anything) from Vulkan without fundamental architectural changes. (2) has been approached on a per-feature basis, without breaking changes to the core API (appropriate for a major release - though I don’t know if the Pipelines feature builds on top of the core or is a refactoring).

My interest in this topic was piqued by recent developments around the WebGPU standard, which seems to be close to finalization (note that preview releases are beginning to ship in browsers and it seems we’re getting close to a 1.0 release of the spec). This is intriguing for jME because in addition to the embedded use in web browsers, Google and Mozilla are producing C(/C++) APIs to use their WebGPU implementations as standalone native libraries. We’ve had similar discussions in the past of using ANGLE to abstract the OpenGL API, and WebGPU could fill a similar role here.

There are multiple potential advantages to abstracting the native API on desktop alone - a consistent interface across platforms (no concern over OpenGL core/compatibility profiles, no more max OpenGL version of 3.3 on Mac), getting the best/preferred graphics driver for the platform with all the benefits that brings (both ANGLE and WebGPU abstract over Direct3D, Metal, and/or OpenGL/Vulkan). This abstraction could also make browser-based games made with jME all the more achievable without a lot of difficulty (see @RiccardoBlb’s demo of a minimal jME in the browser). I expect this could also be of benefit to mobile devs, though I’m not as familiar with that space so I’ll let others address that.

WebGPU brings another benefit - it’s far less verbose than Vulkan is and supports many of the same (or very similar) high-performance techniques. For comparison, consider the difference between a “Hello, World” triangle in Vulkan (~1900 LOC - not the most minimal example, but not unreasonably larger than is typical from what I’ve seen) and roughly the same in WebGPU (85 LOC). Despite the lack of verbosity, WebGPU supports a lot of modern high-performance GPU techniques such as DrawIndirect (which allows compute shaders to perform GPU-driven rendering and achieve some phenomenally fast results).

As others have pointed out, we need to have a clear end-goal and value proposition in mind or we’re probably going to end up wasting time. From what I’ve seen over the years, I’d be inclined towards a direction like the following:

1. Layered modularity. Fundamental modules (math, rendering interface, etc.) export packages/types/classes that can be used by higher levels, but do not depend on higher levels themselves. Behavior that must be provided by a higher level must be expressed via implementing an interface provided by the lower level.
2. Replaceable layers. The functions provided by each layer are expressed in interfaces which describe implementation-independent behavior, and the implementations of which can be swapped out at will (jME already does this to a large degree).
3. Data-driven design. Scenegraphs are difficult to learn and have a lot of caveats to using efficiently - especially for large scenes. They’re also likely to be sparse in memory, which never helps a modern CPU (not that in practice it’s usually terrible for a modern CPU because of larger caches). I’d suggest providing a core ECS-type interface to the engine (the “scene”) and modeling relationships such as parented hierarchies between entities via components and let the renderer implementation determine the most efficient way to cull/render - which in a GPU-driven architecture is probably going to be very different from a CPU-culled BVH. Taking this idea far enough could even result in a situation where (invisibly from the end-user’s view) the renderable objects’ data components are stored in native memory buffers and passed off to WebGPU compute shaders with very little overhead (a handful of CPU->GPU buffer copies, rather than many CPU->GPU buffer copies).

This might look something like the following:

Effect system (animation, audio, etc.) --------\ / → Audio Layer (OpenAL, WebAudio)
ECS (object transform, physical data, etc.) → | → Rendering Layer (Scenegraph/WebGPU)
Physics system ^
Pipelines ^

This is effectively a complete rewrite of jME, at least as far as scene/rendering handling goes (I hope the math/animation libraries could be largely re-used), and I think API compatibility is pretty well out of the question. I don’t see this as a problem - Unreal Engine, for example, tends to make major (and I believe largely breaking) changes between engine releases (UE 4 → 5 comes to mind). jME 2 → 3 was such a change and was appropriate for moving from a primarily fixed-function OpenGL scene to a shader-based scene. The native graphics capabilities have undergone just as seismic and conceptual a shift between OpenGL 2 and OpenGL 4+/D3D 12/Vulkan/Metal, so I think it’s appropriate that the engine changes again to be able to take advantage of those changes. I don’t see JME 4 being totally new and incompatible with JME 3 being any more problematic than JME 3 being incompatible with JME 2.

The last time this topic came up the questions all revolved around Vulkan, and as many here pointed out, Vulkan is incredibly verbose and time-consuming to develop with - and not a panacea for anything. WebGPU changes that picture because it allows expressing most of what Vulkan would allow with a tiny fraction of the code, and in a way that’s uniquely cross-platform. This design also isn’t married to WebGPU - one could just as well replace the WebGPU renderer with a native Vulkan, D3D, Metal, etc. implementation if inclined, but building the core “supported” renderer on WebGPU brings some unique advantages.

Thoughts?

Edit: Java’s Project Valhalla could very well have some major (and highly favorable) implications for a data-driven design.

[RiccardoBlb]

I’ve taken a close look at your thoughts and wanted to respond with a few insights:

1. OpenGL ES 3: It’s pretty clear that OpenGL ES 3 has become the go-to standard thanks to its widespread support. Focusing on just OpenGL ES 3 instead of the GL compatibility profile sounds like a smart move to minimize compatibility issues. But I totally agree, the potential of WebGPU to make compute shaders universally accessible is interesting.

2. ECS Performance: I’ve got some doubts about the whole ECS performance hype. Compared to other engines, jme already has pretty good performances, but there is definitely room for improvement, like looking into a better physics threading model. Maybe a more efficient physics engine could be the answer, or improving our current bullet implementation. It is possible, as i did it in the past for jme3-bullet-native, that might need a makeover for minie, which could seriously boost performance by giving it its own thread to bullet and making all memory transfer thread safe. Another reason for which i don’t believe in ECS is that we use multithreading on non native code, how are we supposed to predict threads executions, order, and memory allocation? And if we really think this can be a performance upgrade, we can just perform it transparently by building and maintaining update queues, just like we do for controls.

3. Blender to JME: I believe improving the Blender to JME support is pretty important. Back in the day, I played around with my own export format “f3b” (based to xbuf for who remembers), and it seriously made my workflow much better. Nowadays, we’ve got glTF that seems to be very good and is supported on many platforms. But it misses support for sounds, physics, and constraints. Since it supports extension, we could add those missing features and improve the user-friendliness of our engine a lot.

[danielp]

Thanks for chiming in! I appreciate your insights here.

1. With the caveat that I have very little knowledge of what’s supported by GL ES 3 (the last time I worked with anything related was a renderer for an internal commercial mapping application years ago, and I don’t remember if it was targeting GL ES 3 or an earlier version), I think the biggest value of the newer APIs is that (a) for the most part, everything is just a buffer (similar to some modern OpenGL versions, don’t know if ES 3 is one of them), (b) rendering is parallelizable via command buffers, and (c) compute shaders (which are powerful enough to do GPU-driven rendering). Command buffers open a lot of possibilities for design and for optimization that OpenGL’s state machine just doesn’t easily allow for.

2. I’m a bit puzzled by this one - I don’t believe I’ve ever heard of ECS designs being proposed as performance boosters vs alternatives. I fully agree with you that they’re not. The one and only exception I can think of is if the particular ECS implementation represented certain components (such as position vectors, bounding volumes, etc) in densely packed arrays (or native memory segments) then (a) maybe there are some common operations that could be accelerated by SIMD (available in Valhalla), and (b) the storage buffers may be able to be mapped directly to GPU memory. As you mentioned, multithreading is relevant here too, and a good ECS implementation will be conducive to multithreading in ways that a scene graph just isn’t by nature. (Java’s new virtual threads could be highly applicable here.) First and foremost, however, ECS to me just means a powerful and consistent way to model game data, and I think that it’s an industry norm (standard?) that presents a better interface to the engine than a lower-level construct such as a scene graph.

3. I do remember F3B and XBuf! I agree we could do more with scene formats (entity systems tend to be helpful here!). I recall @sgold investigating LWJGL’s Assimp bindings recently, and I think that would be a fantastic addition to our import situation. I’ve never dug into packing sounds with models, but a GLTF extension definitely sounds like a good starting point.

[bloodwalker]

The only time I heard of ECS as performance boost is with the Unity Engine and DOTS and to me it has created a confusion about ECS → Performance in development everywhere when it just not entirely true (It only works in Unity under certain conditions). It is also not a standard, since Unreal does not use ECS unless you get a plugin from the marketplace and many AAA games do not use it at all. I only heard so far that most MMORPG games and GTA use it since it eases development.

Scene graphs can work fine with ECS in most cases for indie projects, since you can have physics and game logic run on a separate thread and leave the main one for rendering and small things.

[Ali_RS]

@bloodwalker

ECS as performance boost

Well, I think it can, if used with Project Valhalla (Value classes). I believe that is when ECS can truly shine from a performance and memory footprint perspective.

[danielp]

@Ali_RS

Well, I think it can, if used with Project Valhalla (Value classes). I believe that is where ECS can truly shine from a performance and memory footprint perspective.

That’s pretty much the only situation where I would expect to possibly see a performance boost, but it could be a bit tricky to maintain that across other systems (thinking physics here). I don’t think it would degrade performance with non-buffer-of-values-aware code, but gains would likely be restricted to systems that were aware (in some sense) of that representation and able to take advantage of it. With that I see some risk of excessively tight coupling, but the trade-off may well be worth that risk.

[bloodwalker]

When it comes to rendering alone without a single piece of logic, I have not seen a significant benefit of ECS when it comes performance against a well written scene graph. It is on the game logic when you will see better performance in most cases (depends on the scale of the project). But the best benefit of ECS is that is easier to maintain and expand, so it becomes more dev friendly.

ECS is a tradeoff of performance at the cost of memory from what I was researching and testing. You will use more memory but you will use it a lot faster and project Valhalla or more native array types can make it even faster.

[Ali_RS]

@bloodwalker

When it comes to rendering alone without a single piece of logic

Yeah, I mean in the game logic.
For the rendering side, I have no idea how ECS would work.

[danielp]

@Ali_RS

For the rendering side, I have no idea how ECS would work.

This is what I meant by moving the scene graph into the rendering layer. The game developer would interact with the engine via the ECS, which would have built-in components for model, material, etc. The renderer would sync ECS changes to its internal data structures each “frame” (update tick). The advantage of this is that different renderers may have different structures that are optimal - if using a GPU-driven architecture, you’re going to do little (or no) BVH-culling CPU side, so the optimal data structure for the renderer might be a sparse grid (or even a flat list of everything in the scene). On the other hand, if using a more traditional renderer style, the best might be an octree or something similar. If renderers are to be pluggable implementations as suggested, I think it’s best to shift the burden of optimizing a renderable data structure from the game dev to the renderer implementation. The ECS would serve as an interface between the renderer and the game itself, with things currently modeled by the scene graph (parent/child relationships, local vs global transforms, etc) modeled by built-in components. This game structure vs subsystem structure is already present in jME with the physics engines - each one uses its own internal scene representation and builds its own data structures for simulation, and the jME “binding” code just syncs between physics-enabled scene graph nodes and the physics engine. My idea is to take that one step farther and do the same with rendering so that devs lose having to manage the scene graph (which can be tricky to do efficiently).

[danielp]

@Ali_RS

For the rendering side, I have no idea how ECS would work.

This is what I meant by moving the scene graph into the rendering layer. The game developer would interact with the engine via the ECS, which would have built-in components for model, material, etc. The renderer would sync ECS changes to its internal data structures each “frame” (update tick). The advantage of this is that different renderers may have different structures that are optimal - if using a GPU-driven architecture, you’re going to do little (or no) BVH-culling CPU side, so the optimal data structure for the renderer might be a sparse grid (or even a flat list of everything in the scene). On the other hand, if using a more traditional renderer style, the best might be an octree or something similar. If renderers are to be pluggable implementations as suggested, I think it’s best to shift the burden of optimizing a renderable data structure from the game dev to the renderer implementation. The ECS would serve as an interface between the renderer and the game itself, with things currently modeled by the scene graph (parent/child relationships, local vs global transforms, etc) modeled by built-in components. This game structure vs subsystem structure is already present in jME with the physics engines - each one uses its own internal scene representation and builds its own data structures for simulation, and the jME “binding” code just syncs between physics-enabled scene graph nodes and the physics engine. My idea is to take that one step farther and do the same with rendering so that devs lose having to manage the scene graph (which can be tricky to do efficiently).

Basically we’d just be making the “scene graph” a unique implementation detail of each renderer implementation.

[xuan]

Great initiative kick-starting the discussion.
I agree that starting from a clean slate without promising any API compat. is the way forward.

@danielp

This is what I meant by moving the scene graph into the rendering layer. The game developer would interact with the engine via the ECS

I don’t understand this suggestion, for me as a jme3 user the scene graph is just a convenient data structure to insert the things I want to render. I don’t usually think of culling algorithms as it normally “just works”.
The hierarchical structure makes it trivial to “stitch” multiple things together (at the view layer), e.g. a model, a particle emitter, sound node and done, from the logic is just one thing that moves all together, or I can quickly add an arrow geometry for debugging, etc. Although it’s true that I don’t need anything deeper that that.
Having an ECS based API would mean making it more opinionated by enforcing this architecture to users. It’s not necessary a bad decision but I would prefer having the flexibility to choose.
ECS is not the only way to decouple view and game logic, in my projects I don’t use ECS and I have 0 logic in the main thread.

I think a good scene api should promote decoupling view from logic (MVC style).

[bloodwalker]

With ECS frameworks like Zay-ES or Artemis you can use JME as the main app runner and and be in charge of rendering system in one. Physics, AI and others can run on separate threads without any JME wrapping and updating the components.

JME allows to easily put ECS in and just update the scene graph on the main thread; in most cases it is all you need. This makes the use optional instead of enforced because small games are easier to make with just appstates and controls (specially if you come from other engines like Unity).

[pspeed]

Beware the abstraction of an abstraction of an abstraction of an abstraction slippery slope.

One day this leads to an API to control the objects that control the ECS that control the scene graph that control the rendering. All because at each level someone found “the current layer” slightly distasteful to use.

[danielp]

@xuan

I don’t usually think of culling algorithms as it normally “just works”.
The hierarchical structure makes it trivial to “stitch” multiple things together (at the view layer), e.g. a model, a particle emitter, sound node and done, from the logic is just one thing that moves all together, or I can quickly add an arrow geometry for debugging, etc. Although it’s true that I don’t need anything deeper that that.

For most uses I would believe that this is probably the case. If you’re dealing with a very large scene (both spatially and in number of objects), however, you may need to be more intentional in arranging the scene graph so that large sections can quickly be culled. Preserving the ease of hierarchical relationships, adding effects (particle or audio), debugging aides, etc. is critical regardless of how the API is exposed to the user. Personally I’d prefer the scene API to be distinct from the rendering mechanism used so that it’s purely a logical arrangement of the scene with no culling/performance implications to its structure.

@xuan

Having an ECS based API would mean making it more opinionated by enforcing this architecture to users. It’s not necessary a bad decision but I would prefer having the flexibility to choose.
ECS is not the only way to decouple view and game logic, in my projects I don’t use ECS and I have 0 logic in the main thread.

I think a good scene api should promote decoupling view from logic (MVC style).

The decoupling is precisely what I had in mind by pushing for an ECS API being the primary means of interaction, but I hear what you and @bloodwalker are saying, and without a response from everyone along the lines of “of course it should be an ECS, what else would it be” then I agree with @bloodwalker that it should be optional. I think that as long as we preserve the ease of expressing object relationships and adding visual/audio effects and debugging aides/visual elements we’re in good shape, though I’d like to explore how/if we can refactor the API to more strongly encourage/make default a sort of MVC concept.

@pspeed

Beware the abstraction of an abstraction of an abstraction of an abstraction slippery slope.

I share your concern of excessive abstraction. I think the graphics API abstraction provided by WebGPU will justify that layer of abstraction, though I also think that we should avoid being permanently married to it - GPU technologies can change quickly, and I think we’d likely be better served in the long run if we expressed much of what game developers deal with directly in terms that are generic across renderer architecture.

There’s a “right size” to abstraction - I don’t think jME 3 hit it quite perfectly because of its tight coupling to OpenGL, but I also agree that 3-4 layers of abstraction is too many unless each layer adds significant additional value that cannot be otherwise achieved - and even then we should avoid that in core if possible. 1-2 layers for a renderer and scene API are probably appropriate, though I wouldn’t consider an ECS to be an abstraction but rather an architectural layer.

[pspeed]

Just a note: the one and only reason I’m using JME today and started using it 12 years ago or whenever is because it was nearly a 1:1 scene graph mapping to OpenGL. Everything was already familiar to me once I understood where/how that mapping existed. And I don’t mean that all of the constants were the same, etc… but my conceptual understanding of how things were wired to the driver matched exactly already. Mesh = draw call. Material = shader, Geometry = state+material+draw call

A scene graph is a useful rendering engine API. It is not not not a game object API. The fact that JME conflates the two for convenience is no end of confusion when things fall off a cliff.

…even Unity doesn’t really give you a “rendering scene graph”.

This is what I loved about JME. I could decide how my game objects were managed and I wasn’t locked into existing modes.

Should JME provide such a layer? Maybe. Will it suffer from all of the same problems of other similar layers in other game engines. Probably. Is it still useful anyway? Probably.

…but I hope we don’t lost the simplicity of a scene graph for organizing “what gets rendered”. Scene graphs are prevalent for good reason.

[danielp]

@pspeed

A scene graph is a useful rendering engine API. It is not not not a game object API. The fact that JME conflates the two for convenience is no end of confusion when things fall off a cliff.

@pspeed

Should JME provide such a layer? Maybe. Will it suffer from all of the same problems of other similar layers in other game engines. Probably. Is it still useful anyway? Probably.

Agreed - and the more I think about this, the more I think that what I’m hoping to accomplish is best solved by a scene graph with some tweaks to refactor out some of the awkward bits in the current one. After making the concessions that realistically need to be (or at least should be) made for convenience (I’m thinking specifically of user data here) I haven’t thought of a way to enforce not using scene graph nodes as game objects - and I’m not sure that should be enforced by the API anyway. I wonder if the best way to discourage abuse of the scene graph as game objects is to provide such an (optional) API and use it in tutorials.

I really appreciate everyone chiming in on this. Truth be told, I didn’t expect this much (or so positive) engagement on this topic since it’s been discussed a fair amount before. There are a lot of different ways to approach these problems (only a small handful of which I’ve personally considered in detail), each with unique trade-offs, and I appreciate the diversity of opinion.

For the moment, it seems the biggest make-it-or-break-it topic for everyone is the scene graph and how it interacts with game objects.

[xuan]

@danielp

I wonder if the best way to discourage abuse of the scene graph as game objects is to provide such an (optional) API and use it in tutorials.

I think the wiki is partly guilty for this common misconception:

[danielp]

True - though I’m hesitant to blame the wiki per se since I believe that was common practice in jME before some of the community members began discovering the issues inherent to not decoupling game objects from the scene graph.

[yaRnMcDonuts]

Very interesting discussion so far. I don’t really have much to add, but there is one thing that I am curious about:

@danielp

After making the concessions that realistically need to be (or at least should be) made for convenience (I’m thinking specifically of user data here) I haven’t thought of a way to enforce not using scene graph nodes as game objects

I’m curious what you mean in regards to user data here. You aren’t suggesting removing it entirely, right?

I know many devs dislike user data and avoid it like the plague since it can theoretically allow very bad practice if you use the spatial as a game object with userData for storing health/mana/etc etc.

However, I personally do use lots of user data for scene editing, since it is super convenient to set variables on a node with userData in an editor, and then when the game loads it reads those user data values and creates a decoupled game object using that data, at which point the user data can be cleared and thus the spatial is no longer storing any game data. (someone once recommended I replace this all with .json but that felt like it would be a totally pointless optimization since an empty node with UserData is effectively the same a a json file, but since its being stored in j3o it is harder for players (or me) to break in a way that can cause a crash)

I hope that we don’t worry too much about forcing decoupling game objects from the scene so much that we lose some functionality. I also think that, while this decoupling is very important, there are times that it is not quite as important, and even unavoidable, especially in more complex 3d games (for example, dynamic hitboxes or animations for a combat RPG are techincally game data that control gameplay, but they will always be inherently tied to a spatial, and while it would be possible to separate them, it seems like that would be taking it a step too far and would cause tons of problems).

Aside from that I have not much else to say, alot of this stuff is indeed a bit beyond my level of clear understanding to chime in on more, but I’ve enjoyed following this discussion and always am excited by other jme dev’s excitement to improve the engine!

[pspeed]

Here is a prime example where using the Spatials as game objects seems like a good idea but then ends up causing its own issues down the line.

For example, what if you already had a framework for writing game objects that had proper serialization, etc… then whether the spatial could store/load that or not would not be relevant.

The idea that “I want to save my user’s game” means “I want to save the position of every particle, every mesh, the material parameters, the font used, the bone positions, etc.” is really weird to me. It inherently creates backwards compatibility problems trivially that otherwise wouldn’t even be an issue. So you want to fix the texture of that goblin’s shirt? Too bad… all existing save games will still be loading the old one. Etc.

“I use the scene editor to make my levels and then save all of the spatials, textures, meshes, animation poses, material parameters, etc. in a giant j3o that I load later…” All of the same problems exist. We straddle the line between a general 3D editor and a game level editor and then game devs make levels that can’t be loaded later because j3o changed or their texture refs changed, etc…