I’m really interested in the concept of a generic paging system that is capable of handling any type of custom mesh, geometry asset, control spatial, etc that are registered with the paging system.I haven’t given this a ton of thought yet, but it seems completely possible and talk about an invaluable system for developing games.
The basic idea would allow you to register anything from terrain tiles, to npc/player controls to to be loaded/unloaded based on camera position (potentially direction if specified) and should probably handle level of detail based how the item is registered.
This is probably the wrong place to post this, however… I’d really like to hear other peoples thoughts on the idea… possibly start discussing the best way of implementing this sort of thing?
What prompted this was… most of the latest projects I have been working on have been dynamic terrain, vegetation, etc and each one has required… gasp… A paging system… and gasp… LoD handling. My game has… gasp Paging systems for players/npcs/items and LoD handling. These three are all based off of common serializable objects and are handled by a control, so they were all able to use the same system. But +1 for dynamic terrain… and +1 for vegetation… +1 for paging dynamic lighting (in this case, a lot of reusing)… +1 for paging effects/particle emitters. Anyways, the list potentially goes on and on. They are all performing the same basic functions with slightly different usage/parameters/handled objects… still… it’s the same freakin system.
Anyways… any other people see a need/desire/want for this sort of thing?
It could help define standards for implementing custom meshes (all registered components would require the use of an interface, etc, etc defining a build method, lod handling method, etc, etc. It could potentially help define standards for writing custom controls) and potentially make asset handling much easier on the game developer.
Thoughts?
Definitely interest there, as a proof of concept you could put the whole current terrain system in a Control like this and make it tileable too 
I was often thinking about how far one could move the whole terrain system in a control, this might be a nice proof of concept if the separate terrain geometry makes sense 
But actually thinking about it the LOD part etc… It could all just be a bit more generic controls that discover each other.
Edit: And note that callback/listener systems as in the TerrainGrid are rather untypical for jME, its a relic from another developer really.
We have LOD support for models (but only the Ogre3D tools can generate the data …). We have LOD support for heightmap terrain, it even calculates the error coefficient to maximize the LOD algorithm’s efficiency. We also have J3O based tile loading (right @Normen?). If you put all of that together, you got yourself something …
Also if someone is willing to port a LOD generator from C++ and add it to jME3 as a utility class, that would be great. That way we can also have LOD for Blender models and not just OgreXML.
@Momoko_Fan said:
We have LOD support for models (but only the Ogre3D tools can generate the data ..). We have LOD support for heightmap terrain, it even calculates the error coefficient to maximize the LOD algorithm's efficiency. We also have J3O based tile loading (right @Normen?). If you put all of that together, you got yourself something ...
Also if someone is willing to port a LOD generator from C++ and add it to jME3 as a utility class, that would be great. That way we can also have LOD for Blender models and not just OgreXML.
Yeah, this is LOD for single geometry though, I think @t0neg0d is more talking about general "stuff that depends on cam distance", right? Would make sense to tie that together also to get coherent overhead from that side (e.g. unified loading thread etc).
@normen said:
Yeah, this is LOD for single geometry though, I think @t0neg0d is more talking about general "stuff that depends on cam distance", right?
But in the end it ends up being handled separately by each component. LOD for terrain is different than LOD for models, and LOD for vegetation is different than LOD for particle emitters. You might want to handle such configuration globally, but currently there's no means to do so. For non-global scene elements like particles and models, these global settings would have to be applied at loading, and when the user changes it in the game settings menu it would have to somehow find all of these elements and apply the new settings.
@normen said:
Would make sense to tie that together also to get coherent overhead from that side (e.g. unified loading thread etc).
You can have a unified executor (from java.util.concurrent) that handles these sorts of tasks. For example, the terrain system uses such executor to handle terrain LOD.
Yep, exactly
@Momoko_Fan said:
But in the end it ends up being handled separately by each component. LOD for terrain is different than LOD for models, and LOD for vegetation is different than LOD for particle emitters. You might want to handle such configuration globally, but currently there's no means to do so. For non-global scene elements like particles and models, these global settings would have to be applied at loading, and when the user changes it in the game settings menu it would have to somehow find all of these elements and apply the new settings.
The nice part about this is they all still fall under types:
terrain & vegetation are both custom meshes (interface provided for registering custom meshes to paging system)
Particle emitters are probably specific to themselves (interface provided for registering particle emitter to paging system)
Imported assets are likely the most difficult, though perhaps instead of handling these as the asset themselves, an interface is provided for imported assets that are used by a control class?
Also, if the components are registered with the system, updating lod parameters on a global (registered type-specific of course) level could be as simple as a single method call.
Couple other thoughts about this approach...
1. It can help define and teach game developers best practices for setting up the scene graph, as each registered type would require a base node for either a) swapping in and out geometries... or b) swapping in and out nodes containing geometries.
2. It could help define best practices for implementing custom meshes. At a minimum, standard naming convention for build/lod methods.
3. It could potentially be used for minimizing the number of point light in any given scene... though this isn't directly tied loading/unloading meshes from a scene, it would fall nicely into cache management & the reuse of recent objects.
It could potentially be a one stop shop for management of in-game assets (models, custom meshes, point & spot lights, particle emitters, etc)
@Momoko_Fan said:
You can have a unified executor (from java.util.concurrent) that handles these sorts of tasks. For example, the terrain system uses such executor to handle terrain LOD.
Glad you mentioned this... as it was a question I had asked in another thread that is very relevant to this subject. The terrain system currently makes use of either the ThreadPoolExectuor or the ScheduledThreadPoolExecutor, correct? Is it creating it's own instance of this? Or does it require the user to define a single executor and pass this in as a parameter to the terrain manager? The reason I ask, is I was a bit confused as to the potential performance problems if the developer is using an executor that is defined at say... 2x the available processors and then makes use of the terrain system which creates a second executor using the same settings. Does this become a problem at some point? Like... if a community plugin is released with another feature the developer just has to have... and this plugin (yet again) defines another executor with the same settings. When does this become a problem?
I think you might be overthinking the executor problem. Usually, worker threads are idle… or there is something wrong. And generally it is a super bad idea to share executor services randomly since they might be configured in special ways for a particular set of workers. It might make since to have 2xCPUs for terrain but some other thing may only need two threads…ever… and maybe have its own custom worker threads, etc…
@normen said:
Definitely interest there, as a proof of concept you could put the whole current terrain system in a Control like this and make it tileable too :D I was often thinking about how far one could move the whole terrain system in a control, this might be a nice proof of concept if the separate terrain geometry makes sense :)
Interestingly enough, if done correctly anyways, this would be the first step in providing the OP idea just on a slightly smaller scale. The Control could allow the use of substitute terrain tiles (i.e. their own home-brewed custom mesh) but leveraging the extensive feature sets put into the terrain system that exists currently.
I'm definitely gonna start playing with this idea from this very approach... and at some point try and decouple the actual mesh to see what would be involved.
@pspeed said:
I think you might be overthinking the executor problem. Usually, worker threads are idle... or there is something wrong. And generally it is a super bad idea to share executor services randomly since they might be configured in special ways for a particular set of workers. It might make since to have 2xCPUs for terrain but some other thing may only need two threads...ever... and maybe have its own custom worker threads, etc..
Thanks... this answer that question. I've been really hesitant in trying to implement the use of either executor in a plugin environment for this reason alone :(
+1 for this
It would be really nice if it supported 3D paging not only 2D, but that might be hard to combine in a good way…
Just a few ideas off the top of my head (and I never tried to implement a system like this so they may be terrible ideas).
Maybe the way to do it is to define a Pageable interface. Register them with the paging system and then define 3 levels (near, mid, far (might need more levels but that’s a start)). The paging system just called the Interface with pageChange(oldLevel, newLevel).
Then for example lights and particles might just only attach themselves if newLevel is near, vegetation might get added to appropriate levels.
For handling it you could split the space into an octree and add the Pageables to the appropriate place in the tree. As the camera moves around that lets you quickly identify which pageables are affected and then call the appropriate pageChange method to let the object know.
Well i guess a generic system is possible, however it is a bit more complex than this. for example vegetation needs to be renderd batched, since else the object count explodes.
Yes, but each batch would be the Pageable.
@t0neg0d said:
Interestingly enough, if done correctly anyways, this would be the first step in providing the OP idea just on a slightly smaller scale. The Control could allow the use of substitute terrain tiles (i.e. their own home-brewed custom mesh) but leveraging the extensive feature sets put into the terrain system that exists currently.
I'm definitely gonna start playing with this idea from this very approach... and at some point try and decouple the actual mesh to see what would be involved.
We talked about this a bit internally and while keeping the general paging system compatible to generic Geometry using TerrainQuads for the actual terrain geometry is probably the best solution as things like picking etc. tie into that as well. TerrainGrid in general can largely be ignored / replaced by whatever you come up with though :)
An implementation on my side works on areas/tiles storing lists of asstes based on priority, highest prio for me has the navmesh, then comes the tile basics, and thereafter the details. Basics are visible all the time, details are completely turned on/off for a tile based on tile distance.
A paging manager updates the tiles (which ones should be loaded, which ones purged), these then initiate multithreaded asset loading working on one level after the other.
The assets(models/j3o objects) are just a shell with the ability to receive a loaded j3o object attached at some time in the future at a certain position (or just being loaded for future use, like animations, skeletons, etc.).
Only very little parts of the whole application though expect anything to be present, because you never know when it will show up, or be gone.
@t0neg0d My assetmanager uses two executor thread pools one with size 1, the other with size 2.
So far my tiles are quite simple geometry, with navmeshes around 200-500k per tile and base geometry around 50-100k. The detail objects are planed to use a LoDControl, which also dynamically loads more detailed lod levels.
Currently I am in the process to set up some heavy polygon scenes to stress test the base tile loading and rendering, therafter comes the details tests.
The pageable interface @zarch proposes seems ok to me, but I guess whoever is responsible for loading in your design should be able to differentiate what to load first, some things like backgrounds/scenery, etc, should be loaded before other stuff.
Would be great if jme would have some sort of general multithreaded paging & loding system out of the box…
Its hard to have something efficient as well as something that is general purpose.
LOD is something that is hard to do general purpose.
Its done differently if you’re doing it for heightmap terrain, mesh-based terrain, box world terrain, space simulation, cities/buildings, animated characters, artificial props, organic props, particles, dynamic lights, vegetation, trees, and so on. Each one of those things would need its own “LOD module”, each module with its own configuration and an API to allow user control. For really far view distances, you may even need to combine these “LOD modules” together so that they can batch their objects for higher efficiency. You also need to make sure these modules can work with the physics engine so that things don’t fall through when LOD levels change and such. If you’re doing terrain streaming, then you need to worry about how to load LOD data in and out. If you require some data to be generated offline, such as to create impostor images for trees, then you need to have some asset pipeline to process your assets to generate the LOD data each time the artist decides to change the asset.
Unfortunately its very complex to have a solution for each type of situation that you can encounter. This is why we rely on contributions like BioMonkey to create a “turnkey” option that does something specific that people can use in their projects.
Yes @Momoko_Fan, it would still be nice to have one source for the cam distance and one way to set the LOD radius etc. instead of having to twek 10 different systems. But its definitely true what you say that it would be lots of different areas to cover.
@Normen: Yeah but what does it mean? For particle emitters, LOD is max number of particles to display. For trees, its the distance at which to switch to use imposters. For terrain and models, you have N number of LOD levels, where N can vary between models. Users will want to tweak all of these parameters as the defaults won’t fit every single way that the LOD is used for.