I also added a new feature to it so that if you want to have it execute all Callables in a single execute() call you can setExecuteAll(true). By default it will just grab a single Callable per invocation of execute() so as to keep the OpenGL thread running smoothly. I can't imagine a circumstance where you'd be filling the queue faster than your FPS…if you did you're screwed anyway. 
as long as no threading, events/listeners are forced on me through jme i'm cool 
i have still not yet seen that the pros of doing extensive threading weighs up for the cons…
(edit: but i'll try looking at this with fresh eyes, i hate to be a conservative idiot)
Got the changes and have started moving forward with the conversion. I'm going to add a singleton GameTaskQueueManager class that holds references to named queues to make it easier for various components to get a handle to a queue. There will be defaults for RENDER and UPDATE queues and you can add your own as you wish.
The fast locking in the concurreny package can be good for games, that's true. But that doesn't answer where or when you should use it, and neither does it come without a price if you do decide where to put it. It also is a step back (imho) in terms of usability for some. There's a reason Sun once chose to go with synchronized{} blocks rather than individual locks and atomic objects in the first place. Therefore you should leave the choice open to game designers.
If people want to take multithreaded game design beyond having different non-interfering threads, and/or splitting some tasks over several threads, they better have a clear understanding of concurrency anyway. Us putting some locks here or there (or worse, everywhere) won't help them that much in the end, and it won't do any good to those who don't need it.
Not that I see much of a need for this for many people, it might help you keeping to videocard busy with OpenGL commands, if you make sure one thread is used purely for sending OpenGL commands instead of doing the updates. But how you do updates to your objects while these object or other object are in the render process, etc… that's a decision to make for each game. You can do with a few locks and render different parts while you update the others, or decide to synchronize the updates themselves, etc. etc.
To keep it on topic, a system like this will make it easier to have background threads, which is a much easier way to fill some of those spare cycles. You could also use it to solve irrisor's reentrance problem (with some overhead though).
The only real issue I see in jME regarding threads is invoking methods or instantiating objects (like loading textures, instantiating BloomRenderPass, etc.) that throw strange exceptions when being invoked outside of th OGL thread. If we can change those classes to check to see if they're being invoked in the graphical thread, for things that require it, and if it's not in the graphical thread, to pass the call off to the render thread (likely using this methodology). I will side with llama by default in this circumstance, since I don't believe any of that requires us adding any really synchronization or concurrency support outside of this methodology that is already being provided. Does that seem reasonable?
For example in this situation I have to do this right now:
Callable<BloomRenderPass> callable = new Callable<BloomRenderPass>() {
public BloomRenderPass call() throws Exception {
return new BloomRenderPass(game.getCamera(), 40);
}
};
Future<BloomRenderPass> future = game.update(callable);
BloomRenderPass brp = future.get();
But if we were to fix BloomRenderPass to handle making sure things get sent to the right thread internally it could be simply changed to:
BloomRenderPass brp = new BloomRenderPass(game.getCamera(), 40);
Is that a reasonable expectation?
A lot of us would instantiate the bloom pass in the render thread anyway, which also gives us more control over when things are created. I'm not against such a change if it was optional (otherwise you force everyone to use the queue), or if the class could know that it already IS in the render thread.
On the topic of switching to Callable, the refactor is done. From my CVS log:
Issue: Switching from RenderThreadExecutable to Callable
- Added new singleton queuemanager to provide common access to task queues.
- Converted existing uses of RenderThreadActionQueue and RenderThreadExecutable to Callable and the new queue manager.
- Added update and render task queue access to SimpleGame
- Added update task queue access to LWJGLCanvas (and thus to AWT and Applet)
- Pointed Queue access in StandardGame to the manager.
Of course it could be reduced to something like with an extra method…
My primary goal at the moment is to make a very nice looking progress loading screen when my game is loading. In order to do that I am trying to pull as much out of the graphical thread as possible and into other threads. My next step after that is to make a progressive loading universe that will load scene elements from disk as proximity to them is such they will need to be visible sometime soon and unload objects that are too far away to need to deal with anymore. These two scenarios offer a definite need for threading (sure it can be done single-threaded, but it would be a nightmare). Both should be possible now with GameTaskQueue, but I have yet to attempt the latter, we'll see how it works out. I want to have a passive loading thread that does the effort of loading things without putting any visible strain on the game itself. I think the more that can be pulled out of the graphical thread, the more plausible this will be.
darkfrog said:
...
My next step after that is to make a progressive loading universe that will load scene elements from disk as proximity to them is such they will need to be visible sometime soon and unload objects that are too far away to need to deal with anymore.
...
Take a look at llama's continous terrain demo. that's exactly what it does(except it does it only for terrain meshes).
Yeah, I remember looking at that back when he posted it. It was single-threaded though, right? This is going to be a pretty similar concept most likely, except a little more complicated since it's not just terrain and I'm going for a passive loading so as to keep the game running smoothly so you don't even notice the background loading.
If we only had BSP support or some other level format support it would be somewhat easier for stationary aspects. For dynamic aspects it's going to be the same no matter what.
darkfrog said:
Yeah, I remember looking at that back when he posted it.
Awesome, I guess I didn't look at it closely enough. Is this something you think would be useful in jME or do you think it should stay its own project? I would be interested in looking into it further and helping you develop it further if it can be utilized to provide everything I need. No need for me to start over from scratch if you are willing to extend (or let me extend) the functionality a bit for my purposes. Let me know.
Rather than make code changes to make the entire API "thread safe", a useful first step would just be to unambiguously mark those methods which are currently thread-safe – ie, those methods which are guaranteed not to call OpenGL functions. Tagging these methods requires no code changes, but would help alot.
For example: can I safely construct a scene graph in a separate thread? I assume that linking my new Nodes to the rootNode for the scene must happen in the update thread, but I assume that I can create my nodes, call setLocalTranslation (etc) on them in a different thread. Many jME classes have hidden "gotchas", though, where there are nonobvious camera updates (for example) that occur, and the camera updates trigger an OpenGL call, etc.
Another very useful thing would be for the 'Game.start()' methods to register the identity of the 'game loop' thread, so that we can have a canonical "isInUpdateThread()' method that will let us know if it is safe to do certain operations immediately. A related method might be on to add a callable to the update queue (for example) if we're not in currently executing in the update thread, otherwise to execute the callable immediately. This helps a lot in writing "thread safe" classes which can be reused in different contexts. If you (to be safe) always add tasks to the update thread, then you introduce at least a one frame delay into the update if you actually are currently executing in the update thread.
Look at StandardGame…I did about all of that already (and a lot more) there.
I've seen StandardGame. There are update queues, but there is no 'inGLThread' method, nor is there a method to intelligently schedule tasks (ie, execute immediately iff in GL thread, otherwise defer). StandardGame does seem to store the GL thread in StandardGame.gameThread(), but it is private and thus inaccessible to subclasses. Further, my point was that all the AbstractGame subclasses should store their GL thread away so that core code could depend in it being available.
Further, StandardGame interacts very poorly with StandardGameState – the superclass initializer performs camera updates which it is just not safe to do outside the GL thread (sadly). You have to do an evil hack with StandardGameState.initCamera if you want it to work at all:
protected void initCamera() {
try {
GameTaskQueueManager.getManager().update(new Callable<Void>() {
public Void call() throws Exception {
_initCamera(); return null;
}
})/*.get()*/;
} catch (Exception e) { /* this is fatal */ assert false : e; }
}
void _initCamera() { super.initCamera(); }
And calling 'onActivate' outside the GL thread is not safe, either (as the StandardGame demos do).
Well, there is no 'inGLThread' because unless you are specifically NEEDING to do something in the OpenGL thread it is my opinion you should stay out of it entirely. Why would you want direct access to the OpenGL thread anyway? If you NEED to do something in the OpenGL thread use the GameTaskQueueManager and if you get the Future back and just say future.get() it will lock that thread until the OpenGL thread completes the task in its queue.
It is my opinion StandardGameState should be removed entirely as everyone believes that it's intended to be used with StandardGame but has been around since long before StandardGame was created (that's another topic that will be addressed elsewhere though). If you're looking for something to start out with I would suggest using BasicGameState instead.
And calling 'onActivate' outside the GL thread is not safe, either (as the StandardGame demos do).
Not sure exactly what you're referring to (I don't have the source code in front of me at the moment though), but if there's something that StandardGame is doing poorly explain further so I can correct it.
The point of the 'inGLThread()' method is so you can do something like:
<T> T executeInUpdate(Callable<T> c) {
if (DisplaySystem.inGLThread())
return c.call();
else
return GameTaskManager.getInstance().update(c).get();
}
ie, execute the code immediately if you are in the update thread, otherwise queue & wait for it. Note that you can't safely do the latter if you are not currently in the update thread, since you'll deadlock. If you want to write 'thread-safe' code (which can be used either in a single-threaded or a multi-threaded context), then you need something like this.
In TestLoadingGameState (which uses StandardGame), we create both LoadingGameState and DebugGameState outside of the GL thread. DebugGameState calls several methods on DisplaySystem.getDisplaySystem().getRenderer(), which are likely to be thread-unsafe. LoadingGameState contains two such calls.