BillboardControl and Controls in General

Development
1

Hi All,

thanks for providing us with this engine.

In order to learn about the engine and to get a feel of how it works, I am currently skimming through the code and, naturally, I stumbled over BillboardControl and controls in general.

While I absolutely believe that controls are a very useful concept, I also find some issues in the existing architecture/code that I think could be improved upon.

Perhaps you might want to discuss the below proposals.

First, controls are basically behavioural strategies of some sort. This is what I think of them when comparing the existing controls and the information presented in the wiki. As such, I think that these controls should be made stateless and thus should be made reentrant singletons in order to save “tons” of valuable memory on limited devices. The state itself could be externalized to the spatial. The spatial would then be passed as an argument to all methods called during calls to update/render/postrender and so on.
Adding controls to or removing controls from existing spatials would then be as easy as spatial.addControl(MyControl.class) and there would be no longer a need for cloning an instance of a control, effectfully reducing existing code. Of course, there would be the trade off having to look up the instance of the control via a control registry, but I think that this is negligible when comparing it to the fact that there will be much less instances of these controls floating around in the system. Especially when considering the GC overhead.

Second, when I look at the fixRefreshFlags() method of the BillboardControl, which in turn gets called by all of the rotate*() methods which in turn are invoked bycontrolRender() via a call to rotateBillboard(), I find that it calls upon spatial.updateGeometricState() every time the control is rendered. And it will then determine the root node from the spatial and make it update its world bounds. While this is correct, it adds tremendous call overhead when using “thousands” of billboards in a scene.

Even more so when considering that updateGeometricState() and getWorldBounds() will already be called by SimpleApplication#update() after that all spatials have been updated recursively.

What I propose is that the existing code found in BillboardControl#controlRender be moved to BillboardControl#controlUpdate so that it can be called during the update phase.

This would then allow you to simply remove the BillboardControl#fixRefreshFlags() method, as the geometric state and the world bounds will be updated during the call to the SimpleApplication#update() method prior to that RenderManager#render() is called.

But perhaps I have overlooked something and there is a good reason for having BillboardControl making all these redundant calls?

2

By singleton I mean that there should be only one instance of the control, and not the dreaded singleton pattern by assigning the instance to a static INSTANCE class field…

3

Nope. Controls hold data. Mostly they need to. They need state, they need the current camera, etc… Some controls even build out and maintain entire object hierarchies underneath the node they are attached to. 99% of controls could absolutely not be stateless. Many hold a LOT of spatial-specific info.

In fact, in some cases you can add the same type of control more than once for that reason.

It’s best to think of controls like extending an object without subclassing. In that case, you need all of the benefits of an instance.

4

re: billboard control, you have to do it on render so that all of the updates have happened first. Otherwise, if the hierarchy moves or the camera moves then your billboard will be pointing in the wrong direction.

5
@pspeed said: Nope. Controls hold data. Mostly they need to. They need state, they need the current camera, etc.. Some controls even build out and maintain entire object hierarchies underneath the node they are attached to. 99% of controls could absolutely _not_ be stateless. Many hold a LOT of spatial-specific info.

In fact, in some cases you can add the same type of control more than once for that reason.

It’s best to think of controls like extending an object without subclassing. In that case, you need all of the benefits of an instance.

Okay, I get this. Controls are components and extend the “object” by composition and they can be added more than once to a single spatial.

However, my case still stands. Would adding the same control twice or thrice or even more often than that, not relieve the system and the GC if there was only one instance of the same control class?

It will merely add additional management logic to the way controls are currently being handled, for example:

spatial.addControl(SomeCustomControl.class, “1”);
spatial.addControl(SomeCustomControl.class, “2”);

Of course, upon disable or whatever other control manipulating methods there are, one would have to know about the index key.

The actual control state could be retrieved from the spatial upon update or render, as so

AbstractControl#updateControl(Spatial spatial, String indexKey)
ControlState state = spatial.getControlState(this.getClass(), indexKey);
this.simpleUpdateControl(spatial, state);

AbstractControl#renderControl(Spatial spatial, String indexKey)
ControlState state = spatial.getControlState(this.getClass(), indexKey);
this.simpleRenderControl(spatial, state);
…

and so on.

Likewise, spatial.removeControl(Class<Control> …) would remove the control of the exact same type that was last added to the spatial, or, by using the index keys,

spatial.removeControl(“2”), or even spatial.removeControl(SomeCustomControl.class, “2”);

That being said, knowing only about the index key and no longer having to remember the reference to the control instance itself, would greatly reduce upon internal fields and redundantly held object references. And it would also help to overcome memory leaks in the application itself or a framework thereof.

I know that this might imply (major) changes to the code base dealing with j3o serialization and deserialization and it would certainly break existing applications, however, with a future jme4 release such a compatibility breaking change would not be so much of an issue, or would it?

6

But how would the stateless controls hold state? Controls REQUIRE there own state. Any control that does not hold it’s own state is an anomaly, really. Most do.

Really, you are arguing over a few hundred Java objects in a system of thousands and thousands and thousands of objects. With your approach you lost almost every benefit of controls and gain basically nothing but a few less bytes in RAM.

7

Also, what GC are you trying to save? I sort of get the feeling you may not really understand what is going on.

Generally, controls are added to a spatial for the life of the spatial. There is no GC burden because it is never GC’ed. And when it is then it’s because the spatial is also being GC’ed… and once again the controls are pretty small potatoes now in the midst of everything else that is getting cleaned up. And relatively speaking, this happens rarely.

Given your proposed solution of passing strings (really?) as indexes, I’m starting to believe you may not know Java that well. I’m not sure what language you are coming from but you seem to be harboring a lot of misplaced information.

8
@pspeed said: re: billboard control, you have to do it on render so that all of the updates have happened first. Otherwise, if the hierarchy moves or the camera moves then your billboard will be pointing in the wrong direction.

Hm, as far as I have been able to gather from the existing code, the camera represents a first class citizen. However, considering that the cameras’ viewing direction etc. might be changed during update, for example a camera attached to a vehicle or a character, this is indeed a problem.

How about making Billboards first class citizens, too, by filtering out all Billboard controlled spatials during a first phase update and then, in a second phase, update all of the previously filtered out billboards?

E.g.

SimpleApplication#update
…
rootNode#update
rootNode#updateBillboards
…

and, e.g.

Node#update

for control in controls:
if control instanceof BillboardControl.class:
continue
control.update()

for child in children:
child.update()
…

Node#updateBillboards

if this hascontrol BillboardControl.class:
this.getcontrol(BillboardControl.class).update()

for child in children:
child#updateBillboards

This would allow you to get rid of the redundant updates to the geometry and the world bounds, or would it not?

9
@pspeed said: But how would the stateless controls hold state? Controls REQUIRE there own state. Any control that does not hold it's own state is an anomaly, really. Most do.

Really, you are arguing over a few hundred Java objects in a system of thousands and thousands and thousands of objects. With your approach you lost almost every benefit of controls and gain basically nothing but a few less bytes in RAM.

Basically I propose that the control state is external to the control instance and is stored along with the spatial to that they have been assigned to, e.g.

class Spatial

private Map<ControlKey, ControlState> controls;

addControl(Class<? extends Control> controlClass, String instanceKey) { … }

…

And where ControlKey is

class ControlKey implements Comparable

Class<? extends Control> controlClass;
String instanceKey;

And where instanceKey is either assigned automatically or based on user provided keys. Duplicate assignment of instanceKeys per controlClass must raise an error, whereas duplicate assignment of instanceKeyS for different controlClass will not.

ControlState is either a Map or an interface providing a key-value store type of interface or a more specialized interface, or whatever feels right for implementing such a generalized data store. The main concern is that is can be serialized without having to introduce some kind of object (relational) mapping, unless that the user wants it to, but then again, this should be part of the application’s framework and not any of JME’s concern.

E.g.

interface ControlState

Object get(String key)
void put(String key, Object data)

or similar.

So whenever a now singleton instance of a control is being called upon, it will

CustomControl#update(Spatial spatial, String instanceKey)

// this might return a copy of the state
ControlState state = spatial.getControlState(this.getClass(), instanceKey);
// update state
// optionally set the state
// spatial.setControlState(this.getClass(), instanceKey, state);

and so on.

Internally, when accessing its controls on either update or render, a spatial/node would then

for ControlKey key in controls:

Control control = ControlManager.get(key.getControlClass);
control.update(this, key.getInstanceKey());

and so on.

10
@pspeed said: Also, what GC are you trying to save? I sort of get the feeling you may not really understand what is going on.

Generally, controls are added to a spatial for the life of the spatial. There is no GC burden because it is never GC’ed. And when it is then it’s because the spatial is also being GC’ed… and once again the controls are pretty small potatoes now in the midst of everything else that is getting cleaned up. And relatively speaking, this happens rarely.

Given your proposed solution of passing strings (really?) as indexes, I’m starting to believe you may not know Java that well. I’m not sure what language you are coming from but you seem to be harboring a lot of misplaced information.

What I am referring to are thousands of billboards in a given scene, for example plants or even thousands of characters in a given scene, all being rendered using animated billboards. And all of these objects are also assigned an AnimationControl, an AIControl, and a PhysicsControl.

This makes 1000*4 of control objects, which have to be GCed at any point in time.

For granted, the majority of the available memory will be consumed by the assets.

Consider this:

A game is made up of multiple scenes. Each scene has at least 10000 scene graph objects, including Billboards, and each scene graph object requires at most 3 controls of the same type.

Of course, for each control there might exist a ControlState object which would have to be instantiated with each spatial. In the best case, only 3 controls would have to be instantiated, with zero ControlState. In the worst case, 3 control objects + 30000 control state objects would have to be instantiated.

As a side effect, all serialization concerns have been moved to the spatial, making it even easier to implement custom controls as one would not have to bother about serialization any more.

11
@axnsoftware said: What I am referring to are thousands of billboards in a given scene, for example plants or even thousands of characters in a given scene, all being rendered using animated billboards. And all of these objects are also assigned an AnimationControl, an AIControl, and a PhysicsControl.

You are doing it wrong. Already 1000 objects alone in the scene pushes the max. By 2000 objects your graphics bus will be crawling to a stand-still. And this has nothing to do with controls. Writing games is hard. Writing 3D apps is hard. You can’t just throw everything in and hope it works. Care and thought must be taken at every step of the way.

Plus, you propose moving the state out of the control into something else so you save nothing. Even worse, now you have Map overhead. What have you gained? Less than nothing, really.

It’s funny because by the time you’ve removed the state from the controls they don’t take any memory anymore. You’ve way complicated the code for no reason… and even picked up several downside aside from that. For example, now every control update() call will need to dereference its state.

You seem not to really understand how memory works. There is some kind of a law of conservation here that is appropriate, I think. You can’t make memory disappear by moving it around.

12
@pspeed said: Really, you are arguing over a few hundred Java objects in a system of thousands and thousands and thousands of objects.

Actually, in the Java world this might make a difference in that the less short lived objects created, the better the performance and overall memory footprint. Considering a game where a lot of enemies with at least 3 or more controls added to them will be spawned during an intensive fight, the less objects that have to be instantiated and subsequently will have to be deallocated, the more responsive the game would be, or would it not? Even more so on memory limited devices such as Android based phones, tablets or consoles. The same holds true for iOS based phones and tablets, of course.

And, considering these massively spawned enemies, a proficient developer would optimize so that the game’s framework would be reusing existing instances whenever possible.

13
@axnsoftware said: Actually, in the Java world this might make a difference in that the less short lived objects created, the better the performance and overall memory footprint. Considering a game where a lot of enemies with at least 3 or more controls added to them will be spawned during an intensive fight, the less objects that have to be instantiated and subsequently will have to be deallocated, the more responsive the game would be, or would it not? Even more so on memory limited devices such as Android based phones, tablets or consoles. The same holds true for iOS based phones and tablets, of course.

And, considering these massively spawned enemies, a proficient developer would optimize so that the game’s framework would be reusing existing instances whenever possible.

I’m being trolled, right?

14
@pspeed said: You are doing it wrong. Already 1000 objects alone in the scene pushes the max. By 2000 objects your graphics bus will be crawling to a stand-still. And this has nothing to do with controls. Writing games is hard. Writing 3D apps is hard. You can't just throw everything in and hope it works. Care and thought must be taken at every step of the way.

Plus, you propose moving the state out of the control into something else so you save nothing. Even worse, now you have Map overhead. What have you gained? Less than nothing, really.

It’s funny because by the time you’ve removed the state from the controls they don’t take any memory anymore. You’ve way complicated the code for no reason… and even picked up several downside aside from that. For example, now every control update() call will need to dereference its state.

You seem not to really understand how memory works. There is some kind of a law of conservation here that is appropriate, I think. You can’t make memory disappear by moving it around.

Hm, I don’t get it. I proposed optimizing the way billboards are being rendered as they do present a bottleneck during the actual rendering phase. Whether there are 10 or 100 or 1000 or a multiple thereof, they will cause absolutely redundant calls for updates being made to the model’s state during rendering.

And, externalizing state from an otherwise stateless and thus reusable piece of code is not that bad and has been done before, see for example AppState >D.

In my example, I also provided you with a best case and a worst case scenario. In the best case scenario, where none of the controls have any extrinsic state, only 3 control objects would have to be instantiated as opposed to the old model, where at least 30000 of these objects would have to be instantiated.

15
@pspeed said: I'm being trolled, right?

Hm, read above posts and reconsider. I am trying to make a good point here.

Are you the only developer around here?

I mean, the Mythruna prototype is great, but I’d rather would like to have the other developers’ opinion on that topic, if there is any.

16
@axnsoftware said: Hm, I don't get it. I proposed optimizing the way billboards are being rendered as they do present a bottleneck during the actual rendering phase. Whether there are 10 or 100 or 1000 or a multiple thereof, they will cause absolutely redundant calls for update being made to the model's state during rendering.

How is it redundant? Each spatial with a billboard control will need to be updated to face the camera using whatever parameters the billboard control has. You have to do it for every spatial to which the control is attached.

@axnsoftware said: And, externalizing state from an otherwise stateless and thus reusable piece of code is not that bad and has been done before, see for example AppState >D.

But the state is per spatial, per control. Please explain how the lifecycle of the state somehow changes when you move it out of the control into more expensive data structures?

@axnsoftware said: In my example, I also provided you with a best case and a worst case scenario. In the best case scenario, where none of the controls have any extrinsic state, only 3 control objects would have to be instantiated as opposed to the old model, where at least 30000 of these objects would have to be instantiated.

Can you provide an example of a control that requires no state? I can provide many dozens of examples otherwise.

I’m definitely being trolled… and I don’t really have time for these troll-like discussions. Please read about generational GC, look into the existing controls, understand more about Java memory management in general, and rethink what you mean by “short term” and “long term” in the context of an engine that’s pushing data the GPU at 60+ frames per second.

Also, think about how silly an unoptimized 3000 blades of grass is and what that is doing to the GPU bus. There are many articles on how to build an efficient 3D scene and I do not have time to educate you here.

17

If you have “thousands of billboards” you’re already doing it wrong… No CPU-side optimization will be able to fix the kind of performance issues you will encounter with thousands of billboards or thousands of any sort of geometry for that matter.
The GPU (graphics processor) used in your phone or PC will grind to a halt once you tell it to render thousands of 4-vertex meshes, and update a matrix uniform for each of these meshes to boot.

The only option remaining is to perform billboarding on the GPU via the vertex shader and send all billboards as a single geometry. There are many examples of this online … Typically you would determine where the vertex should be relative to the billboard and then multiply the corner by the camera’s left and up vectors to get the model space vertex position.

18
@axnsoftware said: WTF? I am trying to make a good point here...

Then you just don’t have much of an idea of what you are talking about and I don’t have time to teach you. Even in the post I quoted, you make a few false assumptions and then negate it all in the last sentence anyway.

19
@pspeed said: How is it redundant? Each spatial with a billboard control will need to be updated to face the camera using whatever parameters the billboard control has. You have to do it for every spatial to which the control is attached.

But the state is per spatial, per control. Please explain how the lifecycle of the state somehow changes when you move it out of the control into more expensive data structures?

Can you provide an example of a control that requires no state? I can provide many dozens of examples otherwise.

I’m definitely being trolled… and I don’t really have time for these troll-like discussions. Please read about generational GC, look into the existing controls, understand more about Java memory management in general, and rethink what you mean by “short term” and “long term” in the context of an engine that’s pushing data the GPU at 60+ frames per second.

Also, think about how silly an unoptimized 3000 blades of grass is and what that is doing to the GPU bus. There are many articles on how to build an efficient 3D scene and I do not have time to educate you here.

Hm, you do not have to educate me here and you I must persist that you must no longer continue to insult me. I do not know, but did I ever insulted you? I think that I made it clear in my original post, that the proposals are open for discussion and not open for any insult.

What is wrong with you? Have you not taken your meds?

Perhaps you might want to actually read the posts I made in full before insulting me again.

20

Wow, how ignorant can one be. An example, nothing more and nothing less.

I do not want you to teach me, I want you to realize the fact that BillboardControl#renderControl is eating up valuable resources during rendering. Nothing more and nothing less. Which, when properly done, would be realized during the update phase and not during the rendering phase.

And since you have made this your personal crusade, and begun to insult my patience, I challenge you to prove to me that the BillboardControl, regardless of whether there are 10, or 100, or even 1000 or more billboards in a scene, does not impose a negative impact on scene rendering by having it make redundant updates to the spatial’s geometry and root node’s bounds during rendering.

Again, how ignorant can one be?

Is this JME3, does pspeed really stand for JME3? Is this sucker really a representative of an engine that is being used globally?

I mean, I did not insult anybody in the first place and this sucker continues to insult me over and over again.

Get rid of that gay and JME3 will prosper.