[SOLVED] Garbage Collector Freezes

[Darkchaos]

Hey Guys,
I am currently reworking my game to be entity system based. After solving many problems the endboss appeared:
I have a huge Memory Consumption in combination with the GC Freezing everything for multiple seconds up to a minute (!) for little to no gain.

Now I know that ECS are demanding in terms of GC however that’s not what I expected. I know about an implementation where they used a component pool instead of permanent recreation.
On the other hand I guess this impact is just too heavy to be only that?

Doing the maths: 60 FPS x 1 Component à 40 Bytes (rough estimation of a transform) should only be 2.4kB/s of Memory, however I am rather seeing something like 50MiB/s, which is why I will be searching for memory leaks or something as you read this.

In addition to this the profiler shows a “low” gc time during that freeze but a high cpu time, shouldn’t it be the other way round?

Edit: Sorry that I tend to rather write a post than solving problems on my own, it’s just more easy
Profiling the right way showed:
java.util.concurrent.ConcurrentLinkedQueue$Node 1.287.972.144 B (90,8%) 53.665.506 (96,5%)
60 Mio Nodes taking 1.2GiB, now I need to find out where I am filling that queue.

Edit2: A detailed look at that class shows way less elements (only ~3Mio and only 64MiB), no clue why, however the callstack shows what I feared:
50/50 for two of my Transform-Related Systems:

The first one takes the spatial transform in postRender and applies that to the component. This is needed as I still use the BetterCharacterControl and this is the way to pick up physics events.
The second one invalidateChildren actually invalidates transform children when their parent is changed.

The problem with this? I only have one entity which currently is influenced by physics and only has one child.

The Stack further shows setComponent → entityChange → ConcurrentLinkedQueue.add → ConcurrentLinkedQueue.offer, so I don’t know if this might be me calling setComponent multiple times or if changing the entities each frame just sucks?

Edit:
Sorry for writing a monologue here. Reading the Docs would have helped more.
The issue was simply this:

public static void invalidateChildren(EntityId eId) {
        EntitySet set = Main.self.getStateManager().getState(EntityDataState.class).getEntityData().getEntities(new FieldFilter(TransformComponent.class, "parent", eId), TransformComponent.class);
        for (Entity e: set) {
              // some stuff
              invalidateChildren(e.getId());
        }

        set.release(); //this was missing!! read the docs more careful than I did

So by reading the implementation of entityChanges (Thanks to open source), I found out that it prepares that queue for each EntitySet registered. Then I thought: Why so many sets? And then… Especially since those sets where never read they were filled more and more…

I guess I have many other EntitySets however luckily enough they are stored in AppStates which currently are never removed

[FrozenShade]

You should also play with different garbage collectors algorithms. There is one which (in my opinion) fits real time games. Don’t remember which one it is, I will check it later at home, but I’m curious what your tests will show, if you’ll do some.

[pspeed]

@Darkchaos

Especially since those sets where never read they were filled more and more…

Yes, memory leaks suck… but I’m glad you found yours.

An EntitySet has a lot of overhead if you just want to grab some entities. Never mind the fact that technically it could even be not filled yet the way you are using it. It’s not the right tool for what you are doing.

In the case you showed, you really want to be calling findEntities() instead:
http://jmonkeyengine-contributions.github.io/zay-es/javadoc/zay-es/com/simsilica/es/EntityData.html#findEntities(com.simsilica.es.ComponentFilter,%20java.lang.Class...)

It returns an iterable over the IDs… which is what you are using anyway. So much less work done.

I assume this is a non-networked game or this is running on the server?

[Darkchaos]

@FrozenShade
That would be nice, yep. Currently the default GC is okay as well (one Collection after a few (3?) minutes of growth, no noticeable freeze), however things might change in the future.

@pspeed

I assume this is a non-networked game or this is running on the server?

Currently it is not networked yep, but if that would imply another way of doing I would consider designing it so networking is an option in the future.

And well, I just called EntityData.get and saw what autocomplete gave me

If I am only iterating over all entities over and over (e.g. not caring about applyChanges, getAdded etc), is using findEntities always better?
Like I have a System which is iterating over each related entity each frame, then I could use an EntitySet or I could re-find them each frame, I guess there an EntitySet is appropriate?

[FrozenShade]

try with this:
-XX:+UseConcMarkSweepGC -XX:+TieredCompilation

[pspeed]

@Darkchaos

If I am only iterating over all entities over and over (e.g. not caring about applyChanges, getAdded etc), is using findEntities always better?

Consider that EntitySet is likely calling findEntities() internally… but it also does a ton of other stuff that you are throwing away anyway.

There is no reason to create the whole infrastructure of an entity set just to throw it away.

The approach has negative connotations on the client of a networked game… but I’d argue that if you were doing this kind of logic in the view space that the design is broken in some other way.

[Darkchaos]

@pspeed

There is no reason to create the whole infrastructure of an entity set just to throw it away.

Well I think it might profit from being notified about component changes instead of reiterating over each entity. (Not the upper usecase but like the renderables system which could iterate over each entity).

@pspeed

The approach has negative connotations on the client of a networked game… but I’d argue that if you were doing this kind of logic in the view space that the design is broken in some other way.

Well this design represents Transforms/their inheritance, so this applies to the view, to physics or could even be something AI related.

[pspeed]

@Darkchaos

Well I think it might profit from being notified about component changes instead of reiterating over each entity. (Not the upper usecase but like the renderables system which could iterate over each entity).

If you have a set you will keep around a long time then, yes, use the set that’s designed to keep around for a long time.

If you have a set where you just want to grab the IDs and drop them when you are done then you use the method that lets you grab the IDs and then forget about them.

@Darkchaos

Well this design represents Transforms/their inheritance, so this applies to the view, to physics or could even be something AI related.

Physics… on the server.

AI… on the server.

View… not on the server… should not be doing complex per-frame entity queries to resolve its transforms. That’s unsmart. It should definitely not be calling findEntities() and then doing getComponent() on them every frame. That’s likely a new network request PER getComponent() call.

But I don’t even know why this is a thing. Applying transforms, even in the view, seems like it would be one pass through the “entities with transforms” and they you’d keep that around from frame to frame.

Though… I’m starting to feel like I’m deeply having a discussion about Oranges and you switched to Apples sometime back.

[Darkchaos]

@pspeed

It should definitely not be calling findEntities() and then doing getComponent() on them every frame. That’s likely a new network request PER getComponent() call.

But I don’t even know why this is a thing. Applying transforms, even in the view, seems like it would be one pass through the “entities with transforms” and they you’d keep that around from frame to frame.

findEntities only happens when a parent node is changed and I need to find it’s children (since there is no EntityId[] children field). Similar with the transformations themselves, the system will even only partially recompute the transform up to the “most parental” node which has changed. When a Transform isn’t changed, there is no computation, however when you have a walking character you have changing transforms.

Wouldn’t there be a local getComponent version on the client though and then frequent update packets containing all changes?

But I guess that all depends on the actual implementation and is probably completely unrelated when I am not doing networking.

Anyhow I now understood the different use cases for EntitySet and Set so I am onto the next adventures

[pspeed]

@Darkchaos

Wouldn’t there be a local getComponent version on the client though and then frequent update packets containing all changes?

You mean allllll changes. Like all potentially 1,000,000,000,000,000 components in your whole MMO database?

@Darkchaos

findEntities only happens when a parent node is changed and I need to find it’s children (since there is no EntityId children field). Similar with the transformations themselves, the system will even only partially recompute the transform up to the “most parental” node which has changed. When a Transform isn’t changed, there is no computation, however when you have a walking character you have changing transforms.

I still don’t understand this.

Even in view space…
for( all entities with a transform ) {
attach my spatial to the parent spatial
apply my local transforms
}

done.

[Darkchaos]

@pspeed

You mean allllll changes. Like all potentially 1,000,000,000,000,000 components in your whole MMO database?

Then rather only a few EntitySets containing only proximity information^^

@pspeed

I still don’t understand this.

Even in view space…

Well the above is how I reduce updates (when a parent changes you have to reposition all children obviously), and then: When the Character moves you have to re-apply transforms (and you have children to that character like his weapon, the Tags, Name etc)

However since it’s not networked yet it can stay until then at least

[Darkchaos]

Not directly connected but also not worth an own thread:
Is there a way to have a system run each time an entity has been deleted? Or maybe I’ll put that system inbetween each other system or even put the functionality into a BaseSystem, however that sounds like a bad bad hack.

The problem: With the implementation of Parental Transforms I have now many situations where a Parent has already been removed so the children got orphaned.
Easy Situation: I had a particle component following the rocket. As soon as the rocket exploded, it got removed. This lead to the particle being orphaned and should be removed.

Now I implemented a system which removes orphaned children by listening for the removal of TransformComponents. It then removes all children. The Problem here however is, this system is only run once a update, however when System A removed the rocket and System B wants to access the particle component, we have that crash.

Up until now I threw an Exception when I found an orphaned child because it would be something like a memory leak which is otherwise hard to debug, it has to be fixed asap. I could change that to a warning or even make my system just skip parents which aren’t there anymore, since I now have my Counter Measure which is something like a Garbage Collector for Transform Entities.

Technically after thinking about it (writing a post helps a lot) I should just rely on that system. The only “ugly” thing there is that for one frame the orphaned child will be placed oddly instead of being removed.

Or should I, instead of throwing the exception, manually trigger that “Entity GC”?
What happens if I delete an entity during a system is run, though? Does applyChanges prevent returning null components? I guess entityData.getComponent() will definately return null there?
I guess this would be another source of random NPEs

[pspeed]

First, to me, a rocket trail doesn’t sound like a game object. I don’t know why the server needs to keep track of the smoke particles for a rocket.

Second, if the rocket disappears and the smoke stays around (for however many frames), that sounds pretty realistic to me.

[Darkchaos]

@pspeed

I don’t know why the server needs to keep track of the smoke particles for a rocket.

Not the server, but the game needs to be able to reconstruct the whole scene just by entities.
And yes, one might argue about if a ParticleComponent is required, however it simplifies things which is the main use of a ECS, so why not have an Entity with a Physics and a ParticleComponent.

@pspeed

Second, if the rocket disappears and the smoke stays around (for however many frames), that sounds pretty realistic to me.

Not the “fire”, though the explosion upon hitting would hide that.
But I don’t know if that is no problem for other cases. I could however set it’s size to 0, which in term leads to an exception when inverting the transform

So I guess you would recommend just waiting one tick instead of risking to mess up the systems?
Possibly it’s also better to add proper null checks to the system.

[pspeed]

@Darkchaos

Not the server,

I use that as a point.

Game objects = server. If you think in client server terms.

That means your smoke trail would be managed on the server. It would be available in 2D map views, it would be available to AI, it would be available to anything really.

To me, a smoke trail seems like an effect in the 3D visualization, 100% view.

To me, it’s no different than if you were, on the server, trying to manage each individual spark particle from two swords striking. Maybe you would in your approach. I don’t know. That feels like a sufficiently crazy example, though.

@Darkchaos

however it simplifies things

Apparently, it doesn’t. On the one hand, you’d have just a visualization state problem. “This object is a missile and is moving therefore has a particle effect” where as now you have a whole game system ordering component, which thing is deleting which thing when… problem. (And even then I’d argue it isn’t a problem because in the 3D view the smoke trail should be a child of the missile node.)

[Darkchaos]

@pspeed

To me, it’s no different than if you were, on the server, trying to manage each individual spark particle from two swords striking.

Well at least currently I don’t have a server and use the ECS mainly to have an easy and reproducable “scene setup”, so I can implement some Paging/Zoning.

@pspeed

Apparently, it doesn’t. On the one hand, you’d have just a visualization state problem. “This object is a missile and is moving therefore has a particle effect”

This is true. What I meant with simplifiying things is that I don’t need another system just to handle missiles and it’s smoke/fire trails, but instead composing it of general purpose entities. I assume it would be more appropriate to have an “empty” component Missile or something which the View Systems process?

I don’t know if there are non-view related Transform topics though, only time will tell.

[pspeed]

Having A “missile” component that just handled smoke trails would be like having a “walk left leg” component that handled moving the monkey’s left leg and a “walk right leg” component that handled moving the monkey’s right leg.

It’s a missile. It will leave a smoke trail. That’s part of what it is. It’s part of the model. It does not need to be modeled as a game object at all… whether a missile tail object or a missile component.

At least to me.

[Empire_Phoenix]

What you might want to have however is a EnginePower/EngineFuel/RocketEngine component, that has the fuel level/burn time of the missile, and is used to determine if a trail is necessary, and if a flame effect applies.

At least if this is game relevant, and needs to be identical everywhere.

[Darkchaos]

Probably I should rename this thread since I ask many tiny questions which are probably not worth an own post here as well:

I’ve stumbled upon Java 8’s Stream API and I wonder whether I should let my systems iterate over the entities using a regular for loop or using .parallelStream().forEach().

The benefit of the latter is that it automatically takes care of multicore processors but might have more overhead compared to just multithreading on the system level.

It’s obvious that I can’t combine both: Running the systems multithreaded and then let the render thread wait until all worker threads are done or using this “mikro parallelization”.

If I am honest, I like the new Stream API since it is easy and takes away all possible concerns multithreading usually brings and it especially “balances” the cpu load (some systems might be much heavier than others)

So should I just use the Stream and let that be my multithreading or do you think regular loops might be better?

Edit: Oh and I just came up with another question:
Prior to the ES, I had a two level abstraction for my npcs: I had a control which walked to a Vector3f and I had a control which used a navmesh to feed in those Vector3f’s.
After what you said yesterday about abstractions, I wonder if it was better to merge those controls or to leave them as is (like: One System for Nav Mesh and one which sets the WalkDirection or maybe combine them?)

Or maybe have the NavMeshSystem set the WalkDirection but also have a non-nav-mesh-system which also sets the WalkDirection?

[danielp]

As far as the Stream api goes, I’d suggest utilizing whatever is easier for you to start with. Test it under load, and if it becomes an issue then change course. I’d be a bit surprised if the overhead was significant enough to cause trouble or even a major slowdown. Especially considering how tricky multithreading is without things like the Stream api, I think it’s likely to save you some time AND it also gives you the assurance of correctness.