Am I following the “rules” of an entity component system?

@Empire Phoenix said: Well using a safe datastructure is around 90% of the complexity.

Then (not i speak not for zay es, but for my own es)
I basically have a barrier, that once per tick all systems intrested in doing this tick sync at a specific point in time.
At this point all entity removals additionas or changes take effect. And it starts again.
-> This means you have a perfect System isolation, a slow ticking system will not see it’s data change while it is working on it, instead it is using a snapshot taken at the system start.

Is this like pspeed’s applyChanges()?

By having this two things, a clean determined logic that ensures that even using multithreading I have a completly determined outcome and using threadsave structures (mich mostly boil down to immutables and ConcurrentHashmaps)

That is going to be the hard part for me, (keeping it clean). I have started over so many times because I end up getting so messy. It gets better every time though! I think I am confident now that I have a solid base to work from (thanks all the friendly people). So if I restart, I don’t have to go WAY back to the beginning.

You could also code against a interfaces for components and entities, that way you could always later switch the es below the logic without having to rewrite to much.

@Empire Phoenix said: You could also code against a interfaces for components and entities, that way you could always later switch the es below the logic without having to rewrite to much.

I have tried doing this before, but then I switched back to my newbie ways. I was unsure of how I should name my interfaces so that it was intuitive. This aggravated my obsessive compulsive tendencies and caused me to get out of the habit. I should probably try again now that I know a little more than I did before :slight_smile:

Coding is hard when you have psychological disorders. sorry for the TMI!

@pspeed said: Blindly using ConcurrentHashMap really isn't enough because now you will have a problem of inconsistent state.

The code that handles the threading and updates in Zay-ES is the trickiest part. It has also been described elsewhere in nauseating detail so I won’t regurgitate here.

The long and the short of it, each EntitySet is its own view of the entities. If you follow the rule that components are immutable (and only swapped out when changes) then each EntitySet is a fixed snapshot in time until applyChanges() is called which updates the contained entities to the latest view. This can be thought of as reissuing the entity set query but it’s actually way way faster than that because of internal optimizations.

But conceptually, that’s a way to think about it.

@Jarky each EntitySet is its own view of the entities I've read your concerns about ES, this is a little magic words to help you have your "ahha" moment quick.

EntitySystem concepts may be clear for you; its implementation may be not.

Here ZayES implementation use a kind of collection called EntitySet to manage a “view” of entities for its interesters (obsevers). And because components are immutable, the access to it via Entity are guarantied/ approved about available and consistent even in multi thread enviroment.

1 - Access compoments via a “view” or (considered) a “channel” called ChangeSet, in which all the operation of the EntitySystem are protected about the avaiable (if you just do this kind of access). Access in ZayES actually compose of 3 operations insert-remove-update(broadcast) basis similar to a common database! — Becareful, this is the main source of pain, even in ZayES currently!

2 - the immutable of component cause the “reference-level of isolation”, is a solution for atomicity in Java language, in which if a reference slot is replaced with another (commonly via assign operator) is a atomic operation of the language.
Ex: Entity.slot1 = compoment1 … Entity.slot1 = component2 …
to guaranty multi thread enviroment will not cause contention or inconsistent. Note that this is just one “simple” but “briliant” solution for the concurent problems. Ex: Trasactional model use different level of atomicity. Actor model use “local memory” mailbox to handle communication…You can try Gpars to experiments your own algorimth.

Those 2 above are 2 main featur2es of ZayES compare to others.

Naturally ES design try to eliminate random routines and access, and its restrict linking between components and other kind of “reference” from entity to component. In fact, it depend on you, as far as I know. ZayES are go futher by using best practise about immutable component. It’s fine, also ZayES is relatively small and efficient.

After reading ZayES source code recently, I’ve found out it not pure as in the beginning after integrate network facilities: It not consider “update beat” as its philosophy anymore, this is interesting!

It’s not mean at the very first commit, there is one regular “update beat” or a internal timer but there is not an internal listener.

http://code.google.com/p/jmonkeyplatform-contributions/source/browse/trunk/zay-es/src/com/simsilica/es/EntityComponentListener.java

This also marked by @paulspeed as the “wild” part of his design (beside of some another minors). :roll:

The ChangeQueue in other hand, it make me a littble bit disappointed. Again to keep ZayES is relatively small, it’s fine… or not 8-O ? .

First it’s the use of HashSet<Class>, the second is “types.contains( change.getComponentType())” is properly not very efficient? 3rd, Generic Type can confused the system?

[java]
protected class QueueChangeListener implements EntityComponentListener {

    private Set&lt;Class&gt; types = new HashSet&lt;Class&gt;();
   
    public QueueChangeListener( Class... types ) {
        this.types.addAll(Arrays.asList(types));
    }
   
    @Override
    public void componentChange( EntityChange change ) {
        if( types.contains( change.getComponentType() ) ) {
            add(change);
        }
    }
}

[/java]
After a while reviewing his code, i also tent to write my own ES with annotations, Guice and Guava for better scale, higher performance and better internal communicating. But it come with cost.

For short, there is no “rules” to follow. Be wild, and learn from others, share knowledge. That make you better indeed! Cheers,

1 Like
@atomix said: @Jarky After reading ZayES source code recently, I've found out it not pure as in the beginning after integrate network facilities: It not consider "update beat" as its philosophy anymore, this is interesting!

It’s not mean at the very first commit, there is one regular “update beat” or a internal timer but there is not an internal listener.

http://code.google.com/p/jmonkeyplatform-contributions/source/browse/trunk/zay-es/src/com/simsilica/es/EntityComponentListener.java

This also marked by @paulspeed as the “wild” part of his design (beside of some another minors). :roll:

It is sometimes necessary either for debugging or for infrastructure level services to have something like a global change listener that will receive all updates. Note that nothing in Zay-ES or Zay-ES Net actually uses either EntityComponentListener or ChangeQueue. Zay-ES Net uses EntitySet.applyChanges() like anything else.

In fact, client code can’t even fully expect this support to be there. It’s for “back end” services that already know what implementation they are dealing with. It would be very hard to inject in a subclass so I’ve included it where it is most efficient. (I should probably just remove ChangeQueue though as it’s confusing.)

An example of an infrastructure piece that could not be done any other way would be some kind of a federated back end. In this case you might want to tap into the raw events to spread them around through some kind of shared data structure. This would avoid requiring federated systems to requery data that they don’t “own” locally.

@atomix said: The ChangeQueue in other hand, it make me a littble bit disappointed. Again to keep ZayES is relatively small, it's fine... or not 8-O ? .

First it’s the use of HashSet<Class>, the second is “types.contains( change.getComponentType())” is properly not very efficient? 3rd, Generic Type can confused the system?

First, note that this code is not used anywhere.

Second, note that HashSet.contains() is O(1). That’s bound to be way way way faster than accumulating a bunch of events that you aren’t interested in because you don’t care about their components.

@atomix said: [java] protected class QueueChangeListener implements EntityComponentListener {
    private Set&lt;Class&gt; types = new HashSet&lt;Class&gt;();
   
    public QueueChangeListener( Class... types ) {
        this.types.addAll(Arrays.asList(types));
    }
   
    @Override
    public void componentChange( EntityChange change ) {
        if( types.contains( change.getComponentType() ) ) {
            add(change);
        }
    }
}

[/java]

@atomix said: After a while reviewing his code, i also tent to write my own ES with annotations, Guice and Guava for better scale, higher performance and better internal communicating. But it come with cost.

Well, if there is “better internal communicating” and “higher performance” then I’m interested. You’d have to provide more detailed information on real issues, though.

@Jarky said: Is this like pspeed's applyChanges()?

That is going to be the hard part for me, (keeping it clean). I have started over so many times because I end up getting so messy. It gets better every time though! I think I am confident now that I have a solid base to work from (thanks all the friendly people). So if I restart, I don’t have to go WAY back to the beginning.

Note: Zay-ES and Zay-ES Net are both based on ES code that has been in production on my own servers for about 2.5 years now.

I put it out precisely because it’s the way I would answer “what’s the best way to make an ES?” There are other ways but that’s the one I personally consider the best way or I would have done it a different way. :wink:

So at some point it may make sense to work from the other end and look at Zay-ES to figure out why I did things the way I did them.

First, i said i didn’t write it yet.

I’d like to write the ES framework which can be use in broader usecases than Zay-ES currently.

Issues: It’s also true you don’t use ChangeQueue and ComponentListener anywhere or just internal… you tell me its for expanding the framework, or intergate other piecies toward the direction that but you not recommend to do so. It’s perfectly fine, but under the lens of a software designer, I can’t ignore it should be called a unclean design, sorry.

<span style=“text-decoration:underline;”>It’s there a better way to use HashMap other than <Class></span> is not important anymore, but here you are if you insist:

I also think in a few places your code are confused and be written better:
[java]
protected void loadEntities( boolean reload ) {

    Set&lt;EntityId&gt; idSet = ed.findEntities(mainFilter, types);
    if( idSet.isEmpty() )
        return;

    // Note: we do a full component loop here just to
    // reuse the EntityComponent[] buffer.  We could have
    // just as easily called ed.getEntity() for each ID.
   
    // Now we have the info needed to build the entity set
    EntityComponent[] buffer = new EntityComponent[types.length];
    for( EntityId id : idSet ) {
        // If we already have the entity then it is not a new
        // add and we'll ignore it.  This means that some entities
        // may have newer info than others but we will get their
        // event soon enough.
        // We include this for the reload after a filter change.
        if( reload &amp;&amp; containsId(id) ) {
            continue;
        }
           
        for( int i = 0; i &lt; buffer.length; i++ ) {
            buffer[i] = ed.getComponent(id, types[i]);
        }
           
        // Now create the entity
        DefaultEntity e = new DefaultEntity(ed, id, buffer.clone(), types);
        if( add(e) &amp;&amp; reload ) {
            addedEntities.add(e);
        }
    }

    // I had a big long comment in AbstractEntityData worrying
    // about threading and stuff.  But the EntityChange events
    // are getting queued and as long as they aren't rejected
    // out of hand (which would be a bug) then we don't care if
    // they come in while we build the entity set.        
}    

[/java]

        if( reload &amp;&amp; containsId(id) ) {
            continue;
        }

What is it?

also the
http://code.google.com/p/jmonkeyplatform-contributions/source/browse/trunk/zay-es/src/com/simsilica/es/base/DefaultEntitySet.java#186

Compare to Aspect of Artemis, I have to tell I like Aspect more than this!

Anyway, i’m not telling it have to be perfect at the first time nor i have better solution; but doing all alone can cause deadly trap even for the greatest fighter. Did you ask the same question anywhere else but in your source code:

// I don’t see a way of avoiding it, though. Even if we purge
// them here we would have to do special case code not to send
// them on the server. I suppose we could reset the filter
// and apply all at once even on the server and just send
// the changes right then… but then we potentially have a
// threading problem that we didn’t have before. Right now
// only one thread is applying changes and sending those
// results out. So we’d still delay and we’d have to special
// case things in the send changes loop.
//
// We can always add a flag later, I guess… or just expose
// the one we’ve added here and let the loop see if the filter
// has changed before applyChanges() is called.

If you did, may be you can get some suggestion already?

@atomix said: http://stackoverflow.com/questions/6795025/in-java-is-it-possible-to-create-a-type-safe-map-of-classes-to-instances-of-thei

Not relevant to this case.

@atomix said: http://stackoverflow.com/questions/20272106/java-hashmap-generics-with-class-key http://stackoverflow.com/questions/3456491/hashmap-for-classes-not-objects

Actually, these aren’t either. They have nothing at all to do with performance and are about an entirely different use case so I’m not sure why they are here.

@atomix said: http://stackoverflow.com/questions/2625546/is-using-the-class-instance-as-a-map-key-a-best-practice

This is only a problem when you are hot-redeploying your classloaders but leaving app objects laying around. This is bad in general but I don’t think this covers this case here. Even if you were in a situation to hot redeploy a Zay-ES using app then you’d be recreating all of the objects, too… so everything old gets GC’ed and there is no leak.

@atomix said: I also think in a few places your code are confused and be written better: [java] protected void loadEntities( boolean reload ) {
    Set&lt;EntityId&gt; idSet = ed.findEntities(mainFilter, types);
    if( idSet.isEmpty() )
        return;

    // Note: we do a full component loop here just to
    // reuse the EntityComponent[] buffer.  We could have
    // just as easily called ed.getEntity() for each ID.
   
    // Now we have the info needed to build the entity set
    EntityComponent[] buffer = new EntityComponent[types.length];
    for( EntityId id : idSet ) {
        // If we already have the entity then it is not a new
        // add and we'll ignore it.  This means that some entities
        // may have newer info than others but we will get their
        // event soon enough.
        // We include this for the reload after a filter change.
        if( reload &amp;&amp; containsId(id) ) {
            continue;
        }
           
        for( int i = 0; i &lt; buffer.length; i++ ) {
            buffer[i] = ed.getComponent(id, types[i]);
        }
           
        // Now create the entity
        DefaultEntity e = new DefaultEntity(ed, id, buffer.clone(), types);
        if( add(e) &amp;&amp; reload ) {
            addedEntities.add(e);
        }
    }

    // I had a big long comment in AbstractEntityData worrying
    // about threading and stuff.  But the EntityChange events
    // are getting queued and as long as they aren't rejected
    // out of hand (which would be a bug) then we don't care if
    // they come in while we build the entity set.        
}    

[/java]

        if( reload &amp;&amp; containsId(id) ) {
            continue;
        }

What is it?

What is the issue? The comment for that ‘if’ was pretty clear, I thought.

When the filter is swapped out we only have to reload the entities that aren’t already complete. If the entity survived the purge then we still “have it”… therefore we don’t need to reload it.

@atomix said: also the http://code.google.com/p/jmonkeyplatform-contributions/source/browse/trunk/zay-es/src/com/simsilica/es/base/DefaultEntitySet.java#186 ...

Compare to Aspect of Artemis, I have to tell I like Aspect more than this!

Because Artemis what? Artemis (last I looked) throws away all of this flexibility by calling you instead of you calling it. You have much less control over anything and have to do a bunch of bridging classes to interface with JME. Plus the whole idea of “formal systems” is not really part of an ES.

So you will have to be more specific.

@atomix said: Anyway, i'm not telling it have to be perfect at the first time nor i have better solution; but doing all alone can cause deadly trap even for the greatest fighter. Did you ask the same question anywhere else but in your source code:

// I don’t see a way of avoiding it, though. Even if we purge
// them here we would have to do special case code not to send
// them on the server. I suppose we could reset the filter
// and apply all at once even on the server and just send
// the changes right then… but then we potentially have a
// threading problem that we didn’t have before. Right now
// only one thread is applying changes and sending those
// results out. So we’d still delay and we’d have to special
// case things in the send changes loop.
//
// We can always add a flag later, I guess… or just expose
// the one we’ve added here and let the loop see if the filter
// has changed before applyChanges() is called.

If you did, may be you can get some suggestion already?

Part of the thing about having a library based on 2.5 years of production code is that some things have already been tried a few times a few different ways only to discover bugs later. This method is one of those cases. In the original version there used to be a huge chunk of code commented out after many and various attempts to get it working right in the old update scheme. While I was happy to remove this dead code, I liked keeping the comment history because it reminds me of all of the issues I encountered and if I ever wonder “why do I set a flag and wait for applyChanges()?” then I can instantly remember the 12 or so reasons this is a good idea.

So I’m not sure what the question to ask is.

Bottom line: if you update EntitySet outside of applyChanges() then you totally screw up the calling code and/or make it tremendously more complicated than it needs to be. Conceptually, the query is no “reissued” unless you call applyChanges()… even if you change the parameters that is conceptually using for that query.

I can really only address actual stated problems. Making me look through my own code to guess what you might mean is not as productive.

I understand your comments is for you to remind why it’s doing like that.

The question “one thread is applying changes and sending those results out. So we’d still delay and we’d have to special case things in the send changes loop. We can always add a flag later, I guess…” is still bring me bad mood.

So the whole “changes watching system” of Zay-ES depend on this single thread to accumulate changes for the whole fullview. In my imagination, that system work like conccurent in one phase, then change mode and work single thread in this phase.

Presumes:

  • the remove, add operation is not too much (in common game) to make a stall. Need to be test. May be it’s not happen in Mythura, but i can happen in a push test.
  • the change event queue can work at the same time with entity set building without caution. Also need to be test, why guess here? I also guess that there are situations error can happen, one obvious is memory related. If those two allocated too much memory at the same time, memory can be drain in small system; also the race condition can be increase in which situation, did you experiment it already?
  • broadcast in single thread is a wise choice??? how can your network facilities compare to actor framework where their saccarify memory but messaging go concurent.

It can be extremely more complicated if I’m not doing like this? I don’t think so… even I don’t have a sharp argument yet. In any case, I don’t want to be a Nazy judging every line you wrote…I also do stuff the way i like if I don’t have to open source it… :slight_smile: now consider you have over 20 yrs developed Java and you teach your kid to use:

for (Foo obj:set){
if (_inthread_boolean && _another_set_contains){ // no side effect
continue;
}
}

instead of

Foo obj
while (iterator && …){
obj = interator.next()

}

Aspect and Filter are exactly you call it or it call you problem. They are opposite, Aspect declare interest of system over Entity. But as far as I remember you also never mention Filter as a element of the ES framework, but use them heavily inside of the Transaction change. Filter can cause thread problem if they want. Aspect in the other hand are just marked bit, or a kind of index and can also keep sync with the set.

… My bottom line: I reviewed your code for good purpose and find something (almost minors) that not suite my model of ideal ES. That’s why I kind of want to write up something else. Also this conversation is out of it initial purpose any way, I learnt a lot, so thanks and have a nice day.

@atomix said: I understand your comments is for you to remind why it's doing like that.

The question “one thread is applying changes and sending those results out. So we’d still delay and we’d have to special case things in the send changes loop. We can always add a flag later, I guess…” is still bring me bad mood.

So the whole “changes watching system” of Zay-ES depend on this single thread to accumulate changes for the whole fullview. In my imagination, that system work like conccurent in one phase, then change mode and work single thread in this phase.

Nope. Zay-ES has no threads of its own.

@atomix said: Presumes: - the remove, add operation is not too much (in common game) to make a stall. Need to be test. May be it's not happen in Mythura, but i can happen in a push test. - the change event queue can work at the same time with entity set building without caution. Also need to be test, why guess here? I also guess that there are situations error can happen, one obvious is memory related. If those two allocated too much memory at the same time, memory can be drain in small system; also the race condition can be increase in which situation, did you experiment it already? - broadcast in single thread is a wise choice??? how can your network facilities compare to actor framework where their saccarify memory but messaging go concurent.

It can be extremely more complicated if I’m not doing like this? I don’t think so… even I don’t have a sharp argument yet. In any case, I don’t want to be a Nazy judging every line you wrote…I also do stuff the way i like if I don’t have to open source it… :slight_smile: now consider you have over 20 yrs developed Java and you teach your kid to use:

for (Foo obj:set){
if (_inthread_boolean && _another_set_contains){ // no side effect
continue;
}
}

instead of

Foo obj
while (iterator && …){
obj = interator.next()

}

Are you suggesting the second way is better? I don’t really understand this. The first way is better for about half a dozen reasons.

@atomix said:

Aspect and Filter are exactly you call it or it call you problem. They are opposite, Aspect declare interest of system over Entity. But as far as I remember you also never mention Filter as a element of the ES framework, but use them heavily inside of the Transaction change. Filter can cause thread problem if they want. Aspect in the other hand are just marked bit, or a kind of index and can also keep sync with the set.

When you create an EntitySet you ask for a set viewing a certain set of components. So maybe you only want Position and Name or whatever. A filter can be used if you only want Position matching some criteria or Name matching some criteria or whatever. The different is that in my case I let you control when and what you query. In Artemis it seems like you give them some “System” and “some criteria” and they call you when they want.

@atomix said: ... My bottom line: I reviewed your code for good purpose and find something (almost minors) that not suite my model of ideal ES. That's why I kind of want to write up something else. Also this conversation is out of it initial purpose any way, I learnt a lot, so thanks and have a nice day.

I encourage to write your own then. You are welcome to relearn all of the mistakes all over again. :wink:

Thanks to everyone for all the great information. I have successfully done what I thought I could not do. It is probably very far from ideal, but I have learned so much. I have learned a great deal about multithreading in the last week. I took my experiments further and it actually worked. I was able to update my concurrent maps while reading them from another thread while producing the results i wanted. We’ll see how far I can take it from here :slight_smile:

I have stretched my experimental project quite far. I actually have a simple physics system now. Lots of dynamically generated cubes with position and velocity components that swim around within a boundary. I am now discovering the flexibility of ES and I am ready to add more experimental “stuff”.