My ES in contrib: Zay-ES

pspeed · April 11, 2013, 1:15am

@zzuegg said: I want to keep track of all owners of an item. As said previously i am kind of bound to n:m relationships and this addition allows easy implementation of such queries. For this i have created a component [java] class Owern{ EntityId item; EntityId owern;
Date since;
}
[/java]

Then i can use:
[java]
EntitId someItem= some entitiy
//returns all entities with component UserInformation.class related trough Entities with Ower.class component.
EntitySet history = Database.findRelated(someItem,Owern.class,UserInformation.class);
history = All historical owerns of the item
[/java]

And it works also the other way:
[java]
EntitId user= some entitiy
//returns all entities with component Armor.class related trough Entities with Ower.class component.
EntitySet history = Database.findRelated(user,Owern.class,Armor.class);
history=All armors which ‘user’ ever had.
[/java]

If I understand what you are trying to do… there is no efficient way to model this query (there will really never be an efficient way to do a ‘give me all entities in this set’ query). Therefore component filters don’t really support it. because ultimately it’s trying to amount to a database where clause if it can.

How often do you do queries like this? I’m guessing it’s probably not a once a frame thing, right?

Edit: P.S.: I’m curious to know more about how you are actually using this information. I’d like to try to think about how I would do it if I were in your shoes.

zzuegg · April 11, 2013, 11:12am

You are right, its not a once per frame thing. In conventional SQL databases this would be a ‘x JOIN y JOIN z’ opration, but considering a graph database such queries are probably possible in real time.

With the ‘added passed entity’ the cost for such a operation would be ~twice the cost of a simple query. It’s a two pass query. First get a HashSet with all relationships. Then get all Entities with the need components and apply the HashSet.constains(EntityId) filter.

It depends, in the book keeping application i am developing need such queries mostliy in the evaluation part such as ‘Available Stock’, ‘Used/Bought Substances’. Of course there is no need for realtime.

Following the Game example above i would like to implement such queries for different things:

a) Security. In games like D3 or similar ‘duping’ Items is always a problem. The best known Dupe method is to give all your items to a second account and then claim your account was hacked. In general you have get your account restored once. But if you have high end items in diablo 3 the account could be worth a few hundred dollars.

With a full history of an item available, it would be possible to simply undo the transactions, resulting in no dupes.
Adiditonally, duped items due a exploit or something could possible get tracked down.

b) Immagine a player fulfills a very epic and hard event. All items the player ever had could get a little stats increas and an engraving like ‘This sword was held by the epic pspeed, creator of the Zay-ES’

c) In a street system with waypoints, all crossings would be waypoints with 2 or more streets, you already have a n:m relationship

In general, it’s not that such query’s are essential for a game, (If they where, you probably would already needed them). But if such a ‘simple’ modification of the system allows a wide bunch of new features (Every n:m operation) it might be worth considering it.
For a data-driven application such queries are a must. All workaround with the current component filter are more a dirty hack then a clean solution.

zarch · April 11, 2013, 12:02pm

Honestly I’d store historical information out to an SQL database and just keep “live” data in the entity system.

SQL is excellent for keeping, organising, search, and correlating historical data - use it for its strengths.

And yes - your first defence against dupe is to give each object a unique identity with defined creation and keep track of its movements afterwards. You should also do the same with money. Money is only ever transferred between accounts, never created out of thin air.

You then have specific “source” accounts which are allowed to go negative - but again you can see the balance of those accounts and where the money from them went so at a glance you can see things like where money is entering the system and if someone has more money than it should where it came from.

pspeed · April 11, 2013, 1:48pm

@zzuegg said: You are right, its not a once per frame thing. In conventional SQL databases this would be a 'x JOIN y JOIN z' opration, but considering a graph database such queries are probably possible in real time.
With the ‘added passed entity’ the cost for such a operation would be ~twice the cost of a simple query. It’s a two pass query. First get a HashSet with all relationships. Then get all Entities with the need components and apply the HashSet.constains(EntityId) filter.

It depends, in the book keeping application i am developing need such queries mostliy in the evaluation part such as ‘Available Stock’, ‘Used/Bought Substances’. Of course there is no need for realtime.

Following the Game example above i would like to implement such queries for different things:

a) Security. In games like D3 or similar ‘duping’ Items is always a problem. The best known Dupe method is to give all your items to a second account and then claim your account was hacked. In general you have get your account restored once. But if you have high end items in diablo 3 the account could be worth a few hundred dollars.

With a full history of an item available, it would be possible to simply undo the transactions, resulting in no dupes.
Adiditonally, duped items due a exploit or something could possible get tracked down.

This can’t really happen with an ES because the object/entity is already unique. As long as the player has no ability to clone an entity then one cannot be created from thin air. And if they do have the ability to clone an entity then that’s the problem and history won’t solve it.

@zzuegg said: b) Immagine a player fulfills a very epic and hard event. All items the player ever had could get a little stats increas and an engraving like 'This sword was held by the epic pspeed, creator of the Zay-ES'

This is a nice example. And a one time rare query. And as it turns out, not a join because just getting “all of the items a player ever owned” is enough.

For these rare queries, I would not use an EntitySet at all. There is overhead and you have to remember to clear them or you build up garbage. After all, in the background they are managed by the ES to keep them up to date.

Do one EntityId query to get one set of results then do another EntityId query to get the other set of results… then use set.retainAll() to get the intersection of the two. That’s actually what the ES is doing internally to combine the component query results so you are no slower.

@zzuegg said: c) In a street system with waypoints, all crossings would be waypoints with 2 or more streets, you already have a n:m relationship

This is a conventional query. The relationships are represented by entities and you ask for all entities with head = streetCorner1 OR tail = streetCorner1. You could even write a Guava function to allow you to transform the set into just the other endpoints. (I actually have a bit of pedigree here with graph operations. One of my other projects: http://filament.sourceforge.net/)

In fact, if you make a Guava function as described and make it fstep compatible then you can use fstep to do the traversals for path finding (no A* built in but it’s easy to add). Out of the box, it provides shortest path, priority traversals, etc…

@zzuegg said: In general, it's not that such query's are essential for a game, (If they where, you probably would already needed them). But if such a 'simple' modification of the system allows a wide bunch of new features (Every n:m operation) it might be worth considering it. For a data-driven application such queries are a must. All workaround with the current component filter are more a dirty hack then a clean solution.

I would still post filter them as I describe. Or as zarch suggests, shunt history over to another DB.

Note: there is no strong benefit to grabbing an EntitySet just to throw it away. Especially if you will be post filtering it then it’s actually faster to grab the EntityId sets and merge them… you end up with a lot less garbage and work since you don’t create all of the components for entities that just get thrown away.

jonesadev · April 14, 2013, 12:46pm

Just have a question about entity sync.
I have implemented one DefaultEntityData for players on my server, but how to stream entities to the client without creating a new entity on this last one?
I use serialization between servers and clients.

I may stream the Entity the first time and for others queries, use the EntityId to find entities on the client DefaultEntityData, but how to attach entities with their EntityId on the client EntityData?

Thank for your help.

pspeed · April 14, 2013, 2:09pm

@jonesadev said: Just have a question about entity sync. I have implemented one DefaultEntityData for players on my server, but how to stream entities to the client without creating a new entity on this last one? I use serialization between servers and clients.
I may stream the Entity the first time and for others queries, use the EntityId to find entities on the client DefaultEntityData, but how to attach entities with their EntityId on the client EntityData?

Thank for your help.

In a remote setting, you do not need a different Entity implementation. You just need to manage the EntitySet properly over the network. And clients should be read only. That’s important because you don’t want some hacked client messing directly with an entity’s components.

So on the server, you watch for changes and send them to the clients… they get sent to the EntitySet and the entities will update the next time the client calls applyChanges(). Just like in a non-networked environment. In Mythruna, I go as far as having EntitySets on the server for what the client has requested so that I can minimize the changes sent. There is a version of applyChanges() that will tell you all of the ChangeEvents that were relevant.

jonesadev · April 14, 2013, 5:02pm

Ok, so I’d misunderstood the EntitySet concept.
Actually, I use two DefaultEntityData, one for players and one for accounts.
But I must use a unique EntityData and use EntitySet to dissociate all types of entities, I’m right?

On client side, How to identify what entity need change? I need to store EntityId in my own way?

wezrule · April 14, 2013, 5:03pm

You should have called it EZay-ES

pspeed · April 14, 2013, 5:16pm

@jonesadev said: Ok, so I'd misunderstood the EntitySet concept. Actually, I use two DefaultEntityData, one for players and one for accounts. But I must use a unique EntityData and use EntitySet to dissociate all types of entities, I'm right?
On client side, How to identify what entity need change? I need to store EntityId in my own way?

Sorry… I was mixed up. I was thinking DefaultEntity.

Yes, you will want a RemoteEntityData on the client. My bad. In my case, it is a direct implementation of the interface and does not extend DefaultEntityData at all.

It essentially passes all get/queries through to the server. Even EntitySet uses the ID based query and loads itself up asynchronously… but I just use the DefaultEntityset implementation otherwise.

Bonechilla · April 17, 2013, 9:15pm

When I attempt to create a new entity using DefaultEntityData over SQLEntityData I get a null pointer error am I supposed to set the IDGenerator?

Also the only way to remove a component in Zay-Es is through its EntityData correct?

pspeed · April 17, 2013, 10:08pm

@Bonechilla said: When I attempt to create a new entity using DefaultEntityData over SQLEntityData I get a null pointer error am I supposed to set the IDGenerator?
Also the only way to remove a component in Zay-Es is through its EntityData correct?

it’s supposed to work, can you post a stack trace?
Correct.

ogerlord · April 24, 2013, 3:41pm

com.google.common.base.Objects;
org.apache.log4j.Logger

Do you think the additional librarys are necessary?

pspeed · April 24, 2013, 11:34pm

@ogerlord said: com.google.common.base.Objects; org.apache.log4j.Logger
Do you think the additional librarys are necessary?

JUL sucks hard donkey balls and I really don’t want to use it.

Guava is awesome and I think we will use it more. Some of the classes I removed were using it for the cache and some other things, I think. I consider Guava standard Java kit at this point… and some of the stuff is even making it into Java proper.

Bonechilla · April 25, 2013, 2:06am

@pspeed said: 1) it's supposed to work, can you post a stack trace?

Correct.

Quick simple Test based on your initial post it works fine when I use SqlEntityData but not with DefaultEntityData. When I check idGenerator isn’t set though i’m sure something is probably slightly off with what I am doing. I know that when I use the SqlEntityData model all components which implement PersistentComponent will be added to a table in the HSQLDB and any components withouth will be saved normally within interlinking hashmaps correct. Does this mean that in DefaultEntityData all components are added in the latter way? …btw I used subversion to checkout Zay-ES from the trunk of jmeContributions hosted on Google Code

[java]
/*

To change this template, choose Tools | Templates
and open the template in the editor.
*/
package test.zay_es;

import com.simsilica.es.ComponentFilter;
import com.simsilica.es.CreatedBy;
import com.simsilica.es.EntityData;
import com.simsilica.es.EntityId;
import com.simsilica.es.EntitySet;
import com.simsilica.es.Name;
import com.simsilica.es.base.DefaultEntityData;
import com.simsilica.es.filter.FieldFilter;
import java.sql.SQLException;

/**
*

@author Kyle
*/
public class SimpleTest {

public static void main(String args) throws SQLException {
// The main entry point to my ES is EntityData… I think it better describes what we’re
// doing and avoids the “system” confusion
EntityData ed = new DefaultEntityData();//new DefaultEntityData();//SqlEntityData(“some/path”, 100); // 100 ms write-delay

// Create an entity and set some stuff using raw access
EntityId anakin = ed.createEntity();
ed.setComponent(anakin, new Name(“Anakin Skywalker”));

    EntityId c3po = ed.createEntity();
    ed.setComponent(c3po, new Name("C-3PO"));
    ed.setComponent(c3po, new CreatedBy(anakin));


    ComponentFilter filter = FieldFilter.create(CreatedBy.class, "creatorId", anakin);
    EntitySet es = ed.getEntities(filter, CreatedBy.class, Name.class);
    es.applyChanges(); // not strictly necessary but not a bad idea
    System.out.println("Anakin’s creations:" + es);


    ComponentFilter filter2 = FieldFilter.create(Name.class, "name", "C-3PO");
    EntitySet es2 = ed.getEntities(filter, Name.class, CreatedBy.class);
    es2.applyChanges(); // not strictly necessary but not a bad idea
    System.out.println("c3po’s name:" + es2);
}

}
[/java]

I get the resulting Error

[java]
Exception in thread “main” java.lang.NullPointerException
at com.simsilica.es.base.DefaultEntityData.createEntity(DefaultEntityData.java:105)
at test.zay_es.SimpleTest.main(SimpleTest.java:29)
Java Result: 1
[/java]

If I can ask even though its slightly off the Question i understand getAddedEntities() return entities that are added to the set after gaining the components required, getRemovedEntities() return entities that had a component removed or a component changed that no longer matches the filtering criteria of the set; however under what conditions will getChangedEntities() return something? It seems to never return anything under the small test examples I set up.

pspeed · April 25, 2013, 2:51am

@Bonechilla said: Quick simple Test based on your initial post it works fine when I use SqlEntityData but not with DefaultEntityData. When I check idGenerator isn't set though i'm sure something is probably slightly off with what I am doing. I know that when I use the SqlEntityData model all components which implement PersistentComponent will be added to a table in the HSQLDB and any components withouth will be saved normally within interlinking hashmaps correct. Does this mean that in DefaultEntityData all components are added in the latter way? ...btw I used subversion to checkout Zay-ES from the trunk of jmeContributions hosted on Google Code
If I can ask even though its slightly off the Question i understand getAddedEntities() return entities that are added to the set after gaining the components required, getRemovedEntities() return entities that had a component removed or a component changed that no longer matches the filtering criteria of the set; however under what conditions will getChangedEntities() return something? It seems to never return anything under the small test examples I set up.

Ah, yeah, I probably need to set a default ID generator. That code is a bit new.

The SQL entity data works pretty much exactly the same as DefaultEntityData if there are no persistent components. The only difference will be that the last entity ID is remembered from one run to the next.

getChangedEntities() will return entities whose components were swapped out without removing them from the set. For example, if you change the Name component. The whole architecture is based around immutable components so the only way to change a value is to set a new version of the component. You can do it other ways but then you manage your own threading and changes.

Bonechilla · May 17, 2013, 1:07pm

Okay cool there’s another version somewhere :?:

Also can i ask if there are plans to update the interface of Entity and DefaultEntity with public boolean removeComponent(Class type ).
It gets kind of messy unless I update the interface and it will break every time i update zay-es from repo, also since the systems should really focus on entitySets after initialization i see it kind of weird to also declare an entityData object in each system. Was this done intentionally to avoid something:?:

Lastly as a means of abstraction in my visual system and as an easy way to pass certain information between systems I was attempting to attach an additional Component once an entity is added; however for some odd reason it continues to return null, when its called anywhere. I added a println right after I set the component and it returned null, i back tracked through DefaultEntity, and DefaultEntityData and a few other things to see what i was doing incorrectly however i couldn’t find anything directly. Can I ask where is my error or is this not the right way to go about things:?:

[java]

private void add(Set&lt;Entity&gt; addedEntities) {
    for (Iterator&lt;Entity&gt; it = addedEntities.iterator(); it.hasNext();) {
        add(it.next());
    }
}

private void add(Entity e) {
    VisualRepPiece vrp = e.get(VisualRepPiece.class);
    VisualTypePiece vtp = e.get(VisualTypePiece.class);
    Node root = vtp.isCharacter() == true ? modelNode : sceneNode;
    try {
        Spatial model = loadedList.get(vrp.getAssetName());
        int index = root.attachChild(model);
        e.set(new VisualInfoPiece(index, model.getName()));

System.out.println(VisualInfoPiece.class);//THIS WILL RETURN NULL FOR SOME ODD REASON
} catch (ExecutionException ex) {
SpectreApplication.logger.log(Level.SEVERE, "Unable to correctly load Asset: " + vrp.getAssetName(), ex);
}
}
[/java]

reason why i abstracted add and remove was because i got lazy in change so i just call remove then add again :affe:

pspeed · May 17, 2013, 3:34pm

I’ll answer parts separately…

@Bonechilla said: Okay cool there's another version somewhere :?:

No, there is only one version and I still need to fix this bug. Sorry.

@Bonechilla said: Also can i ask if there are plans to update the interface of Entity and DefaultEntity with public boolean removeComponent(Class type ). It gets kind of messy unless I update the interface and it will break every time i update zay-es from repo, also since the systems should really focus on entitySets after initialization i see it kind of weird to also declare an entityData object in each system. Was this done intentionally to avoid something:?:

if you remove a component from an Entity then it doesn’t exist anymore. Philosophically, you can’t do it. An Entity object is just a view of specific components of an “entity”. As soon as you remove a component then that view doesn’t exist anymore. I’m very uncomfortable adding these methods because it makes things confusing. As soon as you call removeComponent() then the Entity should somehow become invalid.

Removing components is semi-rare, though. Only a few systems should need to do it, really. That being said, I do pass my EntityData around to me JME-side systems because usually they are app states and I grab the EntitySet on initialize() and release it on cleanup… so I need a reference to EntityData.

I’ll look at and answer the rest in a minute.

pspeed · May 17, 2013, 3:43pm

@Bonechilla said: Lastly as a means of abstraction in my visual system and as an easy way to pass certain information between systems I was attempting to attach an additional Component once an entity is added; however for some odd reason it continues to return null, when its called anywhere. I added a println right after I set the component and it returned null, i back tracked through DefaultEntity, and DefaultEntityData and a few other things to see what i was doing incorrectly however i couldn't find anything directly. Can I ask where is my error or is this not the right way to go about things:?:
[java]
private void add(Set&lt;Entity&gt; addedEntities) {
    for (Iterator&lt;Entity&gt; it = addedEntities.iterator(); it.hasNext();) {
        add(it.next());
    }
}

private void add(Entity e) {
    VisualRepPiece vrp = e.get(VisualRepPiece.class);
    VisualTypePiece vtp = e.get(VisualTypePiece.class);
    Node root = vtp.isCharacter() == true ? modelNode : sceneNode;
    try {
        Spatial model = loadedList.get(vrp.getAssetName());
        int index = root.attachChild(model);
        e.set(new VisualInfoPiece(index, model.getName()));
System.out.println(VisualInfoPiece.class);//THIS WILL RETURN NULL FOR SOME ODD REASON
} catch (ExecutionException ex) {
SpectreApplication.logger.log(Level.SEVERE, "Unable to correctly load Asset: " + vrp.getAssetName(), ex);
}
}
[/java]

reason why i abstracted add and remove was because i got lazy in change so i just call remove then add again :affe:

An Entity is a subset view of an “entity”. It only has the components you asked for. Think if it like a JDBC result set. You can’t then ask for the values of fields that you didn’t query. It’s important to remember that an Entity object is not an “entity”.

So, I’m guessing you asked for an EntitySet with components X, Y, and Z and now add a component R and wonder why you can’t see it… because R is not a part of that view. And if it was, then the entity would have never shown up. An EntitySet looking for R should see the change, though.

Still, using a component for visualization-side stuff is kind of broken. Avoiding a visualization-side HashMap is not a good excuse to start storing visualization elements in components. It sort of breaks the architecture and stretches the ES beyond its intent.

Bonechilla · May 17, 2013, 6:45pm

<cite>@pspeed said:</cite> I'll answer parts separately...
No, there is only one version and I still need to fix this bug. Sorry.

if you remove a component from an Entity then it doesn’t exist anymore. Philosophically, you can’t do it. An Entity object is just a view of specific components of an “entity”. As soon as you remove a component then that view doesn’t exist anymore. I’m very uncomfortable adding these methods because it makes things confusing. As soon as you call removeComponent() then the Entity should somehow become invalid.

Removing components is semi-rare, though. Only a few systems should need to do it, really. That being said, I do pass my EntityData around to me JME-side systems because usually they are app states and I grab the EntitySet on initialize() and release it on cleanup… so I need a reference to EntityData.

I’ll look at and answer the rest in a minute.

Ahh i see this actually connects to the next question in that the concept of removing components was to check if it was present and perform some action if it indeed was, however; i see now it is better to simply check for I suppose an sentinel or something similar would work better.

Bonechilla · May 17, 2013, 7:04pm

<cite>@pspeed said:</cite> An Entity is a subset view of an "entity". It only has the components you asked for. Think if it like a JDBC result set. You can't then ask for the values of fields that you didn't query. It's important to remember that an Entity object is not an "entity".
So, I’m guessing you asked for an EntitySet with components X, Y, and Z and now add a component R and wonder why you can’t see it… because R is not a part of that view. And if it was, then the entity would have never shown up. An EntitySet looking for R should see the change, though.

Still, using a component for visualization-side stuff is kind of broken. Avoiding a visualization-side HashMap is not a good excuse to start storing visualization elements in components. It sort of breaks the architecture and stretches the ES beyond its intent.

Okay their were still a few things i didn’t understand clearly thanks for resolving that I spent the last few days trying to figure out the issue when there was none, I should have simply left what I had originally…good thing i can revert from history, thank you Netbeans =p

When you say visualization elements are you referring to the VisualInfoPiece? I was using it to store the index the spatial is attached to the parent as well as the spatial’s name(as a double check) that combined with VisualTypePiece(which i should really change a little later in case a spatial is attached to another node or better yet add the name of the parent and its index to VisualInfoPiece as well and if their is an issue use VisualTypePiece to know the closest node i should traverse, or something along those lines), would help other systems in locating a spatial without directly interacting with another system to get the information, from their I can just store it in the specified system through a hash-map or somthing along those lines.

lol too much kool-aid 8)