OpenGEX format - A game-centric alternative to Collada and FBX

General Discussion
1

http://opengex.org/

From the website:

The Open Game Engine Exchange (OpenGEX) format is a text-based file format designed to facilitate the transfer of complex scene data between applications such as modeling tools and game engines. The OpenGEX format is built upon the data structure concepts defined by the Open Data Description Language (OpenDDL), a generic language for the storage of arbitrary data in human-readable format.

The OpenGEX format was created because Collada, the open standard that we all hoped would provide a well-supported asset exchange format, has proven to be an unreliable mess. The most common source of problems has been the poor quality of the Collada export plugins available for software such as 3D Studio Max and Maya, and we attribute this to Collada’s over-engineered design and its mutating under-specified format.

It’s created and maintained by Eric Lengyel, creator of the C4 Engine.

I quite like the idea of eventually abandoning our OGRE format support in favor of FBX, OpenGEX and Blender.

4 Likes
2

Hm sounds nice, wich applications do support it so far?

3

They have exporters for 3DS and Maya, Blender “coming soon”. But I think they really deserve some support from more sides, they seem to be very aware of the actual needs for such a format and especially what problems the Collada format had.

1 Like
4

I have seen it some time ago, but for me it is has crazy floating point representation.

VertexArray (attrib = "texcoord")
{
  float[2]
  // 24
  {
    {0x3F800000, 0x00000000}, {0x3F800000, 0x3F800000}, {0x00000000, 0x3F800000},
    {0x00000000, 0x00000000}, {0x00000000, 0x00000000}, {0x3F800000, 0x00000000},
    {0x3F800000, 0x3F800000}, {0x00000000, 0x3F800000},
    {0x00000000, 0x00000000},
    {0x3F800000, 0x00000000}, {0x3F800000, 0x3F800000}, {0x00000000, 0x3F800000},
    {0x00000000, 0x00000000}, {0x3F800000, 0x00000000}, {0x3F800000, 0x3F800000},
    {0x00000000, 0x3F800000},
    {0x00000000, 0x00000000}, {0x3F800000, 0x00000000},
    {0x3F800000, 0x3F800000}, {0x00000000, 0x3F800000}, {0x00000000, 0x00000000},
    {0x3F800000, 0x00000000}, {0x3F800000, 0x3F800000}, {0x00000000, 0x3F800000}
   }
}

Really? Only rationale which is mentioned for that is bit-exact representation between platforms/languages, but who cares about single ulp when dealing with game graphics? Any kind of readability gained by chosing text format is lost due to this decision. Yes, it is ‘optional’, but given it is default for existing plugins and all examples have this representation, we can be sure that all files out there will follow that.

I also have some doubts regarding Material node. Most interesting things will be happening there, but for obvious reasons it is underspecified. You will end up having per-source material interpreter anyway (which might be not that bad - entire geometry/parsing would be shared, just material creation would be engine specific).

5

Oh, yes! One format to rule them all…

1 Like
6

Appreciate the feedback @abies. I sent Eric an e-mail pointing him to this thread so he can make a case for his design decisions if he’s so inclined.

It’s an open standard, so we can help push it forward, with new defaults and gradual spec changes. With time, perhaps we can build up a small community around it. It just needs users first. Even if it has some flaws, it’s hard to dispute the fact that the game development industry would benefit from an unencumbered, open, interchangeable standard format for 3D game assets.

1 Like
7

In such case I would need to go through it in more detail…

  • no information about size of arrays. It is a lot more efficient for reading apps if you can see vertex[3][12] rather than vertex[3] and then have to do growing buffer to read unknown amount of values and then scale back into engine efficient storage.

  • some kind of version/style information; format is very broad and every game/exporter would implement its own extensions etc; having something like ‘ElderRolls7’ specified somewhere could help preparing parser/interpreter for the fact that you will be getting subsurface textures for models, heightmap will be encoded in normalmap alpha and you are not supposed to reverse the y-direction of the textures;

  • I do not see details about split file definitions; it is very common for games to for example specify skeletons in one file, number of actual models in different files and various sets of animations in yet another set of files (dependent only on skeleton one); this use case should be explicitly described - either by having consistent behaviour for referencing non-existing names or having some kind of ‘stub’ definitions to indicate slots which will be filled out by different sources

  • even worse, animations seem to be done wrong way around - they are inside things they are modifying, instead of outside; it might be enough for most simplistic cases of passively animated objects, but won’t do for any real world usage; I think that entire animation part should be scrapped and redone in more proper way, external to core geometry graph;

  • if we are at animations (proper, external ones), please add ‘events’ - this is very helpful thing, to inject application specific events into timeline; good example for that is walk animation, where you have ‘footstep’ events at proper times, which could be used to play footstep sound, leave track etc depending on application needs;

  • collision structures seem to be missing; there should be a way to indicate simplified geometry and/or non-mesh primitives for collision resolution

  • particles… this one might be impossible to in satisfactory fastion in generic format, but is certainly needed inside models for games; lack of any support would be a showstopper for most of the games; even some very vague format which would allow putting game specific information inside could solve the issue

4 Likes
8

Greatly appreciate the additional feedback @abies. I really hope Eric will take the time to get back to us now.

I’d love to hear from @kaelthas as well, considering he has built up a lot of experience working with a format less-than-ideal-for-games for years now.

9

Having recently written some code for extracting meshes from Collada files exported from MakeHuman and Blender I can agree that the Collada exporters used in those products are sub-par, the files are often not even valid against the Collada schema.

Which brings me to my chief concerns with “yet another export format” like OpenGEX: If the main motivation behind OpenGEX is the poor support for Collada exporters in various products I have to ask in what way these products support OpenGEX. My guess is that they don’t, so in that respect Collada is infinitely more supported making the whole argument circular or void depending on your outlook on life.

I really, really don’t understand the need to invent another markup language like OpenDDL just for this. You might hate XML all you want but it takes about 15 minutes to run JAXB on the Collada schema and have validating unmarshaller code generated that can read a collada file (making the data into something useful takes a lot of time of course since that depends on understanding the structure of the file).

10
@abies said: I have seen it some time ago, but for me it is has crazy floating point representation.

As you stated, the hex notation is completely optional. An exporter is free to use decimal notation if it wants to, and I don’t think the fact that our exporters use hex necessarily means that everyone else is going to do the same. We decided to use hex for two reasons. First, it does guarantee bit-exact transmission of the data, which is important (see below). Second, it takes less processing time to parse the hex notation than it does decimal, so large files with lots of vertex data will load significantly faster. Think of it as a compromise between text and binary formats.

@abies said: who cares about single ulp when dealing with game graphics?

People who are experienced in game graphics care. Depending on what you’re doing with the model data, if you’re off by even a single ulp, you can see tiny seams when you render your scene, leading to all kinds of unwanted headaches and quality issues.

1 Like
11
@abies said: - no information about size of arrays. It is a lot more efficient for reading apps if you can see vertex[3][12] rather than vertex[3] and then have to do growing buffer to read unknown amount of values and then scale back into engine efficient storage.

This was done intentionally because specifying the size of the whole array explicitly would introduce redundant information, which is a source of errors. We’ve seen Collada files where an array was specified as having a particular size, but then the actual number of elements in the array was different. This can’t happen in OpenGEX. It’s trivial to read data values into an automatically-growing array structure using either STL or the Array class in the open-source OpenDDL code, and the memory allocations are amortized. This is not a performance problem, but if you’re really hung up on it, you can always count the items in the array first, then allocate space for all of them before actually reading the values.

@abies said: - some kind of version/style information; format is very broad and every game/exporter would implement its own extensions etc; having something like 'ElderRolls7' specified somewhere could help preparing parser/interpreter for the fact that you will be getting subsurface textures for models, heightmap will be encoded in normalmap alpha and you are not supposed to reverse the y-direction of the textures;

We are working on a generic extension mechanism for the next revision of OpenGEX.

@abies said: - I do not see details about split file definitions; it is very common for games to for example specify skeletons in one file, number of actual models in different files and various sets of animations in yet another set of files (dependent only on skeleton one); this use case should be explicitly described - either by having consistent behaviour for referencing non-existing names or having some kind of 'stub' definitions to indicate slots which will be filled out by different sources

One of the design parameters for the first version of OpenGEX was to avoid the complexity of inter-file referencing because it makes it so easy to fragment a model and end up passing around data that has missing pieces. We just wanted to keep everything in one place to make it less error-prone. It’s possible that external references will be introduced in a future revision.

@abies said: - even worse, animations seem to be done wrong way around - they are inside things they are modifying, instead of outside; it might be enough for most simplistic cases of passively animated objects, but won't do for any real world usage; I think that entire animation part should be scrapped and redone in more proper way, external to core geometry graph;

We gave this considerable thought and concluded that it’s better to have the greater encapsulation provided by the current design. This is not going to change. Since an animation track pertains to only one node in the scene, we chose to store it with the node itself to avoid unnecessary referencing (yet another possible source of errors). This also mirrors the way that animations are accessed in modeling programs like Max and Maya. No functionality would be gained by changing to external animations.

@abies said: - if we are at animations (proper, external ones), please add 'events' - this is very helpful thing, to inject application specific events into timeline; good example for that is walk animation, where you have 'footstep' events at proper times, which could be used to play footstep sound, leave track etc depending on application needs;

This is a good suggestion (and something we actually have in the C4 Engine anyway). We will likely add this to the next revision of OpenGEX. It does not require external animations.

@abies said: - collision structures seem to be missing; there should be a way to indicate simplified geometry and/or non-mesh primitives for collision resolution

We intentionally left physics out of the first version of OpenGEX so that it wouldn’t take forever to finish the spec and release something. (There is a visibility flag for geometries that could provide some limited support in the area of collision meshes.) We are planning to add physics to a future revision of OpenGEX.

@abies said: - particles... this one might be impossible to in satisfactory fastion in generic format, but is certainly needed inside models for games; lack of any support would be a showstopper for most of the games; even some very vague format which would allow putting game specific information inside could solve the issue

Are particles well specified and/or well supported in another format? I understand that they’re important, but it seems like you’re really reaching in order to bash OpenGEX. If we were to consider adding particle support to the format, I’d be worried about the lack of any kind of standard that would be common to all game engines and modeling programs that support particle systems. In my opinion, developers tend to create particle systems in the game tools and not the external modeling tools, so support for this probably isn’t very useful.

4 Likes
12

My comments were put from the point “what I think would be needed before OpenGEX could be considered as a primary format for modelling tool -> game transfer”. If you are focusing only on basic geometry transfer (better .obj), with required engine-specific postprocessing for every model, then most of my points are invalid. I was thinking about blender/maya->OpenGEX->game engine. I have a feeling that you are more focused on ‘blender of person X’ -> OpenGEX -> ‘maya of person Y’ -> game specific exporter -> game engine. Which is fine, but this is where misunderstanding on my part came. I was comparing it to formats like Gamebryo(nif), Aurora(mdl) etc, rather than Collada/VRML/3ds.

Regarding single ulp of precision - sorry, I don’t buy it. Everything is going through multiple matrix transforms and there is no way that single ulp in float is going to survive model->projection matrix to make enough difference to be visible. But it is probably smaller issue in long run, just spoiling first impression of the format (taking away half of benefits of text-based format). A bit like xmls which are full of CDATA[a lot of binary uuencoded data here].

So, how do you handle named animations? Just having all animations on one long track and documenting that ‘walk’ is 0.0 till 2.7 seconds, ‘run’ is 2.71 till 4.5 seconds etc in readme.txt next to it?

13

The intended usage for OpenGEX is getting model data out of programs like Blender / Max / Maya and into game engines. For me personally, the OpenGEX format is a replacement for the broken Collada format.

@abies said: Regarding single ulp of precision - sorry, I don't buy it.

When you reach my level of knowledge and experience, perhaps you’ll change your mind. I’m not going to argue this.

@abies said: So, how do you handle named animations?

The OpenGEX format supports multiple independent animation clips per file. In the art pipeline where I work, however, we usually just export a separate file for each animation because our engine let’s us import animations separately for each model, and this makes it easier to modify individual animations.

14
@Eric-Lengyel said: The OpenGEX format supports multiple independent animation clips per file. In the art pipeline where I work, however, we usually just export a separate file for each animation because our engine let's us import animations separately for each model, and this makes it easier to modify individual animations.

Separate file per animation was what I originally expected, but you stated “No functionality would be gained by changing to external animations.” Does it mean that you duplicate entire geometry in each animation file to have it fully self-contained? Or just duplicate skeleton data and then merge things with geometry ‘master’ based on node names?

Do you have full example of production quality OpenGEX model, with animations, textures etc? I suppose that one full example could got a great length towards explaining the best practices.

15
@abies said:

Regarding single ulp of precision - sorry, I don’t buy it. Everything is going through multiple matrix transforms and there is no way that single ulp in float is going to survive model->projection matrix to make enough difference to be visible.

Maybe not in model’s projection matrix, but not so long ago I fixed a bug where this single ULP caused a lot of troubles.

I was imrpoving the IK constraint (still not finished but hope to get rid of last bugs soon :roll: ) and I used float’s to read animation bezier curves. The inverse kinematics constraint uses the CCD algorithm which means it iterates many times and each iteration gets you closer to the result.

Even 50 iterations multiplied the floating point errors so much that my model’s animation was completely nonsense.
And the difference that caused this was about 0.0000001 or something like that.

Only after using doubles in BezierCurve class computation was I able to get rid of these errors.

16
@Kaelthas said: Only after using doubles in BezierCurve class computation was I able to get rid of these errors.

I fully agree. I was pointing that if you are in range of troubles when 1 ulp makes noticeable difference, it won’t help you to be 1ulp exact - you will end up needing order or two of magnitude more depending on other factors. And that I don’t think that model/texture coordinates is something where it will be important for current graphic cards. I certainly do not have great experience in graphic development, but I do have considerable experience with dealing with floating point numbers in places where it means money. And if you are having troubles with float resolution, 1ulp-exact representation is NOT a solution - you either go for doubles, BigDecimal or some variation of fixed decimal format.

But we have seen authorative argument from authority here, that 1 ulp inconsistencies in floats are going to cause tears in models in game graphics engines. And this I’m not going to challenge, as it would require ad-hominem rather than technical arguments, so let’s focus on other parts, where discussion is still open and can be productive.

2 Likes
17

I may be wrong, but when i see a format named “open game engine exchange” i think that it will do more than just 3d models/animation.
For exemple the audio. It could be interesting to attach to a model a sound that it should produce when you walk on it, or define a sound environnement (echo, dampen etc) to a room/part of the model.

And a game is a lot more than that. For a format with a name like that i expect more something like a TES mod file format (but open).

18
@bubuche said: I may be wrong, but when i see a format named "open game engine exchange" i think that it will do more than just 3d models/animation.
I can agree that the name sounds a bit broad, but I certainly don't want broad functionality like that. Things like audio and aforementioned particles aren't a problem for us. The 3D import pipeline is, and that's all this format should be trying to solve.

Thank you very much for stopping by @Eric-lengyel. Regarding:

@Eric-Lengyel said: The intended usage for OpenGEX is getting model data out of programs like Blender / Max / Maya and into game engines. For me personally, the OpenGEX format is a replacement for the broken Collada format.
Wouldn't it then be fair to say it's also meant as a replacement/competitor for the proprietary FBX format as well?

I definitely agree that having a “production quality OpenGEX model, with animations, textures etc” like @abies suggests would be the best way to convey best practices.

For anyone who wants to keep close tabs on the development of the spec, is there a mailing list or forum one can subscribe to?

19

Regarding the Float representation, does the format also allow for doubles in that notation, then the exporters that have additional accuracy information could supply it.

20
@Empire Phoenix said: Regarding the Float representation, does the format also allow for doubles in that notation, then the exporters that have additional accuracy information could supply it.

OpenGEX doesn’t support doubles at the moment. OpenDDL part does, so in theory OpenGEX could be extended to allow them - but I don’t think it would do much good in practice, given games being targeted. Maybe for other uses, architecture, medical or whatever - but we are discussing GAME exchange format, rather than VRMLCollada3.14AanotherGenericFormatNotSupportingAnyUseCaseWell". By targeting audience which will downcast it to floats in majority of cases anyway, it would just allow another source of incompatibilities.

Edit:
I think we should really get out of the float exactness topic. It makes files look funny, but it is not a breaking point. Things like managing multiple animations and retargetting them to different models is important, handling game-specific extensions is important, etc.