Xbuf format – A developer friendly game exchange format for 3d data

[david_bernard_31]

Hi,

I started (an other) 3D format + a set of tools: xbuf
The format is in alpha stage, and I need to collect use case, user requirements (other than mine ) and feedback about the format (what is missing (lot of stuff), what is not clear, …).

I label it “developper friendly”. because :

• it is defined via protobuf, so binding (serializer/deserializer) for your language (java, python, C++, C#, …) are generetated : no need to write custom parser,… you only need to write code to adapt pivot structure to your one.
• It’s open source CC0
• It’s hosted on github

I would like to share this format with other community (other game engine, …), but I start with my favorite one

Edit: rename pgex to xbuf
Edit: update links

[abies]

Do you expect this format to possibly get loaded by runtime engines directly, or purely as exchange format between tools, to be post-processed/baked/whatever into engine specific format?

[nehon]

I read the FAQ on the github repo, and in the “why not opengex?” you say that you need a loader and an exporter. I don’t get why your format doesn’t.

[david_bernard_31]

@abies,

The format can be loaded at runtime. For example, I use it to update running jme scene from blender. But it can be used in a pipeline (checker, reorg-data, post-processed), as writing tool to read / write it is easier than …
I already start a list of some basic tool, I could write later.

@nehon,

The phrase is

Why not OpenGEX ?

• require to write parser, loader, exporter

With pgex, you don’t need to write a parser : it’s generated by protobuf compiler + plugin for your language. For example: no need to write and to maintain “ogex”, the equivalent for pgex is generated. including the raw exporter (that generate the bytes). But you have to write the conversion between your format and pgex generated classes. That is more dev friendly (code completion, type checking). I try to reduce the usage of ‘string’ to id and relation (between id).

[david_bernard_31]

I’ve got a pretty basic blender exporter and jme loader:

The format is incomplete but for the memory, I compare filesize :

-rw-r--r-- 1 dwayne users  28K  6 janv. 10:11 test_05.j3o
-rw-r--r-- 1 dwayne users  69K  6 janv. 09:43 test_05.ogex
-rw-r--r-- 1 dwayne users  18K  6 janv. 09:03 test_05.pgex

[abies]

Try gzipping the result files to see a real difference. As it is not direct mem-mappable format, compressing is not harming anything.

[david_bernard_31]

It’s not a benchmark. the format is in early stage (missing some data). A more serious benchmark should take care of time to load (like done on jvm-serializers). Techno like Cap’n proto or flatfbuffer should have better results, there are very young (vs protobuf).

I did the comparison to have an estimation, and to mesure over time how the format will grow.

-rw-r--r-- 1 dwayne users  12K  6 janv. 10:11 test_05.j3o.gz
-rw-r--r-- 1 dwayne users  18K  6 janv. 09:43 test_05.ogex.gz
-rw-r--r-- 1 dwayne users  11K  6 janv. 09:03 test_05.pgex.gz

So pgex take less size than ogex.gzip. it’s a nice start.

[david_bernard_31]

It’s a exchange format, not for jvm only.
When the format will be a little more mature, I plan to ping other community ogre, panda3d, urho3D, … unity.

EDIT: I moved the doc to the github’s wiki.

EDIT2: I forgot the ASSIMP community in the list

[abies]

How do you want to solve animations/skeletons? Inline, separate file with references or both?
What about collision data/simplified meshes?
LOD?

In any case, I think that Material node will be a real breaking point here. I don’t think that current approach of enum+union parameter is good - it allows very malformed materials to exist (color being Vec3, Vec4 or string - somebody WILL come up with that and it doesn’t make it any easier to extend that normal list of fields. I suppose that it might be better to create proper protobuf with explicit types and fields (all of them optional), put some reserved extension fields and possibly allow different Material nodes as an option.

[david_bernard_31]

Lot of questions, thanks.

About material, I though about using explicit typed field. You’re right, today I store Texture or Vec4 for attrib “color”, “normal”. I made it to mimic the Generic one provide by the CustomParams extension. Maybe an error, at least it break my target of being less error prone for dev.

About, LOD, collision,… currently I don’t know, I’m open to suggestion (and code .proto). I start implementation from OpenGEX and then I adapt, transform,… Note that some feature could be in core or in extension (see customparams).

About animations/skeleton, I plan to store then as core entry in Data (not as extension). Then use Relation link them (like Compotent are linked to Entity in ECS). In my vision, user can store 3D data in a single uber pgex Data file or split them across several (one per model, animation,…) the only requirement is to have unique id over a set of pgex.

[atomix]

Hi,

This is an interesting project beside of OpenGEX. I like the idea of a “data format for exchange 3d”. As every game developer, I have a chronic headache with the asset pipeline, and it’s getting worse. I also have some technical experiments with my team about way to solve a small scale asset pipeline, repo; and scalable architecture if chance that your team become bigger, or work in enterprise. Imagine a “file” to be exported, imported by CCDs, engines; “bitstream” flow from CCDs into our rendering pipeline… Well, something like that…

The topic we referenced is “Universal format” or “Unified format”. But can it be “developer friendly” as a “format” without solving several problems, I mean I doubt about its usefulness.

We’ve kind of faced the trade off between “easy in design time” and “fast small in run-time”, and really hard to find a magnitude to fit all cases (impossible?). Later, we decide to move on with a much smaller problem: Design a binary (bitstream) flows (dataflow, workflow…) from other applications toward JME3. For my team, its like Blender, 3DSMax, Maya and PTS into JME (Java and OpenGL).

What we discovered so far is to design the pipeline as running progresses, focused in the flows (structured bitstream) via ports, instead of imaging a static, consistent storage. So 3d files for example are just persisted form, one form in serveral forms of data in the pipeline.

And because we don’t have a sophisticated overall universal structure (only bitstream), we focused in generic decoder with schema in ports instead. For ex: decoder from 3dsmax into generic bitstream. We also use Protobuff to encode and transfer data in the pipe. But note that we stick with the bitstream instead of depend in schema.

We also in early stage but there are some technical benefits:

• Thanks to bitstream abstraction, we have used Preon, Protobuff, Netty to read/write/encode/decode/send/get between pipeline.
• With pipeline abstraction, we’re experimenting Apache Storm to run works over data easily and managable, and most importantly fault tolerant! We can convert massive project models, we can render to movie some part of game if we need.
• Thanks to JME3 asset abstraction, we have hooks into our production pipeline. So models, sounds are reloaded seamlessly in-game. We’re experimenting reloaded through internet connections on devices (yes, reload new assets on phones…). In JME side, it’s piece of cake; but we have to built an assets server for that job and that’s another long story.
• We’d like to push it further, in the direction towards a well defined, well structured storage repository system for assets for JME. It’s like git for assets: multi versions, easy to folk, easy to solve (non-geometry) conflicts.

Now about Pgex, i have a few questions:

First ) it’s not a “Format”? It’s like OpenGEX specs hook into Protobuff decoder facilities. So it has a generic structure (not really totally general but gamedev infos oriented), with a generic decoder. Talking about the “format”, I really like to know about the structure you provide to solve specific game problems.
2) its has external references? The problem @abies talked about is one of problem if this so call “unified format” has external references to other resource in its content. Because its try to self contains as much as possible, it’s either wasteful (also not reusable) or plain useless. In opposite direction, it’s become a hash contains only references to others. This unified format looks a bit funny: like HTML once try to carry data in its own, but references all over the places.
3) it has domain specific infos? Physics, IK, decoding, localization, user’s data… We can not have standards for everything in this world (someone blame COLLADA because it’s the chosen one but become naughty after all)

The solution I’ve talked above (called AtomExAsset) admit the nature of external resources, try to obtain to solid version if it can, retry several times, retry other resolutions (if not configurated beforehand) and return the result or a place holder peacefully. Because in it spirit, it’s only return the assets that suite JME3, and WORK without bad exceptions.

“It will not stop developer to continue working”.
That’s my idiom of “developer friendly” and some thoughts i want to share.

https://wiki.jmonkeyengine.org/legacy/doku.php/jme3:advanced:atom_framework:atomexasset
Well, incomplete wiki like always, but I will back to this sometimes …

[david_bernard_31]

Hi atomix,

Thanks for the feedback (a lot to read). I tell in first post why I label it “developer friendly” vs fbx, opengex, blend,…

I don’t want to create a runtime format that match all need. I used it at runtime but it’s firstly an exchange format, an input of the pipeline not the output.

From my experience, usefull stuff and reusable stuff are the most simple to install, to understand, to use (to combine or to integrate with other stuff). Never the one that want to encapsulate every thing, provide generic, layer,… the one “overkill”. I continue to create framework with layer but for my personal use, because often they are understable only by creators. I don’t want this for pgex : KISS.

Anyway, thanks again for the info, I’ll re-read them this week-end, when I’ll have more time.

EDIT: I never saw successfull “one-size fit all” solution (else company will not need so many dev )

[atomix]

Hi,

I’ve written too much. So one question for TL;DR:
What is the benefit of use this instead of OGEX:
For OGEX:

• you have to use ogex exporter and then write an importer for your specific need?

For pgex:
1 - Exported into ogex, You just have to write an “extracter” that take info from a parser ( generated for your language).
2 - if ogex not supported You just have to write an “exporter”, and also benefit with the parser.

In 1, I see quite attractive advantages, but note that it’s only for 3d model without any other things right now.
In 2, I see a lot of extra steps:

• I have to write into binary format
• write a protobuff schema
• pgex help me in the middle
• later I need an extractor.

Even if I can write a protobuff for blender, it may not worth the effort I have to spend. And use protobuff for parse a text file .obj for example sound pretty dumb.

[david_bernard_31]

@atomix said: Hi,

I’ve written too much. So one question for TL;DR:
What is the benefit of use this instead of OGEX:
For OGEX:

• you have to use ogex exporter and then write an importer for your specific need?

It’s already a hard part, else jMonkeyEngine already have it (error detection, recovery, semantic,…). pspeed wrote jogex (iirc not finished) a java parser for opengex that create the memory structure. This part, that is auto-generated by protobuf for pgex, already takes lot of works and need to be maintained.
On other side, OpenGEX already include bone, animation, skin. stuff that pgex didn’t include (part of the TODO list).

@atomix said: For pgex: 1 - Exported into ogex, You just have to write an "extracter" that take info from a parser ( generated for your language). 2 - if ogex not supported You just have to write an "exporter", and also benefit with the parser.

Sorry, I don’t understand. ogex == OpenGEX != pgex

@atomix said: In 1, I see quite attractive advantages, but note that it's only for 3d model without any other things right now. In 2, I see a lot of extra steps: - I have to write into binary format - write a protobuff schema - pgex help me in the middle - later I need an extractor.

Even if I can write a protobuff for blender, it may not worth the effort I have to spend. And use protobuff for parse a text file .obj for example sound pretty dumb.

You don’t need to write a protobuff schema, except if you want to create an pgex extension for an uncommon usage (no requirement to be supported by other pgex tool).

I don’t understand the relation between protobuff and .obj.

About blender and jme, I already create the base code (blender exporter and jmonkey importer), as POC. Now I add “feature” in pgex and I don’t care about binary encoder/decoder (it’s the job of protobuff + pgex). I only care about mapping between 2 memory “schemas”

I relink a video about a tool I work on where I used pgex (for a part of the communication) of jme3-as-blender-renderer

blender external renderer (a game engine)

[atomix]

For pgex: 1 – Exported into pgex, You just have to write an “extracter” that take info from a parser ( generated for your language). 2 – if pgex not supported You just have to write an “exporter”, and also benefit with the parser.

Sorry, twisted mind.

From what I understand from your answer:
– I have to write into binary format (pgex to be specific)
– write a protobuff schema
– pgex help me in the middle
– later I need an extractor.

.Obj is a text-based 3d model file. How can pgex help in exchange such file?

• Write exporter to save into pgex, then benefit from the parser part. Like what you do with Blender?
• Open in Blender the .obj file, and save into pgex.

Thanks for your answer. Cheers,

[david_bernard_31]

.obj -> .pgex could be done by a converter able to export into .pgex.
Today only Blender is able to export to .pgex. But due “to ease” to write .pgex I hope assimp or other can support it later.

There is no magic or universal. I just try to find a solution of the poor support of existing format (obj, fbx, blend,…) : hard to code valid exporter/importer.

[normen]

I didn’t think this was a good idea but now that Eric Lengyel (the OpenGEX developer) probably moved himself into a corner with his silly Linux-ragequit you might just have a chance with it ^^

[david_bernard_31]

I often had bad idea, it’s why I ask your opinion. Future will tell.

I hope to be able to implement basic animation support before end of january 2015.

[david_bernard_31]

Update:

• pgex is renamed “xbuf” to aboid conflict and confusion with OpenGEX, so I rename this topic, repository, package,…
• I failed to implement basic animation before end of january. Today I have basic animation of object

see

[devlog] jme3 as blender renderer

A Quick update : I’m working on animation (slow progress), I’m not happy with the current work (I’ll change it later) today I rewrote all the java8 code of pgex and remote editor (the jme side) in Xtend, so it generates java 1.7 code and I continue to have syntax sugar on lambda,… => pgex and animation stuff will be usable with java 1.7 (android,…). But the javaFX stuff will only run on jvm 1.8. [Imgur]

[david_bernard_31]

Some news:

1. xbuf (previously pgex) has support for object animation and skeletal animation
2. I’m grouping repository and communication about xbuf, jme in blender, remote editor under xbuf umbrella (so future update of the blender plugin will be on this topic)

The latest video I made one month ago about blender/xbuf/jme

jme3 in blender 2015 02 27 18 53 53

The new repository organisation : xbuf · GitHub
The new website (wip) : http://www.xbuf.org/

comments, bugs, suggestions, logos are welcome.