This is an interesting project beside of OpenGEX. I like the idea of a “data format for exchange 3d”. As every game developer, I have a chronic headache with the asset pipeline, and it’s getting worse. I also have some technical experiments with my team about way to solve a small scale asset pipeline, repo; and scalable architecture if chance that your team become bigger, or work in enterprise. Imagine a “file” to be exported, imported by CCDs, engines; “bitstream” flow from CCDs into our rendering pipeline… Well, something like that…
The topic we referenced is “Universal format” or “Unified format”. But can it be “developer friendly” as a “format” without solving several problems, I mean I doubt about its usefulness.
We’ve kind of faced the trade off between “easy in design time” and “fast small in run-time”, and really hard to find a magnitude to fit all cases (impossible?). Later, we decide to move on with a much smaller problem: Design a binary (bitstream) flows (dataflow, workflow…) from other applications toward JME3. For my team, its like Blender, 3DSMax, Maya and PTS into JME (Java and OpenGL).
What we discovered so far is to design the pipeline as running progresses, focused in the flows (structured bitstream) via ports, instead of imaging a static, consistent storage. So 3d files for example are just persisted form, one form in serveral forms of data in the pipeline.
And because we don’t have a sophisticated overall universal structure (only bitstream), we focused in generic decoder with schema in ports instead. For ex: decoder from 3dsmax into generic bitstream. We also use Protobuff to encode and transfer data in the pipe. But note that we stick with the bitstream instead of depend in schema.
We also in early stage but there are some technical benefits:
- Thanks to bitstream abstraction, we have used Preon, Protobuff, Netty to read/write/encode/decode/send/get between pipeline.
- With pipeline abstraction, we’re experimenting Apache Storm to run works over data easily and managable, and most importantly fault tolerant! We can convert massive project models, we can render to movie some part of game if we need.
- Thanks to JME3 asset abstraction, we have hooks into our production pipeline. So models, sounds are reloaded seamlessly in-game. We’re experimenting reloaded through internet connections on devices (yes, reload new assets on phones…). In JME side, it’s piece of cake; but we have to built an assets server for that job and that’s another long story.
- We’d like to push it further, in the direction towards a well defined, well structured storage repository system for assets for JME. It’s like git for assets: multi versions, easy to folk, easy to solve (non-geometry) conflicts.
Now about Pgex, i have a few questions:
First ) it’s not a “Format”? It’s like OpenGEX specs hook into Protobuff decoder facilities. So it has a generic structure (not really totally general but gamedev infos oriented), with a generic decoder. Talking about the “format”, I really like to know about the structure you provide to solve specific game problems.
2) its has external references? The problem @abies talked about is one of problem if this so call “unified format” has external references to other resource in its content. Because its try to self contains as much as possible, it’s either wasteful (also not reusable) or plain useless. In opposite direction, it’s become a hash contains only references to others. This unified format looks a bit funny: like HTML once try to carry data in its own, but references all over the places.
3) it has domain specific infos? Physics, IK, decoding, localization, user’s data… We can not have standards for everything in this world (someone blame COLLADA because it’s the chosen one but become naughty after all)
The solution I’ve talked above (called AtomExAsset) admit the nature of external resources, try to obtain to solid version if it can, retry several times, retry other resolutions (if not configurated beforehand) and return the result or a place holder peacefully. Because in it spirit, it’s only return the assets that suite JME3, and WORK without bad exceptions.
“It will not stop developer to continue working”.
That’s my idiom of “developer friendly” and some thoughts i want to share.
Well, incomplete wiki like always, but I will back to this sometimes …