New JSON format (import/export) for jMonkeyEngine3

[SwiftWolf]

Hello everybody…

I was thinking of a new way to export our models or scenes within jme3 (file type), and what better than a new file format (in this chaos JSON).

A while ago I created something similar, only with another json library, although
it worked partially, unfortunately it had several errors (after all, I was a newbie ).

Currently we can export in 2 formats; binary and xml (please correct me if this is incorrect). The idea is to create this exporter and importer for scenes saved with json.

You can see the code I have generated for this new exporter here. At the moment the code is dirty…

I did some testing and it’s pretty stable:

Blue is a cube created from code, red is a cube loaded from a json file.

I haven’t submitted any PR yet, what do you think of this idea?

[ NOTE ]

This crazy idea is simply to have another option (format) when saving our worlds made with jme.

[ EDIT ]
We would have an output for the objects as follows:

{
  "metadata": {
    "+/jMe3.Signature": 1246577971,
    "+/jMe3.Version": 3
  },
  "root": {
    "+/jMe3.Class": "com.jme3.scene.Geometry",
    "+/jMe3.Id": "com.jme3.scene.Geometry@1904948097",
    "+/jMe3.Versions": [
      1,
      0
    ],
    "jme3.name": "Box",
    "jme3.lights": {
      "+/jMe3.Class": "com.jme3.light.LightList",
      "+/jMe3.Id": "com.jme3.light.LightList@1601477037",
      "+/jMe3.Versions": [
        0
      ],
      "jme3.lights": []
    },
    "jme3.overrides": [],
    "jme3.controlsList": [
      {
        "+/jMe3.Class": "com.jme3.export.json.Test$CubeControl",
        "+/jMe3.Id": "com.jme3.export.json.Test$CubeControl@727460961",
        "+/jMe3.Versions": [
          0,
          0
        ],
        "jme3.spatial": {
          "+/jMe3.Reference": "com.jme3.scene.Geometry@1904948097"
        },
        "jme3.speed": 1.5,
        "jme3.dir": {
          "+/jMe3.Class": "com.jme3.math.Vector3f",
          "+/jMe3.Id": "com.jme3.math.Vector3f@-627115336",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.x": 1.0,
          "jme3.y": 1.0,
          "jme3.z": 1.0
        }
      }
    ],
    "jme3.mesh": {
      "+/jMe3.Class": "com.jme3.scene.shape.Box",
      "+/jMe3.Id": "com.jme3.scene.shape.Box@1108493039",
      "+/jMe3.Versions": [
        0,
        0,
        0
      ],
      "jme3.modelBound": {
        "+/jMe3.Class": "com.jme3.bounding.BoundingBox",
        "+/jMe3.Id": "com.jme3.bounding.BoundingBox@1387908284",
        "+/jMe3.Versions": [
          0,
          0
        ],
        "jme3.xExtent": 1.0,
        "jme3.yExtent": 1.0,
        "jme3.zExtent": 1.0
      },
      "jme3.vertCount": 24,
      "jme3.elementCount": 12,
      "jme3.instanceCount": 1,
      "jme3.buffers": {
        "0": {
          "+/jMe3.Class": "com.jme3.scene.VertexBuffer",
          "+/jMe3.Id": "com.jme3.scene.VertexBuffer@459811830",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.components": 3,
          "jme3.usage": "Static",
          "jme3.buffer_type": "Position",
          "jme3.dataFloat": [
            -1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            -1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            -1.0,
            -1.0,
            -1.0,
            -1.0,
            -1.0,
            -1.0,
            1.0,
            1.0,
            -1.0,
            1.0,
            1.0,
            -1.0,
            -1.0
          ]
        },
        "2": {
          "+/jMe3.Class": "com.jme3.scene.VertexBuffer",
          "+/jMe3.Id": "com.jme3.scene.VertexBuffer@639272810",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.components": 3,
          "jme3.usage": "Static",
          "jme3.buffer_type": "Normal",
          "jme3.dataFloat": [
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0,
            0.0,
            -1.0,
            0.0
          ]
        },
        "3": {
          "+/jMe3.Class": "com.jme3.scene.VertexBuffer",
          "+/jMe3.Id": "com.jme3.scene.VertexBuffer@252854539",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.components": 2,
          "jme3.usage": "Static",
          "jme3.buffer_type": "TexCoord",
          "jme3.dataFloat": [
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0,
            0.0,
            0.0,
            1.0,
            1.0,
            1.0
          ]
        },
        "9": {
          "+/jMe3.Class": "com.jme3.scene.VertexBuffer",
          "+/jMe3.Id": "com.jme3.scene.VertexBuffer@651101097",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.components": 3,
          "jme3.usage": "Static",
          "jme3.buffer_type": "Index",
          "jme3.format": "UnsignedShort",
          "jme3.dataUnsignedShort": [
            2,
            1,
            0,
            3,
            2,
            0,
            6,
            5,
            4,
            7,
            6,
            4,
            10,
            9,
            8,
            11,
            10,
            8,
            14,
            13,
            12,
            15,
            14,
            12,
            18,
            17,
            16,
            19,
            18,
            16,
            22,
            21,
            20,
            23,
            22,
            20
          ]
        }
      },
      "jme3.xExtent": 1.0,
      "jme3.yExtent": 1.0,
      "jme3.zExtent": 1.0
    },
    "jme3.material": {
      "+/jMe3.Class": "com.jme3.material.Material",
      "+/jMe3.Id": "com.jme3.material.Material@1252633752",
      "+/jMe3.Versions": [
        2
      ],
      "jme3.material_def": "Common/MatDefs/Misc/Unshaded.j3md",
      "jme3.parameters": {
        "BackfaceShadows": {
          "+/jMe3.Class": "com.jme3.material.MatParam",
          "+/jMe3.Id": "com.jme3.material.MatParam@-2139704160",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.varType": "Boolean",
          "jme3.name": "BackfaceShadows",
          "jme3.value_bool": true
        },
        "PointSize": {
          "+/jMe3.Class": "com.jme3.material.MatParam",
          "+/jMe3.Id": "com.jme3.material.MatParam@1994856648",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.varType": "Float",
          "jme3.name": "PointSize",
          "jme3.value_float": 1.0
        },
        "Color": {
          "+/jMe3.Class": "com.jme3.material.MatParam",
          "+/jMe3.Id": "com.jme3.material.MatParam@18566027",
          "+/jMe3.Versions": [
            0
          ],
          "jme3.varType": "Vector4",
          "jme3.name": "Color",
          "jme3.value_savable": {
            "+/jMe3.Class": "com.jme3.math.ColorRGBA",
            "+/jMe3.Id": "com.jme3.math.ColorRGBA@1165711330",
            "+/jMe3.Versions": [
              0
            ],
            "jme3.r": 0.53376704,
            "jme3.g": 0.124457,
            "jme3.b": 0.94186205,
            "jme3.a": 1.0
          }
        }
      }
    }
  }
}

The ‘root’ node is where the entire scene (model) is stored, the ‘metadata’ is the exporter’s own.

[sgold]

It’s not a crazy idea.

[danielp]

I don’t view this as much different from having an xml-based format, but json (and oftentimes, the libraries to read & write it) tends to be significantly more pleasant to work with than xml, in my opinion. I think this is a worthwhile endeavor.

[codex]

Neat idea! Are you using a JsonImporter/JsonExporter, so that Savables can be used?

[capdevon]

Hi @SwiftWolf ,

As you know, the process of reviewing, testing, approving and maintaining new jME engine features is very rigorous and time-consuming. Release cycles are also significantly extended, whether we like it or not.

I took a look at your code. Instead of sending a PR or working on a personal fork of jme, I have a suggestion for you:
create a public project on your GitHub account (possibly with Gradle). This way, you can develop your ideas and showcase your progress and results more quickly. The community can then try your library in their own projects and help you improve it by sending useful PRs and suggestions (see Minie, Heart, MonkeyWrench, and other popular libraries). Once the library is complete, tested, and stable, perhaps the community might decide to integrate it into the engine in the future.

The golden rule is, “If you can outsource new features in a library, do it!”

I am also developing on github a new json format for j3md and j3m files. I will make it public as soon as possible and try to add support for YAML format as well, like for the Ardor3D engine (see YamlMaterialReader.

–

For example, here is the PBRLighting.j3md file converted to .json format:

{
  "MaterialDef": {
    "name": "PBR Lighting", // Assuming this is the material name
    "MaterialParameters": [
      {
        "type": "Int",
        "name": "BoundDrawBuffer",
        "value": null // Assuming no default value provided in the original definition
      },
      {
        "type": "Float",
        "name": "AlphaDiscardThreshold",
        "value": 0.0 // Assuming a default value
      },
      {
        "type": "Float",
        "name": "Metallic",
        "value": 1.0
      },
      {
        "type": "Color",
        "name": "Specular",
        "value": [1.0, 1.0, 1.0, 1.0]
      },
      {
        "type": "Float",
        "name": "Glossiness",
        "value": 1.0
      },
      {
        "type": "Texture2D",
        "name": "ParallaxMap",
        "colorSpace": "Linear"
      }
      // ... and so on ...
    ],
    "Techniques": [
      {
        "name": "Default", // Assuming technique name can be extracted
        "LightMode": "SinglePassAndImageBased",
        "VertexShader": "Common/MatDefs/Light/PBRLighting.vert",
        "FragmentShader": "Common/MatDefs/Light/PBRLighting.frag",
        "ShaderLanguages": ["GLSL300", "GLSL150", "GLSL110"],
        "WorldParameters": [
          "WorldViewProjectionMatrix",
          "CameraPosition",
          "WorldMatrix",
          "WorldNormalMatrix",
          "ViewProjectionMatrix",
          "ViewMatrix"
        ],
        "Defines": [
          {
            "name": "BOUND_DRAW_BUFFER",
            "param": "BoundDrawBuffer"
          },
          {
            "name": "BASECOLORMAP",
            "param": "BaseColorMap"
          },
          {
            "name": "VERTEX_COLOR",
            "param": "UseVertexColor"
          }
          // ... other defines mapping parameters to names ...
        ]
      },
      {
        "name": "PreShadow", // Assuming technique name can be extracted
        "VertexShader": "Common/MatDefs/Shadow/PreShadow.vert",
        "FragmentShader": "Common/MatDefs/Shadow/PreShadowPBR.frag",
        "ShaderLanguages": ["GLSL300", "GLSL150", "GLSL110" ],
        "WorldParameters": [
          "WorldViewProjectionMatrix",
          "WorldViewMatrix",
          "ViewProjectionMatrix",
          "ViewMatrix"
        ],
        "Defines": [
          {
            "name": "BOUND_DRAW_BUFFER",
            "param": "AlphaDiscardThreshold"
          },
          {
            "name": "DISCARD_ALPHA",
            "param": "AlphaDiscardThreshold"
          }
          // ... other defines mapping parameters to names ...
        ],
        "ForcedRenderState": {
          "FaceCull": "Off",
          "DepthTest": true,
          "DepthWrite": true,
          "PolyOffset": [5, 3],
          "ColorWrite": false
        }
      }
      // ... other techniques following the same structure ...
    ]
  }
}

Here Material.json prototype:
The texture definition block may require modification to support TextureArray parameters. In such cases, the block might need to be restructured as an array of elements to accommodate multiple textures.

{
	"Material": {
		"name": "MyMaterial",
		"def": "Common/MatDefs/Light/Lighting.j3md",
		"materialParameters": [
			{
				"name": "Diffuse",
				"value": [0.505882,0.505882,0.505882,1]
			},
			{
				"name": "UseMaterialColors",
				"value": true
			},
			{
				"name": "Ambient",
				"value": [0.2,0.2,0.2,1]
			},
			{
				"name": "DiffuseMap",
				"texture" : {
					"path" : "Models/Ferrari/car.jpg",
					"wrapMode" : "Repeat",
					"minFilter" : "BilinearNoMipMaps",
					"magFilter" : "Bilinear",
					"flipY" : false
				}
			},
			{
				"name": "Specular",
				"value": [0.5,0.5,0.5,1]
			},
			{
				"name": "Shininess",
				"value": 12.5
			}
		],
		"additionalRenderState": {
			"faceCull": "Back",
			"blend": "Off",
			"depthWrite": true,
			"colorWrite": true,
			"depthTest": true,
			"wireframe": false,
			"polyOffset": [1.0,1.0]
		}
	}
}

[SwiftWolf]

@codex

Are you using a JsonImporter/JsonExporter, so that Savables can be used?

Exactly, at the moment I am using the JsonImporter/JsonExporter objects to interact with the Saveables

[SwiftWolf]

@capdevon

As you know, the process of reviewing, testing, approving and maintaining new jME engine features is very rigorous and time-consuming. Release cycles are also significantly extended, whether we like it or not.

Certainly approving a new functionality so that it can be integrated into the engine takes time, several tests must be performed before doing so.

@capdevon

create a public project on your GitHub account (possibly with Gradle). This way, you can develop your ideas and showcase your progress and results more quickly. The community can then try your library in their own projects and help you improve it by sending useful PRs and suggestions (see Minie, Heart, MonkeyWrench, and other popular libraries). Once the library is complete

Thanks for the suggestion, I was looking at this point; It was difficult to do much larger tests since I have it built into a branch that is connected to the engine (yep, it’s bad).

It looks like this new way of exporting jme objects could be integrated into the core engine in the future (possibly). I’ll start working on a small library to make it more efficient to test individually.

[pspeed]

Just a sort of historical note.

j3o is not meant to be an interchange format. It’s meant as a way to package your assets into an engine-efficient form that can be quickly loaded into your distributed game. It has only basic backwards compatibility built in, essentially no forward compatibility, and can break between JME versions (though we try really hard not to). Some major shift in functionality could easily break old j3o files and that’s normal.

The only real reason there ever was an XML format was for debugging import/export issues… which is why it got woefully out of date and broken for a while.

Another fact for consideration: JME tends to not incorporate things into the “core engine” that are “game editor specific”. So whatever the case, historically-speaking, a JSON layer for an engine-specific packaging format would probably never be incorporated into jme-core itself. Any jme-core changes necessary to facilitate it would likely also be frowned upon (ie: no scatters json annotations all over the codebase and things like that).

For reasons stated in other comments and for the above reasons, for sure an external library is the best way to start.

If you are now thinking “Gee, what WOULD it take to make a stable JSON-based interchange format for JME?” ie: without inheriting the fragile nature of j3o… that discussion probably ends in a JME gltf exporter. (Which would be an awesome tool, really.)

[SwiftWolf]

Hi @pspeed

@pspeed

So whatever the case, historically-speaking, a JSON layer for an engine-specific packaging format would probably never be incorporated into jme-core itself

There seems to be a confusion (sorry), I don’t mean it’s in the ‘jme-core’ module, like you said; This module is specific to everything related to games. Furthermore, it would not be sensible to think about introducing a plugin like this in said module.

@pspeed

If you are now thinking “Gee, what WOULD it take to make a stable JSON-based interchange format for JME?”

To think that is strange. As you mentioned, gltf is the most efficient way there is. Thinking about this little plugin being able to interact as an exchange format is a bit much (it would never happen).

JUST TO CLARIFY, THIS IDEA IS ONLY FOR A NEW PACKAGING FORMAT, NOT FOR AN EXCHANGE FORMAT LIKE ‘gltf’, don’t be confused…

@pspeed

exportador gltf de JME (que sería una herramienta increíble, en verdad).

It would be great to have an exporter for gltf (natively, if you can put it that way), however it’s not in my plans (at the moment).

[pspeed]

@SwiftWolf

JUST TO CLARIFY, THIS IDEA IS ONLY FOR A NEW PACKAGING FORMAT, NOT FOR AN EXCHANGE FORMAT LIKE ‘gltf’, don’t be confused…

Yeah, I understand that. I’m just trying to understand what feature it’s meant to provide. Else it just seems like it’s doing the same job as j3o but… poorly? (ie: ginormous text files that take longer to load)

Not trying to be critical… just trying to understand what the use-case is. The only one I could imagine was supporting a game editor or stable ‘save’ format of some kind. .j3o is a packaging format like a .jar file. (I wouldn’t make a json version of a .jar either, for example.)

[SwiftWolf]

@pspeed

…poorly? (ie: ginormous text files that take longer to load)

It is normal for a file structured as JSON to take longer than a serialized and optimized one (j3o)

@pspeed

Not trying to be critical

hahahahaha… (Sorry, I thought you were funny, I’m not trying to offend anyone.), don’t worry; It is necessary to have that attitude to approve or reject anything.

@pspeed

just trying to understand what the use-case is

Maybe this answer seems vague to you, it could simply be a good substitute for the xml format that jme currently has, just one more way to export objects…

[pspeed]

@SwiftWolf

Maybe this answer seems vague to you, it could simply be a good substitute for the xml format that jme currently has, just one more way to export objects…

But the XML format was really on there to debug the binary format. No one was really expected to package giant XML files with their game as assets. (XML was even broken for years until very recently… that’s how often it was used. It was broken an no one noticed.)

The assumption is that if you are trying to package things for a game that you would prefer them to be small and fast… thus .j3o. For things that are big and slow, I’m not sure what the general use-case is.