File loading system (input requested)

Basic questions we need:


  1. What are your feelings on a generic XML loader for jME? After a generic XML loader,there would be converter functions to convert .ms3d or .3ds to that generic loader format, so people could load ms3d files.


  2. Is XML the right way to go? Do you agree with using SAX to load the XML?


  3. How do you propose a generic scene loader/writter be able to write and load custom built controllers/nodes/renderstates without having to change the loader class code?


  4. Do you feel a generic file loader is even needed? What are your takes on how jME should do file loading?







    What’s the communities ideas on jME’s file loading system. Mojo has the idea of writting a generic file loader, which would be optimized by us and to load file formats (.ms3d , 3ds, ect) a converter (which is much simpler than a total laoder) would be written to convert that format to jME’s format.



    jME’s format would be an XML file. There’s an example of what it may look like on the CVS. Look at it and give me your thoughts. There’s also a test file that loads it.



    ALso, there is a problem of writting a generic “scene” loader and saver to laod and save -any- scene that jME has. One idea was to let classes extend loadXML interface with savetoFile and loadFromFile functions a user would write and imbed in their class. Just diffrent ideas.

I have never parsed “schema validating” XML files by SAX, but I have done this some times with DTD defined files.



What I can see (but i can be wrong, i’m far from being an XML guru, you surely are better than me on this matter :// ) :


  • it is definitely better to parse a file by SAX if it can be huge (which can obviously be the case with XML-defined models), because DOM defines an object for each element of the XML tree and therefore can be extensively ressource-consuming…


  • on the other hand, with SAX, all have to be done almost “by hand”, but as the XML structure must be transformed to jme objects and you won’t need to look and search in the XML structure before, I think that SAX will “definitely” do it :wink:


  • as for loading custom nodes without changing the code of the loader, what i can see without thinking a lot about it is have a capability to optionaly define the class fully qualified name somewere in the file when it is needed, then use reflection in the loader to do the job. This could result in heavy time consomption, but it could be reduced by creating a cache of previously loaded classes (for each of these names), and then just create new instances out of them…


  • as for your XMLLoader code is it normal just to return null for resolveEntity ? In this case, i think it will not be possible to validate the file against your schema if the schema file is not in the same directory (or so I think, if it is the same that validation against DTD) : in fact, not returning null can be interesting when you want to store the schema or DTD in the jar of your application, for example…
  • and why do you use this characters method ? I never had to use it myself, but i always used startElement, and iterating through the attributes for each particular element did the job for me. I used to have a different handler than the main class too, I often found it better when the "managing" code for the parsing itself begins to grow.
  • and last, I like very much the stack design :smiley:
- it is definitely better to parse a file by SAX if it can be huge (which can obviously be the case with XML-defined models), because DOM defines an object for each element of the XML tree and therefore can be extensively ressource-consuming...


Agreed, it's the reason I choose SAX. In 3d graphics, you never have enough speed.

- as for loading custom nodes without changing the code of the loader, what i can see without thinking a lot about it is have a capability to optionaly define the class fully qualified name somewere in the file when it is needed, then use reflection in the loader to do the job. This could result in heavy time consomption, but it could be reduced by creating a cache of previously loaded classes (for each of these names), and then just create new instances out of them...


That was my idea. See the sample code at the bottom. I don't think there's any other way to load "any" class. Yea it might be a tad slow to load, but
a) That's only for custom classes
b) they're only slow to load, not to use

- as for your XMLLoader code is it normal just to return null for resolveEntity ?


I'm using an xsd to validate my XML file. As far as I can tell, the built in SAX reader for the current release of java doesn't support xsd validation (only DTD). Of course I could download one that does, and then make everyone else that uses jME download another 2MB file jsut for validation, but i figured it's not worth it. From what I read, Java 1.5 will have a SAX reader that supports xsd validation. Someone correct me if I'm wrong, but this is what I can gather so far.

- and why do you use this characters method ?

<Vertex name="blarg"> 1 2 3 4 5 6 </Vertex>

atts only gives me access to name="blarg", but not 1 2 3 4 5 6. I need to use characters to do that. Of course I could change it to
<Vertex name="blarg" data="1 2 3 4 5 6"/>

but the first seemed more natural when data could be 1000 elements long. I'm open to discussion on which is better though

- and last, I like very much the stack design


My design idea was to use a Stack to keep track of where I'm at in the XML file because all lower elements can be .attachChild() or .addController() to the element above them. I use a hashtable to put the same "thing" in multiple places. So for example to save memory I could make one refrence to a Mesh or RenderState in the XML and put the same refrence in 4-5 places.

Here was my idea for dynamic class loading

interface XMLloader{
public static Object createFromXML(String args);
public String writeToXML();
}

Any class that is unknown to the parser that wants to be saved would have its writeToXML() function called on the -object- and the returning string data would be saved like for example

Box b=new Box("myBox",new Vertex(-1,-1,-1),new Vertex(1,1,1));
b.writeToXML(); // would return "myBox -1 -1 -1 1 1 1"



So the XML file would look like

<genericjmeobject type="com.jme.scene.box" args="myBox -1 -1 -1 1 1 1/>



When the XML file is loaded, the static method of -type- would be called and -args- passed, returning an object and that object would be added to the scene graph in a way fit for that object.

Just ideas of course
As far as I can tell, the built in SAX reader for the current release of java doesn't support xsd validation (only DTD).


I didn't think it was the case, but sadly this is it :'( I agree not to ship a several MB parser with jme just for that, and wait for 1.5. But may be it could be useful to use a built-in provisional DTD (coming from the schema but over-simplified), and then it would be possible for pre-1.5 JVM to have a low level of validation ? (it will still be possible to use xsd validation for files out of jme) ? But of course, you would prefer to code the core of the loader ;)

For the character datas, you're apparently right (I didn't knew it because i tend not to use raw datas in elements, without really noticing it, but I understand you have to do it in this case).

Look forward to see your progress in the loader :D

Don’t know how to remove files from CVS, but I moved SAXReader to a new package (jme.scene.model.XMLParser) because it was starting to be more than 2 files.



I’ve changed how the XML file will look to reflect jME more. Because I don’t know how to model that in xsd the way I want it (inheritance issues), I’ve dropped using an xsd validator for now and will make one later on.



I’ve also written a XMLWriter that writes a node to XML format. I have a testXML writer class (jmetest.renderer.loader.TestXMLWriter) that you can use to test the XMLWriting. I’ve discovered a problem with saving a node to XML format, specificly the TextureStates. A texture state knows nothing about the file it was loaded from so how do I know what to point a texture state to when it’s saved?



I’m also brainstorming ideas of how to signal to a written XML file that a spatial/state/controller in one place is the same one that’s in another place.



I’ve also written a XMLloadable interface that users can implement to store their own custom classes in XML format. It’s definantly not the best way to do it, mainly because interfaces can’t have static methods, and I don’t have any ideas (yet) for an elegant way to get around this.



Anywho it’s there to view.



Here’s an example (so you don’t have to dig thru CVS to look at it) of the file data/XML docs/newSampleScene.xml which shows how I’ve changed XML files to look like



<?xml version="1.0" encoding="UTF-8"?>
<scene xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="">
    <sharedtypes>
        <sharednodeitem ident="red">
            <materialstate emissive="0 0 0 1" alpha="1" ambient=".1 .1 .1 1" diffuse="1 0 0 1" shiny="64" specular="1 1 1 1"/>
        </sharednodeitem>
    </sharedtypes>
    <node name="man">
        <mesh name="head" translation="0 0 3" scale="5">
         <materialstate emissive="0 0 0 1" alpha="1" ambient=".1 .1 .1 1" diffuse="0 1 0 1" shiny="64" specular="1 1 1 1"/>
         <vertex>
            0 0 0 1 0 0 0 1 0 </vertex>
         <normal>0 0 1 0 0 1 0 0 1</normal>
         <color>1 0 0 1 0 1 0 1 0 0 1 1 </color>
         <texturecoords/>
         <index> 0 1 2
            </index>
        </mesh>
      <primitive type="box" params="MyBox -1 -1 -1 1 1 1" translation="7 7 7">
            <publicobject ident="red">
            </publicobject>
            <texturestate file="CVS root/src/jmetest/data/images/Monkey.tga"/>
        </primitive>
        <xmlloadable class="com.jme.scene.shape.Box" name="loadedBox" args="0 0 0 1 1 1"/>
    </node>
</scene>

Just a quick note before I finish packing and head to the airport.



Things are looking good, and it looks like some decent progress has been made. I haven’t been able to run the code, although I updated and glanced at it a little bit.



I notice you put a URL in TextureState to try to keep track of the Texture location. You should actually put the URL/String in Texture itself, as a TextureState can have multiple Textures. (1 per texture unit).



I haven’t had a chance to look at the XMLLoadeable interface yet, so I can’t comment.



Things are looking good and I look forward to seeing the first real model.



P.S. To remove a file you have to do a cvs remove then a cvs commit.

Well it -kind of- works now. In the testWriter file, I actually load a Node using a MilkShape loader class and then I export that node to XML using my XMLWriter, then I read that XML with SAXReader and attach it to rootnode and it work!!

If you have the ability to load other files for meshes, the ability to add controllers, a collision controller, And a library of controllers. You could “snap together” simple games with jme. (A seen editor would help)





for example:



Modal.xml:


<?xml version="1.0" encoding="UTF-8"?>
<scene xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="">
    <sharedtypes>
        <sharednodeitem ident="red">
            <materialstate emissive="0 0 0 1" alpha="1" ambient=".1 .1 .1 1" diffuse="1 0 0 1" shiny="64" specular="1 1 1 1"/>
        </sharednodeitem>
    </sharedtypes>
    <node name="amesh">
        <mesh name="head" translation="0 0 3" scale="5">
         <materialstate emissive="0 0 0 1" alpha="1" ambient=".1 .1 .1 1" diffuse="0 1 0 1" shiny="64" specular="1 1 1 1"/>
         <vertex>
            0 0 0 1 0 0 0 1 0 </vertex>
         <normal>0 0 1 0 0 1 0 0 1</normal>
         <color>1 0 0 1 0 1 0 1 0 0 1 1 </color>
         <texturecoords/>
         <index> 0 1 2
            </index>
        </mesh>
    </node>
</scene>



Main.xml


<?xml version="1.0" encoding="UTF-8"?>
<scene xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="">
    <node name="mesh1" import="Modal.xml"><posishon x=2 y=4 z=10/></node>
   <node name="mesh2" import="Modal.xml"><posishon x=5 y=4 z=10/></node>
   <node name="loop" import="Modal.xml"><posishon x=10 y=4 z=10/><controller scr="com.jme.controller.Rotating"/></node>
</scene>

(A seen editor would help)


The hope would be someone could build the games in another tool (way better than any scene editor we could make) like 3ds or maya or Milkshape3D, and we would have an exporter to export the game to jME format.
"Cep21" wrote:
The hope would be someone could build the games in another tool (way better than any scene editor we could make) like 3ds or maya or Milkshape3D, and we would have an exporter to export the game to jME format.

Milkshape3D can’t(it has primitive materials and you can't set stats or controllers with it), 3ds and maya are expensive }:-@. Gmax would be good but you have to pay to make an exporter. A seen editor would only have to add objects and move them models could still be made in modelers like Milkshape3D.

I was searching for the web and found this which is similar to what i suggested.

Ah I see. Something like that is a huge project. I suppose there are advantages of one built ‘around’ jME, but the hope would be to let users build scenes in whatever modeling enviorment they want and conversion utilities to convert those scenes to jME format. I didnt touch on it in my previous post, but I totally agree with (and hope) that the XML loader should be able to piece together entire subsets of games from a library. Only time will tell…