[SOLVED] BinaryExporter and materials

Yeah, life got him. heh.

I use this now for sweeping the sdk project files into the gradle project and its pretty sweet.


task syncDependencies(type: Sync) {
    description 'Copies the assets directory from external netbeans project to the target directory.'
    from '../PathToSDKProject/assets'
    exclude ('**/*.j3odata','**/*.mesh','**/*.skeleton','**/*.mesh.xml','**/*.skeleton.xml','**/*.scene','**/*.material','**/*.obj','**/*.mtl','**/*.3ds','**/*.dae','**/*.blend','**/*.blend*[0-9]','**/*.bin','**/*.gltf')
    into 'assets'
}

You can automate it as you know with

compileJava.dependsOn(syncDependencies)

I think I may go the way @Ali_RS went and build individual jars for each asset next.

Looking forward to nb13 since there is no JavaFX-Gradle-Plugin for 1.8 now. Like waiting for a new toy at Christmas as a kid.

1 Like

Narrowed down the material problem causes but no solution for a non SDK fix.

It seems that if you use test-data model in the SDK by copy/paste they will work fine in SDK no matter where you put them. Even moving them around works fine.

However, the same model will not find its material in a gradle project unless this model is either in the exact same folder structure as in test-data or it has been converted to a j3o by the sdk and that model is in that exact same folder structure as the converted model.

So the jaime model for example in the test-data folder Models.Jaime will only work in the gradle project asset folder Models.Jaime even though it works anywhere in the sdk. You cant convert it so thats it for Jaime in anything except the SDK.

So to use external assets from the SDK, they must be converted in the SDK first and placed in the exact same folder structure in the gradle project.

What a pain in the ass this has been.

For what it’s worth, I downloaded a bunch of different models from sketchfab to play with and plan to poke at what it would take to write a proper command line converter. (And if any core changes are required/desired to make that work.)

I don’t know how much luck I will have… but I didn’t want you to feel totally alone. :slight_smile:

Misery does love company but I hope this one gets easier for everyone so they can avoid it.

2 Likes
task syncDependencies(type: Sync) {
    description 'Copies the assets directory from external netbeans project to the target directory.'
    from '../PathToSDKProject/assets'
    exclude ('**/*.j3odata','**/*.mesh','**/*.skeleton','**/*.mesh.xml','**/*.skeleton.xml','**/*.scene','**/*.material','**/*.obj','**/*.mtl','**/*.3ds','**/*.dae','**/*.blend','**/*.blend*[0-9]','**/*.bin','**/*.gltf')
    into 'assets'
}

Well discovered some more quirkiness using gradle. Being new to me probably not to others but when syncing folders i found out that you can always create a sub directory, and add files to it in the “from” directory and it will propagate the “into” directory.

However, at least for me, if i were to delete those files and or the sub directory, or move that directory, the “from” directory now has an empty folder where once there was none. The sync, at least for me, would not update the “into” directory now because it found an empty folder in the “from” directory and ignores it. This means my “into” directory was now out of sync with the “from” directory.

I created a chain of task calls to fix it using Pauls default gradle templates createDirs task so now I always have a matching synced folder when I want one.

The first task is still the syncDependencies task but now set to dependOn Pauls default template createDirs task which is set to dependOn a delete task.

task syncDependencies(type: Sync) {
    dependsOn.createDirs
    description 'Copies the assets directory from external netbeans project to the target directory.'
    from '../PolyAssets/assets'
    exclude ('**/*.j3odata','**/*.mesh','**/*.skeleton','**/*.mesh.xml','**/*.skeleton.xml','**/*.scene','**/*.material','**/*.obj','**/*.mtl','**/*.3ds','**/*.dae','**/*.blend','**/*.blend*[0-9]','**/*.bin','**/*.gltf')
    into 'assets'
}

I left the createDirs task intact but it could be cleaned up to remove unneeded things like creating stubs.

task (createDirs).doLast {
    dependsOn makePretty
    def pkg = 'mygame'
    def dirs = [
//        file("./src/main/java/$pkg"),
//        file("./src/main/resources"),
        file("./assets/Interface"),
        file("./assets/MatDefs"),
        file("./assets/Materials"),
        file("./assets/Models"),
        file("./assets/Scenes"),
        file("./assets/Shaders"),
        file("./assets/Sounds"),
        file("./assets/Textures"),
    ]

    dirs.each {
        if( !it.exists() ) {
            println "Creating " + it
            it.mkdirs()
        }
        if( it.listFiles().length == 0 ) {
            def stub = new File(it, 'removeme.txt')
            println "Creating stub file to allow git checkin, file:$stub"
            stub.text = "Remove me when there are files here."
        }
    }
}

The makePretty task.

task makePretty(type: Delete) {
    description 'Deletes all files in the target directory without deleting the target directory.'
    delete file('assets').listFiles()
}

This works real well and I can move things around in the “from” folder, add to or remove them, leave empty directories anywhere I want and its real fast.

If there are better ways to do this please advise.

Edit: Never mind. Failed again for some reason once all the folders were empty.
That loud smacking sound you hear is me punching the shit out of gradle.

Note, I am not doing this way anymore. I do not know what is the reason you want to do this.
Now I am using just the default asset structure from Paul’s template gradle project. One asset.jar for all of my assets and using runtime files('assets') in build.gradle.

Edit:
I am posting my new approach in case:

I found that getdown already has a mechanism for this by generating patch file, which I think I am going to use this instead of writing my own patcher.

Ok. read it in another post. Sounded good on paper.

Found out my sync problem is a gradle bug. At least part of it is.

I had that getdown bookmarked already. I dont even remember doing it but it looks like the way to go to me also.

For what it’s worth, here is the product of my fooling around for this evening.

With a prototype release:

It’s a command line tool that can load a non-j3o model from one place, convert it to j3o in a new directory structure, including copying over its dependent textures.

Next would be to add some scripting to “adjust” things during the process as well as to do things like mark materials for export as j3m, etc…

But it proves that the concept works. I was able to convert some models I downloaded from Sketchfab to j3o into a Models/foo subdirectory, textures were copied into Models/foo/textures automatically… and the j3o loaded fine in my application.

The rest is “usability”. (For example, I really want to be able to rescale nodes, center them, etc… because a lot of models seem to treat 1 unit = 1 inch or 1 unit = 1 centimeter.)

Edit: if you download the zip and run the jmec script in there with no arguments it will provide command line help.

Edit 2: Added the temporary command line help to the project readme… repasted here while I’m spamming the thread:

Current command line help:
        _   __  __   ___    ___
     _ | | |  \/  | | __|  / __|
    | || | | |\/| | | _|  | (__
     \__/  |_|  |_| |___|  \___|

    JME3 Asset Converter v1.0.0

Usage: jmec [options] [models]

Where [models] are a list of j3o files.

Where [options] are:
 -sourceRoot <dir> : specifies the asset root for the models.
       Model paths will be evaluated relative to this root.

 -targetRoot <dir> : specifies the asset target root for writing
       converted assets and their dependencies.

 -targetPath <path> : a path string specifying where to place
       the copied/converted assets within the targetRoot.  Any
       internal asset keys will be 'rehomed' to this path.

Examples:

>jmec -sourceRoot C:\Downloads\CoolModel -targetRoot assets -targetPath Models/CoolModel C:\Downloads\CoolModel\AwesomeThing.gltf

 Converts C:\Downloads\CoolModel\AwesomeThing.gltf to a j3o and writes it
 to ./assets/Models/CoolModel/AwesomeThing.gltf.j3o

 Any dependent textures, etc. relative to C:\Downloads\CoolModel will
 be copied until the appropriate ./assets/Models/CoolModel/* subdirectory.

 For example:
    C:\Downloads\CoolModel\textures\awesome-sauce.jpg
 Gets copied to:
    ./assets/Models/CoolModel/textures/awesome-sauce.jpg

The point of all of this is that a) maybe this is the start of something that will work, and b) you can see the AssetKey magic that makes rehoming assets possible during conversion.

I will use this today and read the code.

This is what this thread is about.

I had no idea you set all that stuff with asset key like that. I was just using newKey and passing it in.

If you do that then it will lose all of it’s subclass-specific information if it’s a TextureKey, ModelKey, etc… JME doesn’t provide an easy way to create a repathed key so I used reflection magic.

I have some thoughts on an overhaul of the asset manager. It only does one thing well and the rest kind of… err… less well. (One thing well = caching)

Worked on gltf box and duck from jme-testdata. Duck looks great. Then I tested it on something I made that has 7 pbr textures I created by hand and it had problems.

Shows texture missing when its there and had missing textures. This was exported with the gltf exporter for blender 2.79 so some of this may be from that. This particular model does not show these problems when using SDK to convert it though. However, your conversion blows it away for clarity and depth even without a probe for the textures that made it through.

Failed on ogre model. Think the missing ogg library played a role in that.

12:39:25,763 ←[1;32mINFO ←[m ←[36m[AssetReader]←[m ←[32mUsing source asset root:
assets\Textures
←[m12:39:25,771 ←[1;32mINFO ←[m ←[36m[AssetReader]←[m ←[32mFound assetConfig:jar
:file:/C:/Users/Robert/Documents/JMonkey/x/x/jmec-1.0.0-SNAPSH
OT/lib/jme3-core-3.2.3-stable.jar!/com/jme3/asset/Desktop.cfg
←[m12:39:25,836 ←[1;33mWARN ←[m ←[36m[AssetConfig]←[m ←[1;33mCannot find loader
com.jme3.audio.plugins.OGGLoader
←[m12:39:25,837 ←[1;32mINFO ←[m ←[36m[Convert]←[m ←[32mConvert:assets\Textures\O
to.mesh.xml
←[m12:39:25,837 ←[1;32mINFO ←[m ←[36m[AssetReader]←[m ←[32mloadModel(assets\Text
ures\Oto.mesh.xml)
←[m12:39:25,838 ←[1;32mINFO ←[m ←[36m[AssetReader]←[m ←[32mLoading asset:Oto.mes
h.xml
←[m12:39:26,331 ←[1;32mINFO ←[m ←[36m[Convert]←[m ←[32mLoaded:Oto-ogremesh (Node
)
←[m12:39:26,332 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mNode(Oto-ogremesh) key:
Oto.mesh.xml
←[m12:39:26,332 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m  Geometry(Oto-geom-1)
←[m12:39:26,333 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m      Material[name=Mat
erial.002/SOLID/TEX/bluewarrior1024.j/VertCol, def=Phong Lighting, tech=null]
←[m12:39:26,333 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Vector4 Diffuse
 : 1.0 1.0 1.0 1.0
←[m12:39:26,334 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Boolean UseMate
rialColors : true
←[m12:39:26,334 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Float ParallaxH
eight : 0.05
←[m12:39:26,334 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Vector4 Ambient
 : 0.2 0.2 0.2 1.0
←[m12:39:26,334 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Boolean Backfac
eShadows : false
←[m12:39:26,335 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Texture2D Diffu
seMap : WrapRepeat_S WrapRepeat_T "Oto.jpg"
←[m12:39:26,335 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Boolean UseVert
exColor : true
←[m12:39:26,336 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Vector4 Specula
r : 0.0 0.0 0.0 1.0
←[m12:39:26,336 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32m        Float Shininess
 : 0.25
←[m12:39:26,336 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mfindDependencies(Oto-og
remesh (Node))
←[m12:39:26,336 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mfindDependencies(Oto-ge
om-1 (Geometry))
←[m12:39:26,336 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mfindDependencies(Materi
al[name=Material.002/SOLID/TEX/bluewarrior1024.j/VertCol, def=Phong Lighting, te
ch=null])
←[m12:39:26,337 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Vector4 Diffus
e : 1.0 1.0 1.0 1.0
←[m12:39:26,337 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Boolean UseMat
erialColors : true
←[m12:39:26,337 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Float Parallax
Height : 0.05
←[m12:39:26,337 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Vector4 Ambien
t : 0.2 0.2 0.2 1.0
←[m12:39:26,338 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Boolean Backfa
ceShadows : false
←[m12:39:26,338 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Texture2D Diff
useMap : WrapRepeat_S WrapRepeat_T "Oto.jpg"
←[m12:39:26,338 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mmaterial asset:Texture2
D[name=Oto.jpg, image=Image[size=1024x1024, format=BGR8]]
←[m12:39:26,339 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Boolean UseVer
texColor : true
←[m12:39:26,339 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Vector4 Specul
ar : 0.0 0.0 0.0 1.0
←[m12:39:26,339 ←[1;32mINFO ←[m ←[36m[ModelInfo]←[m ←[32mChecking:Float Shinines
s : 0.25
←[m12:39:26,340 ←[1;32mINFO ←[m ←[36m[Convert]←[m ←[32mDependency[file=assets\Te
xtures\Oto.jpg (Mipmapped), key=Oto.jpg (Mipmapped)]
←[m12:39:26,341 ←[1;32mINFO ←[m ←[36m[AssetWriter]←[m ←[32mCopying:assets\Textur
es\Oto.jpg (Mipmapped) to:assets\Models\Oto\Oto.jpg (Mipmapped)
←[mException in thread "main" java.io.FileNotFoundException: assets\Textures\Oto
.jpg (Mipmapped) (The system cannot find the file specified)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(Unknown Source)
        at java.io.FileInputStream.<init>(Unknown Source)
        at com.google.common.io.Files$FileByteSource.openStream(Files.java:126)
        at com.google.common.io.Files$FileByteSource.openStream(Files.java:116)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:267)
        at com.google.common.io.Files.copy(Files.java:315)
        at com.simsilica.jmec.AssetWriter.write(AssetWriter.java:103)
        at com.simsilica.jmec.Convert.convert(Convert.java:162)
        at com.simsilica.jmec.Convert.main(Convert.java:197)

Didn’t do a blend model since that library is missing also.

My gltf is awaiting redo when blender 2.8 gets released but it did work with the SDK convert as I wrote.

Hang on about the gltf. I just reran it with sdk and it seems that this model may be a test that was having problems with export.

Edit: This was a test model, sorry about that. Probably screwed up beyond repair.

So the gltf all looked good.

Are you planning on doing blends and ogre?

Found another cool way to sync assets.

project.sync {
   from '../PathToProject/assets'
   into 'assets'
}

With this statement in gradle, everything works like the SDK. May be to much though if large project but liked how it does automate things with no effort.

If you convert a model in the external asset project, you don’t have to do anything special. Just write your add statement as normal in you current project.

rootNode.attachChild(assetManager.loadModel("Textures/Rocket.mesh.j3o"));

Sync will do everything before compile. All the task and buttons work like the SDK now where clean just cleans with the added benifit of synching, build will sync first and build so the distribution zip works. the run button, build and clean, and build will work as it should and you wont have to move any models first.

Adding runtime files() statement to dependencies will add the external files to the project but using sync to me feels cleaner and you do not have to change statements around between build types. Sweep in the files you want, the runtimeOnly project(’:assets’) dependency will take care of the rest using excludes.

If anyone finds a problem with this let me know, I am new to this gradle stuff, just been a few days now using it proper.

Should clarify, sync will copy everything everytime from the “from” directory before it removes anything not copied into the “into” directory. I am not sure how this will affect large projects. With today’s computers it might not be a problem.

Still, I like how this gives a gradle project the same capabilities as sdk. Can convert the models in sdk, move things around and they will work or are working for me just like SDK.

If Paul does his project, it would make this moot if it were a gradle dependency but until then this may fill in the gap.

Blend and ogre should already be there. “OGG” is for music, not ogre. The jme3-blender library is in the distribution with its dependencies.

I’ve updated the release of JmeConvert with some changes.

  1. It now writes build version and memory info to the console when running.
  2. I configured max memory for the startup script… set to 4 gig. (I was trying to load an fbx that kept running of of heap but I think it’s a JME FBX loader bug.)
  3. Fixed an issue with shared textures where only the last one was having its key reset.

So far, I can load and convert every gltf file that I’ve tried from Sketchfab.

I’ve tried downloading the original versions of some of them but I haven’t found one that works yet for obj or fbx. I don’t know if the SDK would have loaded them or not as I don’t have that installed anywhere.

Maybe I should create a new thread for the converter since it’s starting to turn into something “Real”. I’m certainly having a lot of fun playing with it.

Edit: and to be honest, if it only ever works 100% fine with GLTF I’m kind of ok with that.

SDK does load and convert objs. Not sure about fbx.

I’m sure you’re aware @pspeed but for the others, too, obj is a bog standard format. I read some of Paul Burke’s paper on it a while ago. I don’t think the api has changed for it since Jesus woke up in a cave.

He describes the spec as good as any other paper I’ve come across.

http://paulbourke.net/dataformats/obj/

1 Like