Loading a Material Definition from a string

Progressing with abstracting the resource system from the file system, I got to the point where I need to load a material definition (something that is normally stored in a “.j3md” file) from a String. I have made a research on how the loading is performed and I get stuck at the point where DesktopAssetManager.loadAsset() finds the asset already being in cache because prior to that I loaded it from the text string as a text asset in order to get it into the system at all and it finds it in cache and assumes that it is a Material Definition and fails to typecast from String to MaterialDef…

I am totally lost here, can someone please give a brief example of a correct locator and a correct loader to load a MaterialDef from a string?

I assume the same thing will go for shaders then, but I think I’ll be able to sort it out when this is done and load them from a string as well…

GitHub - Pesegato/GoldMonkey: XML Builder for dataset can load strings from a JSON… in your case you would have:

matdefs.json
shaders.json

…and so on

Heres the Locator and Loader interfaces as well as the needed AssetInfo class:

@Pesegato Thanks for the reference, I find there is an elaborate spec reader, but it is not directly related to JME? Can’t find any mention of a MaterialDef there… ?

@normen Yes, that I get. I have also created an AssetLocator which, if the passed AssetKeys name starts with a "+string__" signature, cuts the signature off and creates an AssetInfo with the .openStream() that returns the stream built from the rest of the string that contains the body of the material definition. By design it should just read it instead of the file. However, that does not work. For some reason the InputStream never serves as an input stream for reading and creating a MaterialDef.

The debugger shows that when the Material() constructor gets to the point of getting the instance of the MaterialDef for itself, the MaterialDef is never read from the string. Instead, the AssetInfo that contains this string is found and the string itself is attempted to be cast into a MaterialDef (see the Material constructor – it tries to do that).

I have no idea how do I manage to feed the string into the MaterialDef loader and make it instantiate a MaterialDef there… ?

With GoldMonkey you read data from strings. When you have the string, you can instance your own MatDef or whatever.

That is exactly what is unclear to me – how do you instance a MaterialDef from a string? I can’t find any easy way in JME.

Well, I guess at this point you’ll have to post some code where you setup your custom asset locator (and literally nothing else) that returns a stream for a String_ location.

AssetInfo info = new AssetInfo(assetManager, new AssetKey(whatever)){

publick InputStream openStream(){ return new ByteArrayInputStream(yourMaterialDefString.getBytes()) ;}。
};

new J3Mloader().load(info);

load it directly with J3MLoader.

1 Like

Wow, @yan was faster than me! :smiley:

Meanwhile I was rewriting my code to Java to present here as an example of what am I doing. And while I was rewriting the code, I had an idea and made it to work. But it feels very awkward. The strangeness is that I have to add a fake ".j3md" “extension” to the data string in order for the asset to be created in the way so that it can be picked by the J3MLoader, and then the actual data string has to be stripped off of it when creating the InputStream from it. All in all this feels… umm… inorganic. So, while I did manage to complete my task while doing this rewrite, I still ask for a better option – how do I achieve this in a more simple and correct way?

I like what @yan suggests – he cuts right to the matter :smiley: but there is my concern – what happens when I avoid loading through the JMEs asset pipeline? Will I be able to release/dispose of the resources properly then? Or is this concern a false one and there’s nothing to worry about?

So, there are the 2 small classes that do the job in my example (NOTE: the material definition body is loaded from a file with an absolute location in this example. This is just to abstract the data string generator so that it can be replaced with anything else. Do not think about it to much since in reality this is probably going to be a database query):

file 1:

package matdeftest;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

import com.jme3.app.SimpleApplication;
import com.jme3.material.Material;
import com.jme3.scene.Geometry;
import com.jme3.scene.shape.Box;

public class MatDefTest extends SimpleApplication {

	public static MatDefTest instance;
	
	Material m;
	
	public static void main(String[] args) {
		instance = new MatDefTest();
		instance.start();
	}
	
	public String stringResource(String path) {
		String content = "";
		try {
			content = new Scanner(new File(path)).useDelimiter("\\Z").next();
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		return "string://" + content + ".j3md";
	}
	
	@Override
	public void simpleInitApp() {
		
		String a = "string://Common/MatDefs/Misc/Unshaded.j3md";
		String path = "AbsolutePathToYour.j3md";
		
		assetManager.registerLocator("string://", StringAssetLocator.class);
		assetManager.registerLoader(StringMaterialLoader.class, "string.j3md");
		
		Box box = new Box(2, 2, 2);
		Geometry g = new Geometry("box", box);
		m = new Material(assetManager, stringResource(path));
		g.setMaterial(m);
		g.setLocalTranslation(0, 0, -10);
		rootNode.attachChild(g);
		
	}
	
	public void simpleUpdate(float tpf) {
		m.setFloat("Time", getTimer().getTimeInSeconds());
	}

}  

file 2:

package matdeftest;

import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;

import com.jme3.asset.AssetInfo;
import com.jme3.asset.AssetKey;
import com.jme3.asset.AssetLocator;
import com.jme3.asset.AssetManager;

public class StringAssetLocator implements AssetLocator {

	public static String prefix = "string://";
	
	@Override
	public void setRootPath(String rootPath) {
		// TODO Auto-generated method stub
		
	}
	
	@Override
	public AssetInfo locate(AssetManager manager, AssetKey key) {
		if(key.getName().startsWith(prefix)) {
			return new AssetInfo(MatDefTest.instance.getAssetManager(), key) {

				@Override
				public InputStream openStream() {
					String data = key.getName().substring(prefix.length());
					data = data.substring(0, data.length() - 5);
					System.out.println("DATA = " + data);
					return new ByteArrayInputStream(data.getBytes(StandardCharsets.UTF_8));
				}
				
			};
		}
		return null;
	}

}

From a high level, what is is that you are actually trying to do?

Want to abstract from the filesystem. I want to specify data sources with the schema. For example "string://" or "http://" or "db://" or whatever else :slight_smile:

But why?

Edit: and of those, string:// the way you are doing it is the really weird one because you are actually passing the content instead of some locator for the content. It’s an extremely strange thing to want to do.

Because:

  1. I need to deploy independent installations like VJing nodes, interactive (kinect-enabled) and other stuff across a number of distributed machines, and it is required to have a central repository of assets that can be accessed like a gallery, without a need to download some content file.
  2. Generative assets – an application can request a non-existing asset, based on some sequence of numbers or instructions and an AI shall generate that asset.
  3. The "string://" thing is just an example because effectively the transport should be abstracted, and "string://" is the simplest way to do it to display an example. It should be read as “a generic data source”.

I am just trying to untie the asset system from the filesystem. There is no reason that the asset management facility has to incorporate a hard-coded notion that resources are loaded from a file on disk.

…and you just exactly described AssetLocators.

For example, a clearer example for your StringLocator would have been to give it a hashmap of values to which the string://foo.j3md is just the key.

myStringLocator.put(“foo.j3md”, theMaterialDefString);

new Material(assetManager, “string://foo.j3md”);

…and in any case, you sound like you are essentially recreating Java’s support for URL connections (but it’s a pain to use anyway for simple stuff).

Do you mean that the locate() method could just fetch the strings from the hashmap in the example of this string asset locator?

Well, as for the Java URL - yes, it is there… but I don’t get how do I make it work with material definitions, because when I put a fully-qualified URL into a j3md file to specify shaders, the parser does not accept it and fails to load the definition. That is why I am trying to develop this. BTW, this is why I was using the awkward "+string__" prefix initially, instead of "string://" – because the latter one does not get accepted by the parser of the j3md files when I write it for the shader filepaths. I had to come up with a string that does not have, as I guess, the : and / characters.

Or am I missing something and the j3md parser can actually work with the fully-qualified URL schemas? (the simplest I tried were "string://" and "file://")

Yes. Fetch the string from the hashmap, wrap it in an InputStream.

It’s probably true that material definitions cannot use URLs, I guess. I’ve never tried it. JME wants to abstract things one step further so you don’t have to use absolute paths like that… so probably strings/ would be enough since you just need a root of some kind.

Partially, it’s trying to keep you from doing non-portable things… partly, I think the parsers for things like materials are just as simple as they need to be and didn’t take it into consideration. I wonder what happens if you try to quote the strings in the material def.

Yeah, no support for the standard URL syntax, I’ve checked. Doublequoting does not help either. Hmmm I can’t really agree that this is a good choice and that it favors abstraction since the standard URLs are the most abstract thing I can imagine right now. What we actually get is that we actually must use absolute paths… in the sense that they have absolutely only 1 option tied to the transport/medium – reflect the actual relative filesystem folder structure of the project space.

But I think that was not actually a choice, but that was just, as you said – the most simple implementation without the consideration. Still, still, I can’t believe that abandoning things like standard URL identifiers is a good thing… you find yourself literally cut off from the wide variety of transport, cut off from the world. It even contradicts the Java philosophy a bit. Probably what we do have now is there for historical reasons, like someone made it, it was working for everyone, and they just left it like that.

Looks like I’ll have to go with my funny reimplementation of the standard URL mechanism right now :slight_smile:

It’s an abstraction that favors bundling your game. The game loads “generic relative resource” and your application setup code maps that to some physical URL or other resource.

So the way it’s done does make sense for most games.

The other approach encourages games that only run in one place… on the developer’s computer.

Mhm, it was a conscious choice and its whats done in most games - assets need absolute paths or you’ll end up with things like 10 different texture.jpg files covering each other. Its just like java classes actually, everything is defined by a full path, name and type. And with locators you can implement any source you want, just like big games do with their pak’s or dat’s or whatever. It doesn’t have anything to do with file systems actually - apart from the fact that both asset systems and file systems need unique identifiers for single resources.

The issue with using URLs is that they can cause portability issues. For example, if you have a model referencing a URL like file://C:/MyGame/texture.jpg then you wouldn’t be able to import this model on any other machine. jME3 gives you a virtual file system abstraction so you can refer to assets using a project-relative path.

Regarding the two use cases:

Have an HTTP server host the assets then you can use jME3’s UrlLocator to locate those assets. E.g.:

assetManager.registerLocator("http://yourvjserver.com/assets/", UrlLocator.class);

Now assets will get loaded from the server.

[quote=“noncom, post:13, topic:37405, full:true”]
2) Generative assets – an application can request a non-existing asset, based on some sequence of numbers or instructions and an AI shall generate that asset.[/quote]
It’s possible to extend AssetKey and then implement an AssetLocator that will generate an asset based on the given parameters. This will automatically leverage the built-in caching, cloning, etc features of the asset system. Alternatively, you can just generate the asset on the spot, as that might be easier.