[SOLVED] OutOfMemoryError when converting assets with JmeConvert

Hi,

I use JmeConver in my model-converter GUI app to convert my gltf models to j3o format and generate model icons.

After converting a bunch of models I am getting OutOfMemoryError. After looking deeper into memory usage I noticed the allocated direct memory is never getting freed. Looking deeper into the code I noticed it creates a new DesktopAssetManager every time I am calling convert.setSourceRoot() and for some reason, the allocated buffers from the old asset managers are never released.

After I modified the code to reuse the same AssetReader and just reset the root path instead of creating a new AssetReader then the issue is resolved and I see memory is properly freed.

this is my modified AssetReader class just in case:

(added setAssetRoot() and removeAssetRoot())

public class AssetReader {

    public static final String DESKTOP_ASSET_CONFIG = "/com/jme3/asset/Desktop.cfg";

    static Logger log = LoggerFactory.getLogger(AssetReader.class);

    private Path root;
    private final AssetManager assets;


    public AssetReader( ) {
        this((URL) null);
    }

    public AssetReader(URL assetConfig ) {
        if( assetConfig == null ) {
            assetConfig = getClass().getResource(DESKTOP_ASSET_CONFIG);
            log.info("Found assetConfig:" + assetConfig);
        }

        this.assets = new DesktopAssetManager(assetConfig);
    }

    public AssetReader(AssetManager assets) {
        this.assets = assets;
    }

    public void setAssetRoot(File assetRoot) {
        if (root != null) {
            assets.unregisterLocator(root.toString(), FileLocator.class);
        }

        try {
            this.root = assetRoot.getCanonicalFile().toPath();
        } catch( java.io.IOException e ) {
            throw new RuntimeException("Error getting canonical path for:" + assetRoot, e);
        }
        log.info("Using source asset root:" + root);

        assets.registerLocator(root.toString(), FileLocator.class);
    }

    public void removeAssetRoot() {
        if (root != null) {
            assets.unregisterLocator(root.toString(), FileLocator.class);
            root = null;
        }
    }

    public AssetManager getAssetManager() {
        return assets;
    }

    public Spatial loadModel( File f ) {
        if (root == null) {
            throw new RuntimeException("Asset root is not set.");
        }

        log.debug("loadModel(" + f + ")");
        if( !f.exists() ) {
            throw new IllegalArgumentException("Model file does not exist:" + f);
        }

        try {
            // Make sure the file is completely collapsed into canonical form.
            // Note: apparently getAbsoluteFile() is not good enough.
            f = f.getCanonicalFile();
        } catch( IOException e ) {
            throw new IllegalArgumentException("File cannot be converted to canonical path:" + f, e);
        }
        // Find the relative path
        String path = root.relativize(f.getAbsoluteFile().toPath()).toString();

        log.info("Loading asset:" + path);

        // Make sure the cache is clear
        assets.clearCache();

        // AssetManager doesn't really give us a better way to resolve types
        // so we'll make some assumptions... it helps that we control the
        // asset manager ourselves here.
        String extension = Files.getFileExtension(f.getName());
        if( "gltf".equalsIgnoreCase(extension) || "glb".equalsIgnoreCase(extension) ) {
            // We do special setup for GLTF
            return assets.loadModel(GltfExtrasLoader.createModelKey(path));
        } else {
            return assets.loadModel(path);
        }
    }
}

Any chance we have this change added to the main repo?

And here is an example case to test out of memory issue:

public class MemoryReleaseTest extends SimpleApplication {

    FunctionId F_SPAWN = new FunctionId("spawnModel");
    FunctionId F_DEBUG_MEMORY = new FunctionId("debugDirectMemory");
    FunctionId F_FORCE_GC = new FunctionId("forceGC");

    public MemoryReleaseTest() {
        super();
    }

    public static void main(String... args) {

        BufferUtils.setTrackDirectMemoryEnabled(true);

        MemoryReleaseTest main = new MemoryReleaseTest();
        AppSettings settings = new AppSettings(true);
        settings.setRenderer(AppSettings.LWJGL_OPENGL32);
        settings.setGammaCorrection(false);
        settings.setSamples(8);
        settings.setVSync(true);
        settings.setResolution(800, 600);
        main.setSettings(settings);
        main.setShowSettings(false);
        main.setDisplayStatView(true);
        main.start();
    }

    @Override
    public void simpleInitApp() {
        GuiGlobals.initialize(this);
        flyCam.setEnabled(false);

        System.out.println("BufferAllocator:" + System.getProperty(BufferAllocatorFactory.PROPERTY_BUFFER_ALLOCATOR_IMPLEMENTATION));

        InputMapper inputMapper = GuiGlobals.getInstance().getInputMapper();
        inputMapper.map(F_SPAWN, KeyInput.KEY_S);
        inputMapper.map(F_DEBUG_MEMORY, KeyInput.KEY_D);
        inputMapper.map(F_FORCE_GC, KeyInput.KEY_G);

        inputMapper.addDelegate(F_SPAWN, this, "spawnModel");
        inputMapper.addDelegate(F_DEBUG_MEMORY, this, "debugDirectMemory");
        inputMapper.addDelegate(F_FORCE_GC, this, "forceGC");

        initLight();
        viewPort.setBackgroundColor(ColorRGBA.Blue.clone());
    }

    private void initLight() {
        AmbientLight ambient = new AmbientLight(ColorRGBA.White.mult(10));
        DirectionalLight sun = new DirectionalLight(new Vector3f(0,0,-1), ColorRGBA.Yellow.mult(10));
        rootNode.addLight(ambient);
        rootNode.addLight(sun);
    }

    /*public void spawnModel() {
        System.out.println("Spawning model...");
        assetManager.loadModel(new ModelKey("Models/head/head.gltf"));
        assetManager.clearCache();
        debugDirectMemory();
    }*/

    Convert convert = new Convert();
    public void spawnModel() {
        convert.setSourceRoot(new File("."));

        System.out.println("Spawning model with jmec...");
        URL resource = getClass().getResource("/Models/head/head.gltf");
        String pathName = decodeUrl(resource);
        System.out.println("Loading model:" + pathName);
        convert.getAssetReader().loadModel(new File(pathName));
        convert.getAssetReader().getAssetManager().clearCache();
        debugDirectMemory();
    }

    private static String decodeUrl(URL url) {
        return URLDecoder.decode(url.getPath(), StandardCharsets.UTF_8);
    }

    public void debugDirectMemory() {
        BufferUtils.printCurrentDirectMemory(null);
    }

    public void forceGC() {
        System.out.println("Forcing garbage collection...");
        System.runFinalization();
        System.gc();
        debugDirectMemory();
    }
}

if I comment this line

convert.setSourceRoot(new File("."));

memory will be freed properly and no OutOfMemoryError will raise.

But now won’t you have a problem with any textures or materials (etc.) that happen to have the same name getting reused because of the cache?

Maybe we just need to free up the old DesktopAssetManager properly somehow but I create a new one on purpose to avoid accidental caching.

AssetReader already clears the cache before loading the model so there should be no problem.

1 Like

Still seems like there might be a bug in DesktopAssetManager if things aren’t getting freed.

…but I guess if there is a way to work around it then we don’t have to care.

1 Like

I submitted a PR and updated the release note. Will appreciate it if you can take a look and possibly merge.

I took a closer look but could not figure out what is the issue.

3 Likes

Made the requested changes (from the GitHub conversation).

2 Likes