Help on OOM issue loading terrains from file

Hi,
Hate to ask but I’ve been hacking, searching and reading source code for days now and I cannot figure out whats wrong.
The error I’m getting is this:

Exception in thread "Thread-5" java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) at com.jme3.util.BufferUtils.createFloatBuffer(BufferUtils.java:831) at com.jme3.terrain.geomipmap.LODGeomap.writeTexCoordArray(LODGeomap.java:122) at com.jme3.terrain.geomipmap.LODGeomap.createMesh(LODGeomap.java:89) at com.jme3.terrain.geomipmap.LODGeomap.createMesh(LODGeomap.java:84) at com.jme3.terrain.geomipmap.TerrainPatch.read(TerrainPatch.java:931) at com.jme3.export.binary.BinaryImporter.readObject(BinaryImporter.java:340) at com.jme3.export.binary.BinaryInputCapsule.resolveIDs(BinaryInputCapsule.java:483) at com.jme3.export.binary.BinaryInputCapsule.readSavableArray(BinaryInputCapsule.java:471) at com.jme3.export.binary.BinaryInputCapsule.readSavableArrayList(BinaryInputCapsule.java:587) at com.jme3.scene.Node.read(Node.java:597) at com.jme3.terrain.geomipmap.TerrainQuad.read(TerrainQuad.java:1743) at com.jme3.export.binary.BinaryImporter.readObject(BinaryImporter.java:340) at com.jme3.export.binary.BinaryInputCapsule.resolveIDs(BinaryInputCapsule.java:483) at com.jme3.export.binary.BinaryInputCapsule.readSavableArray(BinaryInputCapsule.java:471) at com.jme3.export.binary.BinaryInputCapsule.readSavableArrayList(BinaryInputCapsule.java:587) at com.jme3.scene.Node.read(Node.java:597) at com.jme3.terrain.geomipmap.TerrainQuad.read(TerrainQuad.java:1743) at com.jme3.export.binary.BinaryImporter.readObject(BinaryImporter.java:340) at com.jme3.export.binary.BinaryImporter.load(BinaryImporter.java:242) at com.jme3.export.binary.BinaryImporter.load(BinaryImporter.java:125) at com.jme3.export.binary.BinaryImporter.load(BinaryImporter.java:109) at com.jme3.asset.DesktopAssetManager.loadAsset(DesktopAssetManager.java:282) at com.jme3.asset.DesktopAssetManager.loadModel(DesktopAssetManager.java:368) at com.jme3.asset.DesktopAssetManager.loadModel(DesktopAssetManager.java:372) at TerrainTiler.TerrainTiler.LoadTile(TerrainTiler.java:210) at TerrainTiler.TerrainTiler.access$900(TerrainTiler.java:45) at TerrainTiler.TerrainTiler$terrainThread.run(TerrainTiler.java:653)

The code is my TerrainTiler project that I’m trying to put together as a replacement to TerrainGrid. The class file is here TerrainTiler Project
Lint 207 has my loader routine, you can see by the commented out section that I’ve used two tests, the active one is using the assetloader to load in a saved TerrainQuad from file (generated and saved by the GenerateTiles function just prior to this one) and the commented out section creates a similar tile from scratch instead of loading.

Now the class essentially tracks the camera and loads new tiles in and removes the old tiles out to follow the camera.
The error appears after about 5 or six blocks have been loaded, I can see from the Profiler and memory usage that the new quads are loaded but is seems something is holding references to the quads I’ve removed so they don’t get GC’d…
I’m only learning so maybe this is my code’s issue but after banging at this for days I cannot see where PLUS when I change the routine as above to create tiles instead of loading them then I get no errors!! memory usage goes up and down as it should.

I know Sploreg is the expert on terrains and that he is away atm so I’m happy to wait, unless someone else can shed some light on this.

Any advice much appreciated.
Thanks

What memory settings are you using for heap and direct memory? Defaults?

Direct memory does not cause GC to be run. So if your heap is so big that GC never runs then your direct buffers will never get garbage collected. No matter how much direct memory is hanging around waiting to be freed, it will never trigger GC on its own.

The best way is to free the direct buffers manually when you are done with them. There is a method on BufferUtils for doing this.

It also helps to use reasonable memory settings, ie: huge direct mem size but a reasonable heap size. For example, Mythruna uses 1024m direct mem but only 512 meg heap. When my heap was 1024m the app would pause a lot and run out of direct memory all the time.

Default settings, reads to be about 256Mb for Direct Memory, not actually sure atm on the heap size.
When running the Create code instead of the Load code I see the Direct Memory usage go up until just before the limit and then drop down again, so its working, but when loading the tiles the memory usage never really drops, just rises every time the tiles are loaded.
Makes me believe the internal loaders are leaving a reference but don’t know where.
I’ve made several passes and dumps from Profiler but just can;t understand it enough to track it more precisely.
Last time I had this issue I did increase the buffer but that only meant it took longer to fill it, didn’t fix the issue itself.
I’d like to free the buffers myself but since I’m not creating them (the loader is) I don’t know how to do it, is there a way?
Thanks.

If it helps here is a dump of the output when I do a [M] to dump the memory usage… each dump is after a pair of tiles is loaded - only 6 on scene at any time,
1st = 4 tiles, 2nd = 6 Tiles, 3rd = 6 tiles (2 new, 2 unloaded), etc
Until the OOM…

Existing buffers: 564 (b: 25 f: 442 i: 9 s: 88 d: 0) Total heap memory held: 15398kb Total direct memory held: 75285kb (b: 553kb f: 70564kb i: 0kb s: 4167kb d: 0kb)

Existing buffers: 743
(b: 19 f: 601 i: 9 s: 114 d: 0)
Total heap memory held: 17398kb
Total direct memory held: 108619kb
(b: 551kb f: 101820kb i: 0kb s: 6246kb d: 0kb)

Existing buffers: 1010
(b: 20 f: 834 i: 9 s: 147 d: 0)
Total heap memory held: 21769kb
Total direct memory held: 153673kb
(b: 552kb f: 144792kb i: 0kb s: 8328kb d: 0kb)

Existing buffers: 1124
(b: 19 f: 918 i: 9 s: 178 d: 0)
Total heap memory held: 21662kb
Total direct memory held: 170439kb
(b: 551kb f: 159479kb i: 0kb s: 10408kb d: 0kb)

Existing buffers: 1369
(b: 19 f: 1131 i: 9 s: 210 d: 0)
Total heap memory held: 21289kb
Total direct memory held: 211978kb
(b: 551kb f: 198936kb i: 0kb s: 12489kb d: 0kb)

Existing buffers: 1461
(b: 19 f: 1191 i: 9 s: 242 d: 0)
Total heap memory held: 27520kb
Total direct memory held: 223679kb
(b: 551kb f: 208557kb i: 0kb s: 14570kb d: 0kb)

Existing buffers: 1643
(b: 19 f: 1341 i: 9 s: 274 d: 0)
Total heap memory held: 35856kb
Total direct memory held: 252932kb
(b: 551kb f: 235728kb i: 0kb s: 16652kb d: 0kb)

Exception in thread “Thread-5” java.lang.OutOfMemoryError: Direct buffer memory

Thanks

Oh… are you using asset manager to load the tiles?

Yeah, it will cache them unless you clear the cache. I can’t remember if there is a way to load them without caching in general.

Oh, I tried bypassing AssetManager and used BinaryImporter directly but had exactly the same result.
I’ll look into the caching and see if I can force it to clear.
Still, an unused cache should clear itself if there is a memory shortage imo :wink:
Hopefully that cache is easy to find and use and clear … IF its the issue
Thanks

Any tips on best way to clear said assetmanager cache? I think I’ve used up all my Google-Fu…
Sorry :slight_smile:

If you’ve bypassed the asset manager then it’s something else. You’ll have to use the memory profiler to see what is still holding the references.

Hrm, ok, I’ll have another go at the Profiler then.
When I followed the howto on the Netbeans site the nearest GC Root I found was “Listener (Java Frame)” type “LwjglDisplay”
Not that I have much of a clue on the profiler, I’ll keep prodding it to see if I can work out something.
Thanks.

omfg. Persistence pays and I’m a persistent bugger. Advice: Never give up searching, reading, analyzing or just plain trying different things until you get it.

Looks like you may be right there pspeed, looks like it may indeed be a caching issue with assetmanager.

I re-wrote the loader again with a direct BinaryImporter method and after banging my head again until I found I had to tell it the AssetManager (still) it appears to now be working as it should. I can now wander all over my terrain (currently 16x16 tile grid) and the memory usage hovers around 200Mb or less.

The new loader code is this:

[java]
private TerrainQuad LoadTile(int tileX, int tileZ) {
try {
String tileName = String.format(tileFilename+"%03d%03d", tileX, tileZ);
System.out.println("Loading Tile: "+tileName);

        // this is for loading the generated tiles from file
        BinaryImporter bImp = BinaryImporter.getInstance();
        bImp.setAssetManager(app.getAssetManager());
        File file = new File("assets/"+tileName+".j3o");
        TerrainQuad tq = (TerrainQuad) bImp.load(file);
        
        //TerrainQuad tq = (TerrainQuad) app.getAssetManager().loadModel(tileName+".j3o");

        tq.setLocalScale(tileScale);
        float ts = tileSize * tileScale;
        float to = ts / 2;
        tq.setLocalTranslation(new Vector3f(tileX * ts + to, 0f, tileZ * ts + to));
        //TerrainLodControl control = new TerrainLodControl(tq, camera);
        //tq.addControl(control);
        return tq;
    } catch (IOException ex) {
        Logger.getLogger(TerrainTiler.class.getName()).log(Level.SEVERE, null, ex);
    }
    return null;
}

[/java]

I still think this shows a potential issue with assetManager and I guess I don’t understand the difference in the proper or direct method but this gets me working again and maybe it may warrant further investigation on the issue? Up to the devs to decide that. My issue is (for now) fixed.

Now I can put the LOD stuff back in and then on to real texture splats… until it breaks again…

Thanks again for listening :wink:

Short lived joy… re-enabled TerrainLOD and its leaking memory again…
Should be able to trace this one though I hope, if not I’ll ask for help again :wink:

Thanks

Alright I’m back from holidays.
Took a quick look and there might be a memory leak that is triggered by the LOD control but located in the TerrainPatch. Each patch remembers its neighbour for fast referencing when calculating LOD, and this only caches if the LOD routines get called.
I think it hasn’t reared its ugly head before because TerrainGrid fully detached and removed terrain tiles all the time, while other implementations using the NeighbourFinder terrain API (for tiling) might not completely detach the terrains from the world.
I will add in a method tomorrow or this evening to clear this cache, or automatically do it when the terrain is set to have no parent node.

1 Like

Yay! Very happy to hear this, I’m very keen to keep at this code to get it working well enough to share with everybody.
So far my main issues has been with LOD stuff so I know I need to look deeper in to how it all works.

@Sploreg: One question I do have is should I attach the LOD Control at generation time before exporting the terrainQuad to file or as I’m doing now is attaching the control when it’s imported into the run-time? at the moment I’m creating plain quads with no LOD control attached and attaching it when loaded. Just not sure if this is confusing the loaders that assetmanager is using as I can see on the debug trace that the loader is still firing off GeoMap LOD routines as its loading.

Currently my code plan is to create a plain quad and material splats and save to a file only once at creation time (ie developing a game mode)
during game Play time the code will load the quad, attach it to the MultiTerrainLOD control, attach to the node. When move away the quad is detached from the LOD control then detached from the Node and removed from the list. Cycle this over and over as the player/camera moves over the map.

I’ll keep at it and watch for news on these changes. Any advice or questions to do with it I’m all ears.

Thanks!

Alright @radanz I committed a method to TerrainQuad that will clear any terrain patch references. It does this automatically when the terrain is removed from its parent node. So give it a try and see it if helps resolve the issue. The method can be called manually too: terrainQuad.clearCaches()
You can add the LOD control any time you want, the only thing you have to remember is that if you save the terrain with the control, you will have to grab the control again and set the camera on it when you load the terrain into the map. The control’s reference to the camera does not persist.
You do not have to persist the MultiTerrainLODControl. It just needs to be on the parent node that has all your terrain tiles. Each individual TerrainQuad does not need its own LOD control.

Awesome!
Just got up and am absorbing the first coffee for the day and I’ll get onto it.

Thanks on clearing up my doubts on the attaching of the LOD Control, seems my hunch was correct. Never thought of attaching the MultiTerrainLOD to the parent tho, thought I just had to attach each quad. I’ll have a play with that.

I’ll have a look at your code changes, I still need to learn a bit more on the mechanics of all this… I learn by building code to understand the api, as opposed to learning the api then building on what’s learnt. A bit odd maybe but only way I do it.

Will keep all posted on any major breakthroughs… or failures :slight_smile:

Thanks again.

(edit) Of course I’ll need to wait for the nightly to run I guess… :stuck_out_tongue:

Ok, got the nightly.
Beat the crap out of the code for a few hours as I was still getting memory leaks with the LOD.
Finally seem to have a sequence of unloading tiles that clears the memory and NO LEAKS!
not sure if it’s optimal but hey it works.
This is the current unload code:
[java]
} else if (tileSet.get(key).State == 4) {
// flagged for removal
terrainMLOD.removeTerrain(tileSet.get(key).Quad);
tileSet.get(key).Quad.setNeighbourFinder(null);
//tileSet.get(key).Quad.clearCaches();
tileSet.get(key).Quad.detachAllChildren();
parent.detachChild(tileSet.get(key).Quad);
//tileSet.get(key).Quad.resetCachedNeighbours();
tileSet.remove(key);
once = false;
}
[/java]

The LOD Detail still isn’t right though, getting normal errors and gaps as seen in this pic:

I’ll keep hacking away at it, gotta be something I’m doing wrong.
At least the leak is fixed for now…

Committed and pushed the code up if anyone wants to look at it and point out the blaring NOOB mistakes I’m probably making :wink:

-till next time

@Sploreg
sorry to prod again, but reading through the code for MultiTerrainLODControl.java I see:
[java] /**
* Add a terrain that will no longer have its LOD handled by this control.
* It will be removed next update run. You should only call this from
* the render thread.
*/
public void removeTerrain(TerrainQuad tq) {
removedTerrains.remove(tq);
}
[/java]
… and can’t help but think that it should be “removedTerrains.add(tq);” instead, so that in the prepareTerrain function it actually uses this list to remove the terrains from the terrains list?
Sorry if I’m wrong but had to ask…
Thanks.

Nope, you are perfectly correct!
I committed a fix for that just now, thanks @radanz

Oh wow! I wasn’t just going crazy :smiley:
One more bugfix toward release day :wink:
Tho I’m surprised I wasn’t getting a leak from that. Would’ve slowed it down tho

Now I need to figure out exactly what direction the neighborFinder calls need to point to…

Also I’m a little worried about a comment in the code for TerrainTestTile.java that says the order in which the terrains are added to MultiTerrrainLODControl seems to matter. Since the way the paging system works there is no real way to control that…

Still I’m up now with fresh coffee and mind ready to have another go…
Thanks.

Ok fixed the LOD not sewing up the terrain edges with MultiTerrainLodControl…
Just a BLARING NOOB error as I said above…

The neighborFinder calls my class was implementing were using the reference center’s x & Y instead of x & Z… oops… oh well found and fixed now.

Thanks for listening to me waffle :slight_smile: