I was thinking of all the changes I made to my code recently to see why the memory usage went through the roof again and on a whim I thought of one thing: The String I give when creating the terrainQuads to save to file was a bit longer than before ( was: “Tile-xxxxzzzz.j30” changed to: “DIRxxzz/TILExxzz.j3o”) and when running the NB Profiler I saw lots of CHAR[]'s filling up and going crazy. So I thought to myself, lets try reducing the size to something minimal (eg “xxxxzzzz”) and holy crap now I can walk back and forth across the test map and not go OOM?!!!
dAssetManager.clearCache(); still sends it all to hell but I’m using dAssetManager.clearCache(); again and its not going out, direct memory usage slowly climbs to about 250MB (current limit back at 256MB) and then a very slight pause and it drops to about 50mb… climb and repeat.
So maybe something in the way the terrains are loaded or handled does not like long strings for the name? Dunno, got me confused…
Dunno, I’m going to test it some more: I have two tile sets now, one with the short name and one with the old long name and I’ll go back and forth with tests and double check.
I swear each day brings different behaviors…
I guess the strings should be kept short anyways and I wondered if the slash “/” was upsetting it too…
I do this crazy stuff, blame me
As a side note the code can now use different tile sets just by changing the root dir and can now handle (theoretically) 16 million tiles… Multiple Massive Worlds!!! … now if I can make it actually work…
Well its still not totally consistent enough for me to say 100% but:
Loading the tiles with the long name I can get to OOM fairly quickly and easily if I push the speed up even a little (go slow and it struggles at the limit but seems to GC eventually)
Loading the tiles with the short name I cannot get to OOM no matter how hard I try, it gc’s usually well before the limits and a lot quicker (tho I’m going to push the speed up to ludicrous levels to test it)
So in this I am noticing that it does seem to struggle/pause/delay getting the cache/memory cleared when the limits are reached and I think that’s whats giving me OOM is that it cannot clear it quick enough or soon enough before the next load of tiles are pushed through… BUT shorter names do seem to help…
Still freaky but food for thought.
ps: and .clearCache() still throws a hissy fit and sends it all to hell before 6 tiles get loaded…
Back to more testing for me… just thought I’d throw this in…
TerrainQuad doesn’t save the patch size, it is only needed when you create the terrain from scratch. Loading it from a j3o already has the patches generated. I can add it in however.
I’m not sure how changing the material on the terrain would change how it is cached or saved. And the file name length causing OOM I am unsure about too.
Here is a rundown on what terrain saves:
It saves the heightfield. That is, just the height values. And it saves the total size, quadrant info, etc. It does not save the actual mesh. The mesh would be massive and take forever to load/save.
On loading the terrain, the heightfield is taken and the mesh recalculated (normals, binormals, UV, etc.)
You can look at TerrainPatch.write()/read() for the details.
Beyond that, terrain saving/loading is fairly standard with every other jme savable.
Sorry for the slow reply too, I am getting zero email notifications now. Forum is fukt =/
Yeah, many things not making sense atm, the inconsistency is making me insane…
The code I had in the video and before and after was working fine, so I re-enabled the post water filter, ran it, frame rate dropped to about 70 from 170 with the filter, watched the memory climb and drop, climb and drop BUT was only dropping small amounts like 250-200 instead of 250-50, eventually OOM. :’(
So I commented out the filter again, back to the code before, seemed ok then same thing, only small drops, eventually oom…
restarted the sdk, oom… clean & build, close sdk, ran from dist folder, oom…
at that point was past midnight got the shits and went to bed… I think I went oom a few times in my sleep…
Tried again this morning, fresh start on the pc, looked good at first, then started to build and oom…
off to work now, play again tonight, going to re-do parts of my code and am considering scrapping it and starting again…
I still have doubts on the thread-safety but I’m very grey in that area… If the devs assure me the asset loaders and the terrain mesh generation are thread safe and are ok with me calling them from my own thread then it must be my code and I’ll keep trying…
Sorry for my lack of knowledge, I’m trying to understand all this but as I said the inconsistency is crazy…
@radanz said:
I think I went oom a few times in my sleep...
lol, happens to us all =)
The asset manager is supposed to be threadsafe. It’s not all that complicated of a class, so don’t be threatened to look at the code. I can’t see how it will OOM however if you constantly clear it.
Instead of scrapping the project and starting again, can you just svn (or git or whatever) revert back to the point where you made the video?
Sadly no commit to roll back to, tho the code is the same anyways… so not sure… see what it wants to give after work tonight.
What I haven’t been able to solve and is consistant is the .clearCache() function causes the tiles to load VERY slow and sends the memory usage through the roof…
only when using terrainLighting with assets, using showNormals is fine… ???
Still no progress, but here’s a few more observations:
Seems when the rendering is more busy, like when I have the Post Water filter enabled, it seems to struggle a lot more to clear the memory. ie I go OOM far quicker with water on than I do without it…
Also very occasionally I get hit with this error, doubt its related but anyways thought it may interest someone:
Mar 10, 2013 9:12:13 PM com.jme3.terrain.geomipmap.TerrainLodControl updateQuadLODs
SEVERE: null
java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: 0
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at com.jme3.terrain.geomipmap.TerrainLodControl.updateQuadLODs(TerrainLodControl.java:223)
at com.jme3.terrain.geomipmap.TerrainLodControl.updateLOD(TerrainLodControl.java:166)
at com.jme3.terrain.geomipmap.TerrainLodControl.controlUpdate(TerrainLodControl.java:155)
at com.jme3.scene.control.AbstractControl.update(AbstractControl.java:112)
at com.jme3.scene.Spatial.runControlUpdate(Spatial.java:570)
at com.jme3.scene.Spatial.updateLogicalState(Spatial.java:688)
at com.jme3.scene.Node.updateLogicalState(Node.java:145)
at com.jme3.scene.Node.updateLogicalState(Node.java:152)
at com.jme3.app.SimpleApplication.update(SimpleApplication.java:244)
at com.jme3.system.lwjgl.LwjglAbstractDisplay.runLoop(LwjglAbstractDisplay.java:151)
at com.jme3.system.lwjgl.LwjglDisplay.runLoop(LwjglDisplay.java:185)
at com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:228)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at com.jme3.util.SafeArrayList.get(SafeArrayList.java:258)
at com.jme3.terrain.geomipmap.TerrainQuad.calculateLod(TerrainQuad.java:373)
at com.jme3.terrain.geomipmap.MultiTerrainLodControl$UpdateMultiLOD.call(MultiTerrainLodControl.java:130)
at com.jme3.terrain.geomipmap.MultiTerrainLodControl$UpdateMultiLOD.call(MultiTerrainLodControl.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
Currently I’m just passing over and over the code and cleaning/optimizing it hoping I fix it that way…
@zarch said:
Clearly not a very good Java programmer ;)
Dunno but he’s an Australian who recently spent 7 years in Japan as the lead programmer for a large firm developing and maintaining the software they use for tracking and handling rental and leased properties for tenants, landlords, agents, utilities and insurance companies…
Cleaned up the code a little, generated a new set of tiles, size 129 tQuads with patch size of 33 instead of 65 previously, ie 16 patches instead of 4.
Loop runs fine, memory doing exactly what I expect it to…
Add the post water back in using this code:
[java]
lightDir = dirl.getDirection();
fpp = new FilterPostProcessor(assetManager);
water = new WaterFilter(rootNode, lightDir);
water.setWaterHeight(initialWaterHeight);
water.setDeepWaterColor(ColorRGBA.Cyan);
water.setReflectionMapSize(256);
fpp.addFilter(water);
viewPort.addProcessor(fpp);
[/java]
12/03/2013 11:05:18 AM com.jme3.app.Application handleError
SEVERE: Uncaught exception thrown in Thread[LWJGL Renderer Thread,5,main]
java.lang.NullPointerException
at com.jme3.water.WaterFilter.setReflectionMapSize(WaterFilter.java:856)
at mygame.Main.simpleInitApp(Main.java:121)
at com.jme3.app.SimpleApplication.initialize(SimpleApplication.java:226)
at com.jme3.system.lwjgl.LwjglAbstractDisplay.initInThread(LwjglAbstractDisplay.java:130)
at com.jme3.system.lwjgl.LwjglAbstractDisplay.run(LwjglAbstractDisplay.java:207)
at java.lang.Thread.run(Thread.java:662)
Line 121 is the water.setReflectionMapSize(256);
I comment out this one line and all works fine, even the memory. however I get no reflections of the terrain in the water.
Tried different values and same.
Looked back on my previous code where I used the value of 128 for the setReflectionMapSize and on most views the reflection was actually wrong, looked like I was seeing the reflection upside down or flipped left/right…
Theories? Maybe the terrains aren’t re-generating right so the reflection maps are confused… maybe the water filter has a dislike for me… not sure…
For now it looks like it will work without reflection so I’ll continue on until it changes again…
Sorry. Reflections are there and only look flipped when I move the camera with code (ie cam.setLocation(new Vector3f(x, py, z)))
Moving around manually and the reflections are like they should be.
Guess I’ll need to learn Cinematics next
water.setReflectionMapSize(256); Still throws NPE though… going to try some variances and some older generated tiles… gotta be a link there…
GAH, I guess no-one read my code or understood my code
Just discovered I was attaching all the tiles to the ROOTNODE instead of my own TerrainTiler node…
–
I was attaching with this: parent.attachChild(tileSet.get(key));
but from the terrainState’s update loop which is an appState the parent was actually the rootNode…
So now I attach with this: TerrainTiler.this.attachChild(tileSet.get(key));
Now the tiles get attached to my node…
Surprised anything worked before…
Not sure this fixes anything either
Oh well… back to it…
Update:
After a lot of playing and generating many sets of tiles I’ve come to see that there are restrictions on either or both terrainQuads and TerrainLighting:
If I generate tiles with only 4 patches per tile I get memory issues
– ( tested with new TerrainQuad(tqName, 128 + 1, 64 + 1, hMap); ) <= this is what I was using for most of the above frustrations…
If I generate tiles with 16 patches per tile I get no memory issues
– ( tested with new TerrainQuad(tqName, 256 + 1, 64 + 1, hMap); AND new TerrainQuad(tqName, 128 + 1, 32 + 1, hMap) <= today's trials
also:
If I generate terrainLighting material where a texture when scaled is larger than a patch then it does not repeat correctly they get cropped to each patch.
– (probably this is by design, did make me panic though…) see the effect here: http://thegame.radans.net/wp-content/uploads/TerrainTiler-3.png
So right now I’ve got tiles of TerrainQuad(tqName, 256 + 1, 64 + 1, hMap); with terrainLighting material that I scaled the textures right down so they are smaller than the patches and now they look right AND I don’t seem to be having any memory issues… (textures look correct scale when I’m 1.8m off the ground - human viewpoint - but look a little too tiled at a distance… guess I need better textures to fix that)
FINALLY progress… couple more tests then I’ll commit and push to my repo…
They definitely require multiple GCs to happen before VM realizes its dumb 
