@vear:
Yes, boundries are fixed by giving the more detailed blocks that touch a less detailed block a special indicis buffer. All those indicis buffers are pregenerated, but they're shared between all different blocks in the GPU's memory. (so you have about 16 per detail level), so for the entire terrain it's still only a few MB at most.
Currently triangle lists are used, mainly cause it was simple. Could be converted to triangle strips without too much pain though probably.
There is no vertex morphing. Escp. if you set the LOD to be more agressive as described in the readme, then you can definatly see some popping. Probably a good solution would be vertex shader to reduce somewhat. Note however, that the way the current terrain is generated (hill heightmap) under a lot of circumstances it seems to create plateaus (a tiny size bump, and then an all flat surface) or jagged terrain (very smallest hills). Exactly the two things that encourage noticeable popping. But in a way it's good to show the problem, and HillHeightMap pretty fast for generating, and it takes arbitrary sizes for the terrain.
I'll give you a little preview of the current system:
manager: this is where all heightmap data is stored and tracked, and contains the functions for loading/unloading heightmap data based on plugins for it. This is where heightmap data is read from (and written to) by other compents. Also allows you to track other objects you've loaded (such as meshes), based on the same coordinates as the map/block system. There's a reference implementation for a simple file based system (with zlib compression), an example of memorymapping, and the AbstractHeightMap based system used for this demo.
view:
takes a Camera and a manager as an argument, and is a system that helps plugins to load/unload data for maps/blocks based the camera position, and track what maps/blocks are currently loaded. There are currently plugins for loading/unloading map data (compressed or uncompressed), decompressing data, creating meshes for it, and my LOD system. The intention is you could use this for other things too, eg. write a plugin to dynamically generate some positions to place trees on. The "API" for this plugin system is very messy at the moment though.
mesh-data: takes block coordinates and a manager as arguments, to generate verticise and normal buffers to use with the shared texture co. and index buffers.
task system: Actually an independant system I wrote, for managing and scheduling task in different thread models. Also still in the early stages of development. Is used by the plugins for the view system to make sure the rendering can go on while other things are happening 
@Badmi:
Could you tell me what you mean exactly? Do you mean stuttering, or a general slow down, etc? Under which circumstances, the default test? If you used 0.0.2 did you turn on all optimizations except ditch? Specs?
Version 0.0.3 released, made some changes to the "ditch" optimizations, but they're off by default. Other optimizations are on again, but can be turned off (see README). If any of you nVidia users could try running with enable:ditch I'd appriciate it.
enable:ditch still crashes. What exactly is it doing, anyway? (Not that I'd probably understand too much of the answer, but always trying to learn 
)
Yep, crashes for me (NVIDIA Quadro4 980 XGL).
[...]
ditched vertex buffer(1386018648):1
ditched normal buffer(1386018648):33
created an id: 65
Found a cached id for indices: 17
org.lwjgl.opengl.OpenGLException: Invalid value (1281)
at org.lwjgl.opengl.Util.checkGLError(Util.java:56)
at org.lwjgl.opengl.Display.update(Display.java:569)
at com.jme.renderer.lwjgl.LWJGLRenderer.displayBackBuffer(LWJGLRenderer.
java:514)
at com.jme.app.BaseGame.start(BaseGame.java:80)
at SimpleTest.main(SimpleTest.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.simontuffs.onejar.Boot.run(Boot.java:247)
at com.simontuffs.onejar.Boot.main(Boot.java:105)
Great to see my terrain generation code in action. ;) Do you plan on releasing the code anytime soon?
llama said:
Currently triangle lists are used, mainly cause it was simple. Could be converted to triangle strips without too much pain though probably.
I've asked because i'm working on exactly that now, and believe me its a pain.
There is no vertex morphing. Escp. if you set the LOD to be more agressive as described in the readme, then you can definatly see some popping. Probably a good solution would be vertex shader to reduce somewhat. Note however, that the way the current terrain is generated (hill heightmap) under a lot of circumstances it seems to create plateaus (a tiny size bump, and then an all flat surface) or jagged terrain (very smallest hills). Exactly the two things that encourage noticeable popping. But in a way it's good to show the problem, and HillHeightMap pretty fast for generating, and it takes arbitrary sizes for the terrain.
Using a normal-map texture and calculating light in the fragment shader reduces popping somewhat too. This is because normals of removed vertices are still accessible trough the texture, and shades are still calculated correctly, altough no vertex is present. Also have a morphing shader which works, except it doesnt take into account fixing triangles on borders the proper way yet.
I'll give you a little preview of the current system:
manager: this is where all heightmap data is stored and tracked, and contains the functions for loading/unloading heightmap data based on plugins for it. This is where heightmap data is read from (and written to) by other compents. Also allows you to track other objects you've loaded (such as meshes), based on the same coordinates as the map/block system. There's a reference implementation for a simple file based system (with zlib compression), an example of memorymapping, and the AbstractHeightMap based system used for this demo.
view:
takes a Camera and a manager as an argument, and is a system that helps plugins to load/unload data for maps/blocks based the camera position, and track what maps/blocks are currently loaded. There are currently plugins for loading/unloading map data (compressed or uncompressed), decompressing data, creating meshes for it, and my LOD system. The intention is you could use this for other things too, eg. write a plugin to dynamically generate some positions to place trees on. The "API" for this plugin system is very messy at the moment though.
Sounds promising. ETA?
Llama,
This sounds a lot like Hugues Hoppe’s work with Geometry Clipmaps. He wrote a GPU version for the GPU Gems 2 book. Have you referenced it at all? It mentions a solution to control or mask the popping. His 80MB video is spectacular.
Here’s the link: http://research.microsoft.com/~hoppe/
I hope your work progresses equally impressive.
llama said:
@Badmi:
Could you tell me what you mean exactly? Do you mean stuttering, or a general slow down, etc? Under which circumstances, the default test? If you used 0.0.2 did you turn on all optimizations except ditch? Specs?
The movement just becomes jurky when it uses prosising power to load more.
1.9 Ghz CPU
ATI 9600
hevee said:
enable:ditch still crashes. What exactly is it doing, anyway? (Not that I'd probably understand too much of the answer, but always trying to learn ;) )
It destory the native buffer for vertices and normals after a VBO is created for them, so you use less system memory. I guess I'm deleting them too soon for nVidia drivers or they're still left as mapped somehow and Ati driver happen not to care. Anyway I think I'll leave it at that for now (off by default), a lot of people people will want to keep the vertexbuffer for collisions and such, and normals aren't always the optimal way of doing lighting for a terrain in the first place.
@digitalntburn
I haven't referenced anything really when I was building it, except the existing TerrainBlock/TerrainPage system from jME a bit. Near the end when most was done I started looking at other implementations a bit, and a lot of them had the same ideas I had.
Geometry clipmaps is quite a bit more advanced though. Though some of the things mentioned on that webpage aren't that far off from what I want to do eventually. But there's so many things I still want to do, I thought I better just release the code soon (though some people will their eyes when they see what state it is in), because I have way too little time for it.
@vear
A morphing shader was my first thought for making the popping less noticable. It would morph vertice buffer to look exactly like what it would look like once it has the new indexbuffer. Adjecent buffer should synchronize the morphing.
Other improvements can be made, like the ones you suggest, and most combine with changing the way geometry is created in the first place. I could spend years working on it, and I don't have years of time
Just llama's.
If you would like, I can set you up an FTP account so you can update it yourself? :-p
darkfrog
first of all : congratulations llama for such a fine implementation of geomipmapping
i've been waiting a long time to see this one.
tested here on an athlon xp 1700 whith a geforce fx 5200 : no problems except the movement slowdowns badmi reported too. usually in a game using such huge terrains you would define a maximum movement limit somewhere. that way you could determine at what speed the terrain should be loaded/generated. another idea would be to generate the mipmaps incrementally: terrain that's far away would only have low level mipmaps (no high detail information at all). that way you only load/generate what needs to be seen, and memory consumption would be reduced.
btw: the terrain normals look a little bit strange. did you use the same method terrainblock/terrainpage uses?
digitalntburn said:
This sounds a lot like Hugues Hoppe's work with Geometry Clipmaps. He wrote a GPU version for the GPU Gems 2 book.
AFAIK, GLSL doesnt support accessing textures in the vertex shader, so a geometric clipmap terrain engine as described in GPU Gems 2 cannot be made with GLSL. The relevant part had to be run in software. What Llama is about to give us is more like the engine described here:
http://www.gamedev.net/columns/hardcore/geomorph/
llama said:
A morphing shader was my first thought for making the popping less noticable. It would morph vertice buffer to look exactly like what it would look like once it has the new indexbuffer. Adjecent buffer should synchronize the morphing.
Other improvements can be made, like the ones you suggest, and most combine with changing the way geometry is created in the first place. I could spend years working on it, and I don't have years of time :D That's why I decided no more new features and redesigns, only very basic cleanups, which is why I hope the ETA will be very soon! I can't make any exact promises, every time I do that I get flooded with work :x
Those improvements already sit in my comp, eagerly waiting for a cool terrain system to be added to. ;)
I recall having asked you several times for the engine.
Great to see it in action now.
Just some observations:
For me the framerate-ranges are striking.
On my notebook (ati rage Mobility 9000) your solutions drops to 50 frames in the worst cases (sporadically when reloading while moving) and gives me some 120 in the best case. (normal average is around 100) (using the first java call in the readme in both cases). Disabling LOD gives me something in the 30-40 range !
I also notice some periodic speed ups / downs in movement especially when swapping.
In the meantime we were trying the same thing although we didnt use any optimizations so far, no LOD, no nothing, but ulimited terrain like yours using a mechanism to dynamically load new terrainblocks.
I opened up a new thread as not to "misuse" this one.
First: sorry it took so long When you work in your own buisness it's hard to control your working hours sometimes.
Well, there's a good reason for that. You see that ugly yellow texture on the terrain? And how it's repeated over and over? That's a 128x128 texture. So the application is fillrate limited, which is particalury bad on notebooks. If I turn off the textures, or even just decrease the repeat factor by 3 (1 texture per "mesh count"), my framerates are about the same as in your test.
Secondly, what of course matters is the triangle count. That's much higher in the default view of my application. If you take a view with a comparable triangle count, not surpringly the difference is much less. My application has (in the default view) somewhat more triangles, but a LOT more vertices (which you can call detail or view distance, whatever you prefer). Of course that detail is only visible when you're close because of LOD. If you turn of LOD and draw all those vertices then yeah, the application becomes triangle limited and… surprisingly when my applications draws 10 times more detail than yours my FPS is lower
In the end it's the same geometry after all, assuming you don't use tri-strips or fans yet. It won't be faster in one application than the other. We probably have some technical differences, but I doubt they'll matter much in a standard situation like this.
The biggest differences can be in loading. How does your application currently do loading? Heightmap based or serialized terrainblocks or something like that? How is the geometry created? Anyway, in testing your application I found that it sometimes has the same problem as mine, stuttering when loading the next block. My blocks are smaller than yours, which is less efficient for loading, but better for culling and LOD. Note that you can change the blocksize in my demo (look in the README). Also the movement speed in your demo is much lower.
The speedup/speeddown/stuttering problem seem to be the biggest problem in both application. I think the SimpleGame structure (using smoothed times for updating controllers) is actually part of the problem… The effect changes based on the FPS too (cause it uses a fixed number of frames to do the smoothing). After I release the source (I checked in the needed changes for jME last night) I'll focus on this problem.
Hi llama,
i suppose crypter will post later this day about the details of our mechanism. He deserves the credit for it.
So just some comments from me.
Of course in the end its clear that the more triangles and vertices you have the slower the fps rate will be. And so its no surprise that both solutions would normally do similarly with the same amount of data to display provided nobody wrote any code that keeps the cpu and gpu occupied with other things.
I think you do not really "load" the terrain but instead have it completely in memory ?
So it seems you optimized the process for big (single) maps ? Do you plan to implement a loading mechanism ?
Or do you use some serialization and swapping technique after initial creation of the terrain ?
We went the exact opposite way. Since our terrain is unlimited we tried to find a way of loading new tiles without slowing down the system. Aside of that we use nothing special, just a plain old Heightmap with no extra tricks. The data is stored in simple raw files with integer format and loading of the terrain and the textures is done by two threads (crypter correct me if i am wrong).
Interestingly when i turn LOD on the framerate drops dramatically in our solution. I guess thats related to the loading mechanism as it must not only load the terrain fully but also construct the different lod data. In your solution its the exact opposite. Of course you then display way more triangles still. Up to three times our numbers. And you have traded the performance when using LOD with the initialization time but i wonder what happens if you increase the size of the terrain so much that you must load new blocks.
I was wondering only since i think you use very many triangles and vertices compared to us. Do you really need them or is it a feature to have the possibility to use them ? The number of triangles and vertices surely allows for more detail in your terrains.
I guess we could learn a lot from your optimization experiences.
The speedup/speeddown/stuttering problem seem to be the biggest problem in both application.
Do you mean the frame rate or do you mean the "experience" that movement suddenly becomes faster and slower (the second is what i meant).Strange ... i didnt see that effect our solution yet, but perhaps thats related to our (generally) slower movement where the difference is not so obvious.
Todi said:
Interestingly when i turn LOD on the framerate drops dramatically in our solution.
I did not notice any LOD in your implementation. Could you please describe how to enable/disable it, and what to do to observe the LOD algorithm at work?
As I tried to explain in the opening post, I have a complete system for loading (and writing) to disk. Actually it is a pluggable system, so for this test I wrote another plugin, one that uses the classic AbstractHeightMap as a source. The plugin system for example means you can use a huge AbstractHeightMap as a source, save it do disk (compressed), and from then on use disk loading. With disk loading I've tested maps of up to 10.000x10.000, but one could go as far as one desires (like you I use separate files for blocks in the reference system (again I can't emphasize this enough, people can write whatever system they want, eg. dynamically generate everything)).
The only reason my application shows more triangles is, of course, because of the default settings (and the fact that your terrain only goes in one direction). You can change them to whatever you like for this demo, just look in the README. You can't set the speed of the camera however, because of the faster movement, there's a lot more slowdowns (cause more new terrain has to be loaded).
I do not use CLOD (which indeed slows down things, and has many other issues for this) but my own LOD implementation (which also has some problems but if anything it's fast).
I mean the second, it happens when I see in the console a new block is loaded. I can also hear the HDD.
Llama, this system looks great, combined with a sliding loading/unloading algorithm could allow for absolutely huge game worlds. Very exciting. The popping is more than acceptable in my opinion.
Sfera said:
Todi said:
Interestingly when i turn LOD on the framerate drops dramatically in our solution.
I did not notice any LOD in your implementation. Could you please describe how to enable/disable it, and what to do to observe the LOD algorithm at work?
We can change that only programatically at the moment.
Actually i didnt "plan" to release anything at the moment, but when llama came up with his terrain engine i wanted to compare it and since i had written that small test class i just thought i could publish it as well. You might have noticed that the code is a bit "bigger" than it should be for a small test like this. ;)
llama said:
First: sorry it took so long :) When you work in your own buisness it's hard to control your working hours sometimes.
No need to be sorry from my point of view. This gave me the opportunity to start working with JME and to experiment with dynamic terrains. :)
Well, there's a good reason for that. You see that ugly yellow texture on the terrain? And how it's repeated over and over? That's a 128x128 texture. So the application is fillrate limited, which is particalury bad on notebooks. If I turn off the textures, or even just decrease the repeat factor by 3 (1 texture per "mesh count"), my framerates are about the same as in your test.
Ok, that a good explanation :) More detailed textures are part of my schedule, but at the moment that's in the distant fog. We will see how this changes our performance.
Secondly, what of course matters is the triangle count. That's much higher in the default view of my application. If you take a view with a comparable triangle count, not surpringly the difference is much less. My application has (in the default view) somewhat more triangles, but a LOT more vertices (which you can call detail or view distance, whatever you prefer). Of course that detail is only visible when you're close because of LOD. If you turn of LOD and draw all those vertices then yeah, the application becomes triangle limited and.. surprisingly when my applications draws 10 times more detail than yours my FPS is lower ;)
In the end it's the same geometry after all, assuming you don't use tri-strips or fans yet. It won't be faster in one application than the other. We probably have some technical differences, but I doubt they'll matter much in a standard situation like this.
True. I am still not sure how much details for a terrain will be needed in the end. For the the smooth hills we are using at the moment it seems we have enough. Nevertheless with more objects coming later, I'd love to improve performance right away. This leads to some questions, which you possibly can answer. But I will start with my answers.
The biggest differences can be in loading. How does your application currently do loading? Heightmap based or serialized terrainblocks or something like that? How is the geometry created? Anyway, in testing your application I found that it sometimes has the same problem as mine, stuttering when loading the next block. My blocks are smaller than yours, which is less efficient for loading, but better for culling and LOD. Note that you can change the blocksize in my demo (look in the README). Also the movement speed in your demo is much lower.
We are using TerrainBlocks. The height maps are saved as int-arrays as used in the constructor for TerrainBlock. For the loading I used a asynchronous queue, which is processed by a background thread (so in fact this cannot lead to any stuttering). But surprisingly I learened that ljwgl is not thread safe and run into problems. As those seem to occur during texture loading / applying only, I wrote another synchonous queue, which is processed in the update() method. This sometimes makes the application stutter, but I haven't seen it lately. Still there is room for improvement, e.g. separating loading of textures from applying them. In the end there should be only fast operations in the synchronous queue. Which means no stutter from loading at all.
We haven't experimented with the movement speed yet. No idea how fast we could go before the background thread starts to glow.
LOD is an interesting theme. I tried to use LOD by setting the clod-flag in the TerrainBlock. The results were:
- creation of on TerrainBlock lasten 20 secs !!! => Now the background thread was gasping
- Even in a fully loaded terrain I saw no improvement of the performance :? You seem to be deep in those details. Any idea what is wrong?
The speedup/speeddown/stuttering problem seem to be the biggest problem in both application. I think the SimpleGame structure (using smoothed times for updating controllers) is actually part of the problem.. The effect changes based on the FPS too (cause it uses a fixed number of frames to do the smoothing). After I release the source (I checked in the needed changes for jME last night) I'll focus on this problem.
:? Now you got me. I have no clue, what you are talking about???
Anyway I'll be off for one week of skiing. Afterwards I'll be checking for news.
Thanks for the insights.
PS: Sorry, this post took me some hours (with major interruptions). So it might seem out of sync in some points.