Terrain engine for large maps (demo 0.0.3 released, source available)

For a while I've been occasionally working on a terrain engine for on the fly loading very large (practically infinite) maps. I've even referred to this a few time on this forum. Unfortunatly I could not spend much time on it, and I felt the urge to rewrite some parts, which is now basically done.



UPDATE: version 0.0.3 released based on feedback about the nVidia crashes. Thanks guys!

UPDATE: Source now available: http://www.jmonkeyengine.com/jmeforum/index.php?topic=3098



I do not have the space to host very large files for the maps, so I decided to make a quick demo that uses the com.jmex.terrain system for generating the heightmaps during runtime. The full release also contains code for reading and writing (compressed) files, and some example utilities for converting some existing heightmap data formats into the example format. Just to be clear, this demo is still a pretty good demonstration of what the engine can do.



Features:

  • automated on the fly loading/unloading of heightmap data and generating geometry from it, based on the camera position
  • pluggable system for loading/saving and optional (de)compressing of heightmap data
  • a fast LOD system using indicis buffers (more or less geomipmapping based). (press T to see it in action)
  • low memory footprint: geometry data is reused where possible (indicis buffers, texture coordinates) and stored on the videocard where possible (using VBOs)
  • uses NIO Buffers instead of java arrays, allowing for cool tricks like memorymapping to a file and direct NIO loading.
  • customizable threading model for performing almost all of these things.



    Now the bad:

    Don't exactly expect a workable engine yet, almost none of functions that TerrainPage has are implemented yet (though they should be easy to port). Some pieces are still rather "hacked together" right now too, though in the end it works pretty for me. I'll release the source soonish with some of those quirks still in, but I have to check in some changes for jME itself first for this code to work. The demo will work fine though (it includes it's own custom copy of jME)



    Now, the link:



    jME Terra



    And a copy of the README:



    [pre]Author and copyright (2005-2006):

    Tijl Houtbeckers

    xmpp: llama@mordax.com



    Also see: www.jmonkeyengine.com/jmeforum/



    ----



    Usage:



    After extracting all files to the same directory, just open the JAR archive with:

    java -jar terra.jar

    Or on some platforms just "double clicking" the jar will also work.



    There are some commandline options too, all are simple int values seperated by spaces. For the default value use -1



    (default value listed)

    mapsize: total size of the map, 2449

    iterations: number of iterations for the HillHeightMap, 60000

    hill size: max. size of a hill, 75

    blocksize: size of the geometry blocks created, 80

    fogstart, fogend: begin and end of the fog, 100, 350

    rendering distance: distance at which geometry blocks will be created and removed, 500

    loading distance: distance at which map blocks are loaded and unloaded, 600



    lod: a series of parameters, that set the distance at which the next lod level begins. -1 will disable lod completly.

    default:

Llama, drop me a message, my wonderous server is online and I'd be happy to host the demo if you'd like?



darkfrog

This is great! Ran really well on my dev box.

Can't wait to get my hands on this  :smiley:

Thanks darkfrog :slight_smile:

I'd be thankful enough if you can host even this small file for me though (just drop me a link of the new URL).



When I release the source, I'll include a little guide on how to convert jmex.terrain maps, and as I mentioned the converters for some existing formats, and people can use planet earth as their map if they want to.



It's really just a demo this. It even has the just-a-demo-horribly-yellow texture.



Oh, and I forgot to mention in my post. The LOD system currently calculates distances based on where you are on the x,z plane of the terrain, it disregards how high above you are. This is done on purpose so you can see how the LOD system works by flying above the terrain. As long as you stay close to the terrain, it should give a pretty good idea of what it would look like. (press T to really see how it works)

doesn't work on my machine )=



INFO: Created Heightmap using the Hill Algorithm

14.03.2006 23:18:28 com.jme.scene.Node <init>

INFO: Node created.

14.03.2006 23:18:28 com.jme.scene.Node attachChild

INFO: Child (my world) attached to this node (rootNode)

14.03.2006 23:18:29 com.jme.scene.Node attachChild

INFO: Child (My Terrain:40:-120) attached to this node (my world)

14.03.2006 23:18:29 com.jme.scene.Node attachChild

INFO: Child (My Terrain:-40:-120) attached to this node (my world)

org.lwjgl.opengl.OpenGLException: Invalid value (1281)

at org.lwjgl.opengl.Util.checkGLError(Util.java:56)

at org.lwjgl.opengl.Display.update(Display.java:569)

at com.jme.renderer.lwjgl.LWJGLRenderer.displayBackBuffer(LWJGLRenderer.java:514)

at com.jme.app.BaseGame.start(BaseGame.java:80)

at SimpleTest.main(SimpleTest.java:204)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at com.simontuffs.onejar.Boot.run(Boot.java:247)

at com.simontuffs.onejar.Boot.main(Boot.java:105)

14.03.2006 23:18:29 com.jme.app.BaseSimpleGame cleanup

INFO: Cleaning up resources.

14.03.2006 23:18:29 com.jme.app.BaseGame start

INFO: Application ending.





windows xp, geforce ti 4800 - 128 mb

Sadly, it crashes on my machine, before I am able to see anything:


INFO: Child (My Terrain:-40:-120) attached to this node (my world)
build Vertices: 0
 Loading took 0 XYKey: 0:0:m
build Normals: 0
Terrain Block took:0 -1:0
created an id: 65
Found a cached id for indices: 17
ditched vertex buffer (-1707183644): 65
ditched normal buffer(-1707183644):81
org.lwjgl.opengl.OpenGLException: Invalid value (1281)
        at org.lwjgl.opengl.Util.checkGLError(Util.java:56)
        at org.lwjgl.opengl.Display.update(Display.java:569)
        at com.jme.renderer.lwjgl.LWJGLRenderer.displayBackBuffer(LWJGLRenderer.java:514)
        at com.jme.app.BaseGame.start(BaseGame.java:80)
        at SimpleTest.main(SimpleTest.java:204)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.simontuffs.onejar.Boot.run(Boot.java:247)
        at com.simontuffs.onejar.Boot.main(Boot.java:105)
14.03.2006 23:06:05 com.jme.app.BaseSimpleGame cleanup
INFO: Cleaning up resources.
14.03.2006 23:06:05 com.jme.app.BaseGame start
INFO: Application ending.



System specs:
P4M@2,6 GHz, 512 MB DDR RAM, GeForce FxGo 5250 (my NB manufacturer, Gericom, won't release any current drivers, though, and the Omega drivers give me jerky texture behaviour, so that's possibly the issue).
Oh, and WinXP Pro SP2, I'll try Debian linux in a minute.

It crashed before displaying anything I assume?

Hm I'm running on an Ati card here. What about you azathoth? And darkfrog, did you try it yet? I'll try and make an updated version, probably tommorow (it's getting late here).


It crashed before displaying anything I assume?


Yes. Same on linux, btw. What a pity, I really need something like this (though we have the same thing almost done for a project of mine, but looking at your code couldn't hurt at all, I suppose...)
EDIT: I used to have that problem some time, but can't recall now what it was / how I solved it. When you get that source code published, I might be able to spot the problem, If you can't properly reproduce it on your ATI card. I am really looking forward to see this thing working!

Working on my Nvidia box now

Llama,



Here ya go:



http://www.captiveimagination.com/download/terra_0_0_1.zip



darkfrog

Ati here. lemme check on my nvidia box

hevee said:

It crashed before displaying anything I assume?


Yes. Same on linux, btw. What a pity, I really need something like this (though we have the same thing almost done for a project of mine, but looking at your code couldn't hurt at all, I suppose...)
EDIT: I used to have that problem some time, but can't recall now what it was / how I solved it. When you get that source code published, I might be able to spot the problem, If you can't properly reproduce it on your ATI card. I am really looking forward to see this thing working!


Well, what this stacktrace shows is just that an OpenGL call gave an error, somewhere in rendering the frame. It's probably because I added some new OpenGL functions..
Who am I kidding, I won't be able to sleep like this

It crashes on my nvidia box

Ok, thanks all of you…



I put up a version 0.0.2 that has some optimizations turned off by default. I hope this will at least enable it to work. If it does, you can help me out by re-enabling the optimizations. Starting with VBO and then the rest. Copied from the updated README:



[pre]

This is a test version, in which optimizations are disabled.

To re-enable them use an extra command line parameter: enable.

followed by the following keywords (you can seperate them with

an extra character, but not spaces)



vbo (use VBOs)

indicis (use VBO for indices)

delete (delete unused VBO buffers)

ditch (ditch vertices and normal buffers unused because of VBO)

lock (lock geometry)



example use with a small map with small hills:

java -jar terra.jar enable:vbo:ditch:lock 1000 10

[/pre]

Well that's a relief, can try enabling some of the optimizations? Starting with vbo, and then the rest



I have a feeling vbo + indicis is the problem…

java -jar terra.jar enable:vbo:indicis:delete:lock

works,

java -jar terra.jar enable:vbo:indicis:delete:lock:ditch

crashes.
No other parameters used.

Thanks hevee… (and azathoth, though I didn't see the edit at first!). That gives me an idea what it could be. Now I can sleep tonight :slight_smile:

The program seems to slow down when it loads more.

Great stuff. And the speed is good too. Cant wait for the release of sources.  I have some questions:

It seems that you are fixing boundaries of detailed blocks against less detailed, which is good thing. You do it by making more versions of indices?

Do you draw blocks as strips or triangle lists?

Is there vertex morphing?



I also have a suggestion for introducing a "Terrain" interface. By referring to this interface instead of TerrainBlock, it would be possible to plug together different managing classes (eg: TerrainPage) and actual drawing classes (eg: TerrainBlock). Here is what a basic Terrain interface might look like:



public interface Terrain {
    // terrain types
    // TerrainBlock classical terrain block
    public static final int BLOCK = 1;
    // TerrainBlock terrain block using clod
    public static final int CLOD = 2;
    // TerrainPage terrain page with fixed quadtree
    public static final int PAGE = 4;
   
    public float getHeightFromWorld(Vector3f loc);

    float getHeight(float newX, float newZ);

    Vector3f getSurfaceNormal(float newX, float newZ, Vector3f store);
   
    /*
     * Returns the type of the terrain system item
     */
    int getTerrainType();

    void multHeightMapValue(int col, int row, int toMult);

    int getQuadrant();

    void setDetailTexture(int unit, int repeat);

    void updateFromHeightMap();

    void setQuadrant(int i);

    int getSize();

    void addHeightMapValue(int col, int row, int toAdd);

    void setHeightMapValue(int col, int row, int newVal);
}