Source for Terrain engine for large maps (0.0.5)

User Code & Projects
1

I’ve managed to put together the sources for the terrain engine I release a demo for in this thread:

http://www.jmonkeyengine.com/jmeforum/index.php?topic=2979



The source can be downloaded here:

jME Terra



The latest source is in SVN over at Google Code:

see the development thread:

http://www.jmonkeyengine.com/jmeforum/index.php?topic=5170.0





It’s under the GPL for now, when it becomes more stable it’ll become LGPL or BSD (like jME).

As of 0.0.5 the license is now LGPL.



I normally wouldn’t have released this without more polishing, but I don’t have always have enough time to work on it, so I thought it’d better release it. Because of this there are still some silly bugs, most noteably:


  • do not attempt to change the scale of the terrain (except the build-in heightscale as used in the tests).
  • do not attempt to add a TerraView anywhere other than at (0,0,0).
  • lack of a solution for terrain following.
  • lack of a decent logging structure.
  • lack of documentation.
  • occasional stuttering during loading.



    General Package structure:



    org.llama.jmex.terra : everything terrain related.



    org.llama.jmex.task: another uncompleted project of mine this project uses, a framework for executing tasks in jME to do background loading and such.



    org.llama.test: The tests.



    For now I'll just discuss the tests, later I'll add a section to this thread about the general structure of the code, and one on how to import some real world height data (there are converters for 2 formats included in the source) to make really big maps with ease :slight_smile:

First test is ViewJMEXTest. This is basically the demo as seen in the topic mentioned above. So see that topic for the command line properties you can use. Textures are now off by default and can be enabled with enable:textures, and there is also enable:sample2 sample4 sample6 and sample8 for multisampling the scene. For the rest some default values changed a bit from the 0.0.3 test, and height above the terrain is now counted for calculating LOD distances.

The second test is SaveJMEXTest. This has the same command line parameters as ViewJMEXTest, but is meant to create map files for loading your world from disk. It's a console app that will ask you to enter some info in the console.

First is the file name, files will be saved in the current directory (if you launch from Eclipse this is your workspace dir) but you can gave a relative or absolute path in from of the file name. eg. "mymap" or "res/hillmap/myhillmap" or "/bla/blamap"

Next the "Number of blocks to fit in a map". A world consists of several square maps (each map is 1 file) overlapped by several blocks (the pieces of the geometry). I'll explain this better later on, but a good value would be 2 or 3. (The block size is specified through the command line parameters as explained, by default it is 128, so 2 would make a separate file for each 256x256 block).
If everything goes well your map will now be compressed and written to disk.

With ViewMapFileTest you can open this or other maps you've saved. It takes roughly the same command line parameters as ViewJMEXTest with some differences:

[enable:novbo:textures:etc. (see demo readme or thread)]

filename (will prompt you on the console if not given)

block size (typically be the same as when you saved, will prompt if not given)

Number of blocks to fit in a map (block size * this value MUST be the size of a map file, so typically the same as when you saved. however you could eg. change blocksize from 128 to 64 and this value from 2 to 4, will prompt if not given)

memorymapping 0 (default) to disable, 1 to enable. (Memory mapping does not load a file, but instead maps it to memory, potentially reducing disk usage)

fogstart

fogend (both like in the demo, for where fog begins and ends. Fogend is also the drawing distance)

render distance (distance at which geometry is created)

decompression distance (NOT in the demo! distance at which heightmap data is decompressed)

loading distance (distance at which compressed heigthmap data is loaded from disk/memorymapped)

lod levels (like in the demo, several values at which the next LOD level will begin)

So I hope this will give those who want to the chance to play around with this work a bit. If anyone wants to port code over from TerrainPage to TerraManager so people can do terrain following it'd be most welcome. Other things too of course. The code will still change a lot in my local copy, but if anyone contributes something I'll do my best to work it into my code. On my own TODO right now:

- fix the "bugs" listed above (espc. the stuttering)
- introduce gradual loading: right now maps and geometry are always loaded at full detail, Even if for the current LOD this is not really needed. By reordering way data is stored in the map files and geometry buffers (first low detail, then added higher detail) loading can be done in stages as you get closer.
- pluggable system for creating geometry.
- refactor some of the weirder parts of the source (like TerraView)
- optimize LOD.

and in the longer term:

- dynamic texture splat maps.
- normalmaps or lightmaps instead of normals.
- morphing.

And finally, in TerraData.java there is a method sferaNormals() that makes better normals than the default ones. In line 109 change buildNormals() to sferaNormals() to enable this. It looks better but has a problem with seams (the line between two block), that the default method does not have. As you can guess this method was contributed by Sfera :)
2

Converting heightmap data files



For converting heightmap data to the reference map format for jME Terra, I've written two rudimentary importers. One for the ArcInfo ASCII format, and one for the HDR format. These can be found in the source as AscTerraBuilder and HdrTerraBuilder.



Both can be used the same way, either as a console app or by command line parameters:

filename, the name of the file to be imported (without the extention, make sure the file has a lower case extention on *nix). This name will be used as the basis for saving the data too.

blocksize: size of a block. If you want to use LOD make sure this value is divideable by two enough times to accommodate the LOD levels (see post above, and the demo thread)

map multi: number of blocks that can fit in a map (see above posts)

prefix path in: the path at which the input file can be found (can be empty, program will use the current dir, if you launch from Eclipse this is the workspace dir)

prefix path out: the path at which the output will be written "



These importers are not perfect, if a map does not fit (because part of it is outside the edge of the data provided) it will simply be dropped. There can also be other bugs, I did not study any of the specs for the fileformats, I simply reverse engineered them. Both importers raise landmasses slightly to they are distinguisable from areas with no data (oceans).



For HDR make sure you place the .bin and .hdr file in the same directory, and that you use 16 bit data (big endien/unix/mac). HDR loads the whole file into memory first, ASC only one row. Make sure to use the VM switches for allocating enough memory if needed.



The final method of importing heightmap data is through the jme AbstractMap class (image maps, raw files, etc), see the SaveJMEXMap.



Finding real world heightmap data



There's several sources for this. What they differ in most is resolution. Data with a higher resolution tends to give smoother maps.



You can make your own HDR map with a reasonable resolution (about 30 arc seconds, one point is ~ 1KM) on this site: http://www.ngdc.noaa.gov/cgi-bin/seg/ff/nph-newform.pl/seg/topo/customdatacd (make sure you set the file format to Unix/Mac)



This site has ArcInfo data at a much higher resolution (3 arc seconds): http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp (make sure to download ArcInfo ASCII data).



Finally there is this site (unfortunately did not work well in my Opera or IE, but firefox worked well): http://seamless.usgs.gov/

It has lot's of different data (not just heightmap data either) and allows you to select your own area. There's the same 3 arc second data for the world, but the highest resolution can be found for US data. The entire US is covered with 1 arc second data (SRTM), but there's partial coverage for 1/3 arc seconds (NED 1/3) and even some 1/9 arc second data (NED 1/9). That's about 4 meters per heightpoint I think :)  You can clearly see roads and railroads at this resolution



You can let the map show what the coverage area for these different data types is by checking the data type under "layer extend". Even though 1/9 covers only some small areas, it's still a lot of data (the biggest area covered is west Virgina it seems, still 100s of gigabytes I'd reckon)



The catch? I have not written an importer for this data format (yet?). But with FWTools http://fwtools.maptools.org/ you can convert many different data formats to ArcInfo ASCII. This is mostly done with the gdal_translate command line tool in the bin/ directory of this package. ( gdal_translate -of AAIGrid source.adf dest.asc ).



I might upload a map I imported + test somewhere if I can find the webspace.


3

the first things i did was to add the terrain at coordinates different from (0, 0, 0) and to change the scale. setting the terrain to coordiantes different from (0, 0, 0) seemed to have no effect. the scale seems to work (no bugs). to my disappointment there was no crash or exception :stuck_out_tongue:

well, don't worry i'll keep trying until i find a way to crash it :slight_smile:



the LOD seems to work better and the framerate is very decent for the amount of triangles drawn. on a second thought, maybe there are too many triangles drawn whith the default settings you provided. maybe if i play arround whith the settings i can make the LOD more agressive.

a thing i noticed is that whan starting, the terrain behind the camera isn't loaded. only after you move the camera it loads. maybe you assume the camera is at a different position when starting.

4

Ah yes scale works…

5

Version 0.0.5 is released, to work with the latest jME changes (after .10). Some other changes:


  • full 32-bit range of height values now supported.
  • reconstructing parts of the map is now easier (change to VBO handeling).
  • some other small changes/fixes.
  • license changed to LGPL.



    Oh, and you can also see this package in action at the E3. don’t ask me which stand though XD But it’s there… somewhere…
6

any chance you'll implament a octree renderer or something of the sort to cull terrain that can't be seen.  I guess the bsp could do it but it's not as effective right?  Just wondering,  also what program exports 32 bit height maps? 

7

usually there are not that many blocks showing at the same time so octree isnt needed(in my opinion)



freeworld 3d exports 32-bit heightmaps for example, and some more but i don't have their names in my head at the moment…

8

Culling takes only a miniscule amount of CPU time in most cases. I'm not gonna waste any time on that myself…

9

the only problem i currently see with the terrain is that stuttering when generating the blocks. i'll have to take a closer look at the source, but an idea would be to generate only the lod levels(index buffers) which are really needed, when they are needed(if not already available).

is that a bad idea?

10

Uhm, index buffers are all pre-generated. Only vertex and normal buffers are generated on the fly.



There are several possible causes for the stuttering, and I hope to tackle most in the next version.

11

Out of curiosity… Any reason why this is lgpl instead of something more compatible with jME?

12

Well, not to get all license technical… but you can use LGPL together with jME (of course).



Anything I'll be doing myself with this project will most likely be "free software", so the GPL license suits that well. But right away some people wanted to use it in commercial or closed source projects, so I changed it to LGPL. If there are several other people who will contribute to this project, and they think another license is more appropriate (for example so it can be included in jME) it can always be changed again (or have a dual license).

13

Hi llama,



i've done a look to your engine and thinking of some additions/changes/features the last days.



I have two main thinks:


  • Loading of pre modeled terrain tiles/blocks/how ever you call it - so you can save a generated terrain tile/block load it in a 3d tool an model overhngs etc.
  • and static meshes (houses, rocks etc.) - so they are un-/loaded if the terrain tile/block is un-/loaded



    I take a deeper look in the implementation but can't see how to achieve this.



    The TerrainMesh is a TriMesh so no childs can attached.

    The map stores simple store the heigh points, right?



    My main problem is the fileformat and loading. The file is accessed with a xy-coordinate, then i have to know if the terrain tile/block/map consists of height points or is pre modeled. I think somethink like a index file is needed.



    What do you think about this?



    Regards MadLion
14

One

15

Thanks for the answer llama.



I think i didn't understand all (and the code). And i'm not sure we can achieve our needs with it.



For now we are trying something different.

16

getting some of these when running ViewJMEXTest, test runs fine though otherwise :slight_smile:

updated LOD:XYKey: 5:13:td: 2
java.io.IOException: Outside of range
   at org.llama.jmex.terra.format.JMEXMapStore.loadMap(JMEXMapStore.java:62)
   at org.llama.jmex.terra.HeightMapManager.loadMapFromStore(HeightMapManager.java:230)
   at org.llama.jmex.terra.HeightMapManager.loadMap(HeightMapManager.java:222)
   at org.llama.jmex.terra.pass.DistanceLoadingPass$1.runTask(DistanceLoadingPass.java:82)
   at org.llama.jmex.task.Task.runTaskFromRunner(Task.java:116)
   at org.llama.jmex.task.TaskRunner.runTask(TaskRunner.java:55)
   at org.llama.jmex.task.AbstractThreadTaskRunner.runTask(AbstractThreadTaskRunner.java:165)
   at org.llama.jmex.task.ContThreadTaskRunner.runThread(ContThreadTaskRunner.java:42)
   at org.llama.jmex.task.AbstractThreadTaskRunner$TaskThread.run(AbstractThreadTaskRunner.java:101)
Set LOD level XYKey: 3:13:td: 3

17

that's because the camera is near the edge of the terrain and the program tries to load terrain blocks beyond the edge of the terrain. it's actually not an error.

18

Wow, this is fantastic! I dl’ed apart of England from http://srtm.csi.cgiar.org/SELECTION/inputCoord.asp, as an ASC file, S_36_02.asc. Strolled around a bit. I did notice some massive spikes  around the ocean. not sure if it is in the original data or in the conversion (I used your ASC convert app). Perhaps it’s a wrap around if part of the data is below sea level?






19

Never looked at ASC but a military project I worked on used DTED, which if there were undefined areas it used -999 as elevation. Perhaps ASC uses a very large number for undefined areas. I would guess this as most elevation data has holes in the oceans.

20

That's in the data set, afaik the cgiar site does keep improving it's data set so you could keep an eye on it when they'll release updates. You could also try the http://seamless.usgs.gov site, though you'll have to use FWTools to convert the data first. (the benifit is you can select the exact area you want).



You could also use interpolation, the spikes all have the same value (it's actually defined in the ASC file what that value is, if I remember correctly).