Really nice work!
Works quite nice even on my computer, which really isn’t fast :facepalm:
Just one little issue: The physics didn’t work for me until I changed the line 63 in NoiseBasedWorld to
[java]
tq.addControl(new RigidBodyControl(new HeightfieldCollisionShape(heightmap), 0));
[/java]
[java]
tq.addControl(new RigidBodyControl(new HeightfieldCollisionShape(heightmap,tq.getLocalScale()), 0));
[/java]
I’m not absolutely sure why this fixes it, I found the solution just through playing around 
Probably the issue is the height scaling, because I had just flat physical terrain, some distance under the graphical terrain, but, as I said, I’m not really sure.
Still, it works now.
Just tested it out, have to report a bug. I set the view distance to 5 (if that matters) and when I got to what I think was the edge of the map, fps went down and there were noticeable studders. Then it wouldnt load any more tiles, like if I went back towards the center it just wouldn’t load them. When I got far enough away from the other tiles though (near the end of the map) they left the cache and then I was lost in blue space.
EDIT: I thought the tiles were being loaded from images. Just saw it is noise based. This is probably a problem then.
Oh, and the loading of tiles just seemed a lot slower than with the default jmonkey terrain grid. Probably because the default one has a 2x2 grid and caches 16 tiles outside of that grid as well. EDIT: This is probably just because its generating it based on noise, nevermind.
@8Keep123 The generation of noise is time-intensive, there’s little I can do about that without implementing my own noise generator, and even then, I doubt it would be justifiably different, and there’s the fact that we would be moving away from the jme implementation.
I can cache an extra tile around the circumference of the view distance, that’s not a problem, and attach the tiles in a more logical way as @monkeychops mentioned.
Regarding saving… Saving a tile after generation and loading a tile instead of generating it every time will be orders of magnitude faster. I guess @Sploreg could shed some light on whether this system should take care of that, or whether should this be done by the user? I guess this situation only arises on noise generated worlds.
@Toboi i’ll look into that, I havent actually added a player to the scene yet 
And also, lets not forget: If we take a world unit as a meter, a player would be 2m in height, and 100km/h would equate to 27.78m/s. So given the fact that 100km/h is 62mph - 186mph would be 83.34m/s - so a flycam speed of 300 is unrealistic in any sense unless you were in a rocket or flight simulator.
I am really pleased to see everyone is commenting on this and @jayfella is working on it so actively… I think this is going to be awesome when we’re done
I certainly think the noise generation is slowing things down from my testing… there must surely be some way we can prepare the tiles ahead of when they’re needed.
What about if we had one thread that just continuously generates the tiles around the circumference of the current terrain, spiraling out to one or two rows further than the view distance, and adding them to the cache and then another thread which generates any missing tiles that aren’t in the cache as they’re needed?
Granted, we’d be generating tiles that weren’t needed, but rather than try to generate all the tiles in surges, they’d be just smoothly streaming in and out.
Also, I guess a disk-based cache would really help… just keep making tiles until you have the whole world on the disk or the count hits the specified limit 
Well I believe @pspeed had a very good idea, in that instead of iterating through the que to check if they are done, just put the newly generated tiles into a Deque instead. We can then simply check each frame if that deque .isEmpty() and pull the first one from it if not. This saves not only on iteration of a hashmap, but also a Deque is a lot faster in removal of the first object than a hashmap is removing from a random index, so we get a double performance boost. I’m going to do this now. I’ll let you know when I’ve pushed the results.
Continuous iteration (of generating and saving tiles continually in the background) would chew the hell out of hdd space pretty quicky, but I think we can take a pinch of this idea and generate/save an extra tile around the circumfrence of the view distance. We wouldn’t even have to put it in the cache, because loading a pre-generated tile is waaaay quicker than generating it, but I guess we can discuss that once we’re in the position to do so.
@jayfella said: Well I believe @pspeed had a very good idea, in that instead of iterating through the que to check if they are done, just put the newly generated tiles into a Deque instead. We can then simply check each frame if that deque .isEmpty() and pull the first one from it if not. This saves not only on iteration of a hashmap, but also a Deque is a lot faster in removal of the first object than a hashmap is removing from a random index, so we get a double performance boost. I'm going to do this now. I'll let you know when I've pushed the results.Continuous iteration (of generating and saving tiles continually in the background) would chew the hell out of hdd space pretty quicky, but I think we can take a pinch of this idea and generate/save an extra tile around the circumfrence of the view distance. We wouldn’t even have to put it in the cache, because loading a pre-generated tile is waaaay quicker than generating it, but I guess we can discuss that once we’re in the position to do so.
ConcurrentLinkedQueue.poll()… and then just check if it’s null or not.
1 Like
Thanks for the fixes you’ve done @jayfella, I still don’t have much time until later today, so can’t help much at the moment however I’ve been able to create a sample which I think is somewhat like @pspeed suggestion. Maybe you can use it somehow…
[java]
import java.util.ArrayList;
import java.util.Random;
public class Example {
private static ArrayList<Integer> queue = new ArrayList<Integer>();
public static void main(String[] args) {
// This would be the thread checking if new tiles should be created etc,
// and also creates them in this thread, adds them to queue when done
Thread tileCreator = new Thread(new Runnable() {
@Override
public void run() {
for (;;) {
// Create tiles here instead of this random stuff
Random random = new Random();
int randomNumber = random.nextInt(5000);
synchronized (queue) {
queue.add(randomNumber);
}
try {
Thread.sleep(randomNumber);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
});
tileCreator.start();
// This simulates the simpleUpdate, which gets a terrain piece from the queue if its not empty
for (;;) {
// Get the oldest item of the queue and add it to the rootnode etc.
synchronized (queue) {
if (!queue.isEmpty()) {
System.out.println("Received: " + queue.get(0));
// add queue.get(0) to scene graph here
// and remove it from queue
queue.remove(0);
}
}
}
}
}
[/java]
“synchronized (queue)”
If you find yourself doing that ^^^ You are doing it wrong wrong wrong.
Look in java.util.concurrent, live it, learn it. ConcurrentLinkedQueue is your friend in this case… which is why this is my third time mentioning it.
1 Like
Yeah I did it the way you suggested, @pspeed and it worked a treat.
I’ve pushed the changes to github. The tiles are being added to the scene at a noticably quicker rate now.
@jayfella said:That could be a possibility. You would have to watch out that the disk IO isn't bogged down with tile loading so that it delays sounds that are loading, or other assets. As you stated with the performance metrics and the sheer size of the terrain, I don't think you have an issue with lag. 1 meter units for terrain is quite detailed to begin with too. Using the pre-loaded border tiles is a good solution, if the tiles aren't too large! If they are big, you will eat up memory quickly.Regarding saving… Saving a tile after generation and loading a tile instead of generating it every time will be orders of magnitude faster. I guess @Sploreg could shed some light on whether this system should take care of that, or whether should this be done by the user? I guess this situation only arises on noise generated worlds.
Well done, I’ll check that out then jayfella
I really haven’t done much with threads in java before, except for running separate tasks that didn’t really require passing info between them etc. so sorry for giving a bad example.Why exactly is this so wrong wrong wrong then @pspeed :p? I’m wondering because I’ve seen it popup a lot of times in code examples etc.
Ok, so I’ve just tested this and some things are still wrong :.
First of all, I still notice framedrops when terrain chunks are being added which are not from the cache, the chunks from cache are very smooth and fast now though.
The biggest problem was, when I was randomly flying around I got a crash after ~1 minute
[java]
Exception in thread “AWT-EventQueue-0” java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
at sun.nio.ch.IOUtil.read(IOUtil.java:196)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
at sun.font.TrueTypeFont.getTableBuffer(TrueTypeFont.java:839)
at sun.font.CMap.initialize(CMap.java:154)
at sun.font.TrueTypeGlyphMapper.<init>(TrueTypeGlyphMapper.java:53)
at sun.font.TrueTypeFont.getMapper(TrueTypeFont.java:1526)
at sun.font.FileFontStrike.<init>(FileFontStrike.java:187)
at sun.font.FileFont.createStrike(FileFont.java:95)
at sun.font.Font2D.getStrike(Font2D.java:344)
at sun.font.Font2D.getStrike(Font2D.java:293)
at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:77)
at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1012)
at javax.swing.JComponent.getFontMetrics(JComponent.java:1624)
at javax.swing.plaf.basic.BasicGraphicsUtils.getPreferredButtonSize(BasicGraphicsUtils.java:276)
at javax.swing.plaf.basic.BasicButtonUI.getPreferredSize(BasicButtonUI.java:376)
at com.sun.java.swing.plaf.windows.WindowsButtonUI.getPreferredSize(WindowsButtonUI.java:142)
at javax.swing.plaf.basic.BasicButtonUI.getMinimumSize(BasicButtonUI.java:366)
at javax.swing.JComponent.getMinimumSize(JComponent.java:1742)
at javax.swing.plaf.basic.BasicOptionPaneUI$ButtonFactory$ConstrainedButton.getMinimumSize(BasicOptionPaneUI.java:1444)
at javax.swing.plaf.basic.BasicOptionPaneUI.addButtonComponents(BasicOptionPaneUI.java:692)
at javax.swing.plaf.basic.BasicOptionPaneUI.createButtonArea(BasicOptionPaneUI.java:630)
at javax.swing.plaf.basic.BasicOptionPaneUI.installComponents(BasicOptionPaneUI.java:178)
at javax.swing.plaf.basic.BasicOptionPaneUI.installUI(BasicOptionPaneUI.java:141)
at javax.swing.JComponent.setUI(JComponent.java:664)
at javax.swing.JOptionPane.setUI(JOptionPane.java:1861)
at javax.swing.JOptionPane.updateUI(JOptionPane.java:1883)
at javax.swing.JOptionPane.<init>(JOptionPane.java:1846)
at javax.swing.JOptionPane.showOptionDialog(JOptionPane.java:858)
at javax.swing.JOptionPane.showMessageDialog(JOptionPane.java:667)
at javax.swing.JOptionPane.showMessageDialog(JOptionPane.java:638)
at com.jme3.system.JmeDesktopSystem$1.run(JmeDesktopSystem.java:93)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:251)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:721)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:682)
at java.awt.EventQueue$3.run(EventQueue.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:691)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
[/java]
The latest version is a whole lot better… much smoother and the slowdown is much much less perceptible. Way to go @jayfella and @pspeed
The terrain holes seem to have got worse again though… thats the main “bug” I’d say now.
@Sploreg - on another note, I was wondering if we could do something with LOD based normal and detail textures on the terrain, so that the far terrain just uses the base texture but then closer terrain gets fine detail, normal and spec?
@reveance said: Well done, I'll check that out then jayfella :)I really haven’t done much with threads in java before, except for running separate tasks that didn’t really require passing info between them etc. so sorry for giving a bad example.Why exactly is this so wrong wrong wrong then @pspeed :p? I’m wondering because I’ve seen it popup a lot of times in code examples etc.
Because the synchronized keyword is the absolute most heavy-handed way to gate access that there is. java.util.concurrent does much smarter and much faster things.
As a rule, avoid “synchronized” when there are alternatives. Even when there is no thread contention, it’s pretty expensive because it has to interact with the object monitor, etc. (ie: avoid in inner loops at all costs). And when there is thread contention, it is still more expensive than other threading approaches and you are likely to drop a full time slice from the waiting thread… which would ultimately kill throughput if you have a lot of cases of concurrent access attempts.
Proper design with threaded data structures in mind can eliminate all, or nearly all, uses of the “synchronized” keyword… and often it turns out to be a better design for other reasons, too (see double queues above).
Also, many of the data structures in java.util.concurrent employ lock-free designs. Very very smart people wrote that code. Some of it is quite clever and all of it is well reviewed. Many of us “old dogs” were employing and admiring Doug Lea’s work long before it made it into the JDK. It’s a real boon to Java developers and so sad that it seems so unknown to most.
1 Like
@pspeed said: Because the synchronized keyword is the absolute most heavy-handed way to gate access that there is. java.util.concurrent does much smarter and much faster things.As a rule, avoid “synchronized” when there are alternatives. Even when there is no thread contention, it’s pretty expensive because it has to interact with the object monitor, etc. (ie: avoid in inner loops at all costs). And when there is thread contention, it is still more expensive than other threading approaches and you are likely to drop a full time slice from the waiting thread… which would ultimately kill throughput if you have a lot of cases of concurrent access attempts.
Proper design with threaded data structures in mind can eliminate all, or nearly all, uses of the “synchronized” keyword… and often it turns out to be a better design for other reasons, too (see double queues above).
Also, many of the data structures in java.util.concurrent employ lock-free designs. Very very smart people wrote that code. Some of it is quite clever and all of it is well reviewed. Many of us “old dogs” were employing and admiring Doug Lea’s work long before it made it into the JDK. It’s a real boon to Java developers and so sad that it seems so unknown to most.
Alright, I definitely didn’t know that :p. In that case It’s a shame it’s used in so many code examples when it turns out to be such a bottleneck :O, anyways thanks alot for the explanation 
@monkeychops said: on another note, I was wondering if we could do something with LOD based normal and detail textures on the terrain, so that the far terrain just uses the base texture but then closer terrain gets fine detail, normal and spec?
Yea stuff can be done with that. The technique I am considering is called Multi-UV Mixing as described here.
No time to implement it right now though, sorry. Gotta do paid work so I can eat =)
It wouldn’t be difficult to implement however.
Currently there is no spec map on the terrain material. I think for that we would really want to switch to an OpenGL 3.0 based Texture Array implementation to allow for the increase in textures.
@Sploreg Cool Do we really need to increase textures though? We can put specular in the alpha channel of the normal?
Maybe others of us can have a look at this… I have to say now that we have a paging solution from @jayfella, and hopefully trees to come, I am really inspired to dig into the internals a bit more myself
@reveance I do actually notice a slight drop myself when generating the terrain. I’ll see what I can do. It looks like you ran out of memory - probably the cache getting too big. I’ll look into making it a fixed size.
@monkeychops the LOD we are using in this implemenatation is the TerrainLodControl straight from jme so unfortunately I can’t do anything with it. @Sploreg mentioned some shader wizardry, and cards on the table, I know next to nothing about shaders so I couldn’t implement that even if I wanted to. We are at the whim of the wizard himself.
Sure the specular can be placed in the alpha channel of the normal. It’s just that then the normal texture needs to be manipulated before it gets sent to the card, and it could be a bit of a maintenance headache.
@jayfella the TerrainLODControl just handles geometry, not the materials.
@jayfella said: @reveance I do actually notice a slight drop myself when generating the terrain. I'll see what I can do. It looks like you ran out of memory - probably the cache getting too big. I'll look into making it a fixed size.@monkeychops the LOD we are using in this implemenatation is the TerrainLodControl straight from jme so unfortunately I can’t do anything with it. @Sploreg mentioned some shader wizardry, and cards on the table, I know next to nothing about shaders so I couldn’t implement that even if I wanted to. We are at the whim of the wizard himself.
Alright thanks,
I actually don’t think it’s the cache getting to big though, it happens after some time, the cache always remains <250 for me, in previous cases it crashed after about 1-2 minutes, while still being somewhere around 200-250 :\