Two things… I’ve definitely seen Java give memory back to the system since JDK 5… though it’s hit or miss. I still use the task manager to track this. I’ve seen application go from using 1 gig back down to 400 meg a time or two.
However, the thing that throws the numbers off between the JVM’s view of its heap and the system’s view of its heap is the direct memory that is allocated for meshes, etc… the stuff that needs to go to OpenGL. This is not counted in the JVM’s visible memory (from Java) but it is memory still held by the application until GC gets around to freeing it… in its own good time.
I’m half looking for general solutions to this issue as well since Mythruna churns through direct memory and the garbage collector doesn’t always keep up so well. In my case, my final plan is to reuse the buffers when I get to that kind of optimization.