Hello, I am having a very strange issue. I have not found any other posts about a similar issue.
I am trying to use save my custom geometry (it and everything it saves implements Savable), but after its write method completes it hangs, not progressing, and CPU usages spikes to around 90%!.
If anything is happening I don’t know what it is. The file it produces is not being enlarged as far as I can tell.
Wow, I’m not sure why It didn’t occur to me to check that but memory usages is at 4.6gb! Pretty sure that is my heap size limit.
Why in the world is the exporter using so much memory? The object I am saving is very large, probably about 1gb (in memory) itself, is that the problem? Maybe I should save it in chunks somehow?
The memory usages is very large during the exporting (~2.5 gb), but after it spikes to the limit.
Should I just increase the heap size? That does not bode well for performance of the exporter, though.
As for as platforms go, there are several ways I plan to optimize the memory usage of the objects, but the exporter seems to use several times the memory of the object to export it, which seems surprising to me. Further questions:
Any suggestions for the best way to split a highly complex geometry for export? The other objects can just be saved separately, which should be easy.
Is there anyway to reduce the memory usages of the exporter? If I had to guess from the code, the Maps used for keys to the binary data are getting enormous.
Would the XMLExporter be any better in these regards?
@TisAFleshWound said:
As for as platforms go, there are several ways I plan to optimize the memory usage of the objects, but the exporter seems to use several times the memory of the object to export it, which seems surprising to me. Further questions:
Any suggestions for the best way to split a highly complex geometry for export? The other objects can just be saved separately, which should be easy.
Is there anyway to reduce the memory usages of the exporter? If I had to guess from the code, the Maps used for keys to the binary data are getting enormous.
Would the XMLExporter be any better in these regards?
The current exporter is quite simply not meant to operate on such large models. I don’t think XMLExporter would be better, it will probably be worse actually.
How many geometries are we talking about here? And how many vertices are in each geometry?
@Momoko_Fan said:
The current exporter is quite simply not meant to operate on such large models. I don't think XMLExporter would be better, it will probably be worse actually.
How many geometries are we talking about here? And how many vertices are in each geometry?
Um… you know, only about 1 main geometry with let’s say about 4 million vertices. Not too big.
Seriously though, when you put it like that, I’m not surprised. I’m probably expecting too much of it.
Users will definitely not need to load a geometry that big, but that are some very useful production purposes for it.
is there maybee a physical rigibdbody attached to it? that tries to serialize its acceleration structures? As this might easily several times the amount of the mesh.
if so you could try to remove it before saving, and adding it again after save.
is there maybee a physical rigibdbody attached to it? that tries to serialize its acceleration structures? As this might easily several times the amount of the mesh.
if so you could try to remove it before saving, and adding it again after save.
I’m actually not sure what that even is, so it seems unlikely. Also, unless it is trying to save that after I did super.write() it doesn’t seem to make sense.
Okay, I have established that the problem is definitely not with the Geometry. I have done tests with every combination of parts of of the class being saved, and saving only the Geometry works fine, actually. So the main problem is the objects I am saving
Increasing the heap size shows that trying to save everything I want to save, minus the duplicate vertices (why did I even save those, who knows) actually works but takes about 9gb of memory and 160 extra seconds. Additionally, the extra memory does not seem to get garbage collected after saving is done. Does anyone know why that is?
So, technically the problem was as suggested, JVM was running out of heap and trying to burn out my CPU to garbage collect.
Now my problem is optimizing the long saving time, My main question now is, are certain types of objects harder for the Exporter to organize after the write method?
My guess is that the ArrayLists are hard for it, unless somebody knows something I don’t I was considering write/read of the contents of the list manually instead.
My second guess is that the amount of data is simply too large for it to save quickly. The only way I could see fixing that would be saving things in individually smaller files
It seems you have a huge amount of data. Nothing you do will help much until you can fix that. You might want to take a step back and rethink your whole approach and/or explain here what you are actually trying to accomplish so we might offer better advice than “run it in a profiler to see where the memory is going”…
@TisAFleshWound said:
Okay, I have established that the problem is definitely not with the Geometry. I have done tests with every combination of parts of of the class being saved, and saving only the Geometry works fine, actually. So the main problem is the objects I am saving
Increasing the heap size shows that trying to save everything I want to save, minus the duplicate vertices (why did I even save those, who knows) actually works but takes about 9gb of memory and 160 extra seconds. Additionally, the extra memory does not seem to get garbage collected after saving is done. Does anyone know why that is?
So, technically the problem was as suggested, JVM was running out of heap and trying to burn out my CPU to garbage collect.
Now my problem is optimizing the long saving time, My main question now is, are certain types of objects harder for the Exporter to organize after the write method?
My guess is that the ArrayLists are hard for it, unless somebody knows something I don’t I was considering write/read of the contents of the list manually instead.
My second guess is that the amount of data is simply too large for it to save quickly. The only way I could see fixing that would be saving things in individually smaller files
Okay so saving the 4 million vertices geometry works fine. So besides that one, how many objects / geometries are we talking about and how many vertices per each object? Are you actually rendering this data? If you’re having trouble saving it, I am wondering how is it even possible to render it…
@Momoko_Fan said:
Okay so saving the 4 million vertices geometry works fine. So besides that one, how many objects / geometries are we talking about and how many vertices per each object? Are you actually rendering this data? If you're having trouble saving it, I am wondering how is it even possible to render it..
Sorry for slow response. I’ve been tinkering with the exporter. I’ve managed to improve things somewhat, but not a satisfactory amount.
The 4 million vertices geometry is one large mesh. I am indeed rendering it, and since I am only using one mesh it is rendering just fine, at least on my fairly powerful computer. Most of the other data is actually not needing to be rendered.
The geometry itself is actually not the problem, there is a lot of other information to go with it that is however.
Basically the geometry is a heavily triangulated icosphere. That by itself makes a lot of vertices. The problem I’ve been having is all the information I am generating that goes along with the geometry. This includes ways of representing the individual triangles of the icosphere for references, but it mostly includes other information that I use for each triangle. This means quite a lot of data indeed.
My plan right now is to save parts of the data seperately, or saving smaller, less complete data I can extrapolate from.
Have you thought about writing a own serializer for your data?
Jme alloews you to create a own class and implement the save logic for it.
It might help greatly (or nothing at all) depending on the case.
Then you could also try to simplyfy your data model.
Eg try to store data in more byte[] like structures raw without the type, and extract the data from it as necessary.
@Empire Phoenix said:
Have you thought about writing a own serializer for your data?
Jme alloews you to create a own class and implement the save logic for it.
It might help greatly (or nothing at all) depending on the case.
Then you could also try to simplyfy your data model.
Eg try to store data in more byte like structures raw without the type, and extract the data from it as necessary.
Do you mean by implementing savable and making custom write/read methods? I’ve been doing that but I’m not sure that is what you mean.
Yeah, from what you say it sounds as if it should be possible for your data to store it in a more descriptive way, say “sphere with radius x” instead of storing all vertices of the created sphere.
Yes I am indeed storing data descriptively when possible. In regards to the format, I would ask for more clarification on what types of formats you think would be helpful. The majority of the data is various types of Vectors, which are simply composed of floats. I guess you mean decreasing precision in these cases on saving? I would appreciate an expansion on this because it could be helpful.
For example (regarding data description) I have made passes eliminating the saving of information I have noticed is calculable on read(). I have also avoided duplicating information wherever I have noticed it There may be more I can do here, however.
Unfortunately the solution of not saving all the vertices doesn’t help much as I can see. The primary reason being that the sphere is not guaranteed to be regular. As in, the amount of triangulation in different parts of the sphere is not guaranteed to always be uniform, and in fact, likely to be otherwise. This is for development, where I may need different levels of detail in different parts of the sphere, and for end users, where optimization involves combining areas that are similar by de-triangulating them.
