This latest version includes a few bug fixes, and all of the regressions that were reported during the alpha phase should now be fixed.
So now is a good time to try implementing the beta branch into your projects to check for any remaining issues and report back with your results.
I plan to move to stable only after a solid week or two of no new regressions being reported. It will also be useful to hear reports of any testers who run the beta with no issues to ensure that enough successful testing has occurred before stable.
I appreciate your work on the new release, but I have some doubts about the addition of the new jme3-screenshot-tests module. While I understand its potential value for image study and analysis, I have some concerns about its placement within the core engine.
Specifically, I’m wondering if it aligns with the primary needs of our developers. From my perspective, it seems to add files that everyone needs to incorporate into their local engine copies, which then require ongoing maintenance (making the job even more difficult for future release managers!). Given that it didn’t seem to generate significant community interest upon its initial introduction, and wasn’t utilized during the recent shader restructuring, I’m questioning if it’s truly essential for the main project.
Perhaps we could explore alternative ways to manage this module, possibly as a separate project? This would allow those who find it useful to utilize it, while keeping the core engine streamlined.
The key benefit of having it be part of the main project is that a PR can’t be merged if it breaks the tests. Making it the problem of the person who broke things to either update the tests (if it really is better) or to fix it.
If it is a seperate library then that makes it someone else’s problem to retrospectively unbreak what someone else broke (which is often the case today with things found in manual tests). My hope is that if these tests are expanded we can release more quickly without risking more bugs (i should find some time to write some, but help is always appreciated). This should make things easier for future release managers not harder.
I agree with the sentiment that more testing is better.
However I also have to point out that most of the regressions that did occur were related to a near entire rewrite of jme’s PBR shaders, and it was expected that there would be more regressions in this area than usual during alpha testing. Especially considering it was a 2 person job where riccardo wrote up a quick initial draft for pbr with structs as an alternative to my design, and he specifically indicated that he had not done thorough testing himself and there were many mistakes in this quick draft. But he was busy and I still wanted to include his struct based design, so I accepted the task and did my best to keep the engine in alpha until I was confident that the new design was thoroughly tested.
Most of the pbr related regressions were actually just minor mistakes that he or I made in the rewrite process. And two pbr issues that I initially referred to as regressions actually ended up being long standing issues that were overlooked in the past that would not have been found had it not been for the extra testing I did as a result of this rewrite.
Overall, I am confident in the testing that has occurred throughout the 4 alpha releases, and the PBR shaders are in a better state than ever: the final render is closer to industry standards for pbr, there are less bugs than before, and it is more organized and more extensible than ever before.
I agree, there was already one situation where the screenshot tests found an issue with a PR that was otherwise overlooked by the other checks/tests on github. So I would say that these tests are a useful addition to the engine.
I did initially misunderstand how the screenshot tests work though, so maybe it would be useful to have some type of official write up or documentation that explains what the screenshot tests are doing and how to analyze their results, so that way their benefits will be more apparent.
I did not intend to criticize your work for this release, i was following the issue tracker and know how much work has gone into this.
It was a more general statement that the screenshot testing is a vital feature since it is most likely the only way to actually unit test a graphics engine.