I tried Allure reports and I wasn’t a fan for this. You couldn’t just open the generated HTML report in chrome because of a bunch of CORS errors, you had to create a server and host the thing. It felt like a hassle. So I’ve used Extent reports and gotten good results.
Here is the way it looks
I have the expected image, the actual image, and a red lined diff image. I have messed with the reference image and drawn a mouse on it and you can see that outlined in red in the diff image.
I tried creating a single PDF document but it was a bit horrible (tiny images, and the Extent extension for that was a bit hard to use). I think this works for quickly clicking through.
As an interesting aside this is what the water image looked like on my computer vs the software renderer reference image
Almost no pixel that was the same between the two!
I’m thinking of having 2 levels of test:
- On the reference infrastructure the image generated is consistently the same. If a difference is detected in these images the test is marked as “Failed” and the step is a fail (maybe this could run before a PR is merged, blocking a failing merge?)
- Even on the reference infrastructure the image generated is variable (non deterministic). These are marked as “Warning” in the report and can be manually reviewed but do not fail the step