Hello,
So I am trying to get anti-aliasing to play nice with my desktop application. Using the app settings and setting the sampling to 4 works create for any scene connected to the default viewport, but other viewports I setup myself with their output framebuffers don’t seem to be anti aliased at all (many jagged lines).
The process I am using is like this:
[java]
settings = new AppSettings(true);
settings.setWidth(1280); // render width (will be overridden)
settings.setHeight(720); // render height (will be overridden)
settings.setBitsPerPixel(32); // 32bit pixel
settings.setSamples(4); //
…
forwardRenderViewPort = renderManager.createMainView(“forwardRenderViewPort”, deferredViewPort.getCamera());
…
deferredDepthData = new Texture2D(1280, 720, numSamples);
Texture2D combineDeferredResultsTexture = new Texture2D(1280, 720, numSamples, Format.RGBA8);
…
FrameBuffer forwardRenderResultsViewPortOutputFrameBuffer = new FrameBuffer(1280, 720, numSamples);
forwardRenderResultsViewPortOutputFrameBuffer.setMultiTarget(false);
forwardRenderResultsViewPortOutputFrameBuffer.setDepthTexture(deferredDepthData);
forwardRenderResultsViewPortOutputFrameBuffer.addColorTexture(combineDeferredResultsTexture);
forwardRenderViewPort.setOutputFrameBuffer(forwardRenderResultsViewPortOutputFrameBuffer);
[/java]
When I set the samples for the framebuffers, all textures must have the same number of samples as well. I don’t really understand how AA works exactly, but I was under the impression that the samples for textures would be texture samples (trilinear, etc) and the samples for frame buffers would be edge smoothing sampling.
Can anyone help me understand how to get the AA (edge smoothing kind) to work on viewports other than just the main one?
Actually handling multi sampling and framebuffer is a bit tricky. What you did is fine for opengl 3.1+ but for prior version you have to copy the multi sampled frame buffer result to a single sampled framebuffer.
Maybe you hit that issue.
pseudo code would be
createMSFrameBuffer();//actually what you did : a multisampled framebuffer attached to the viewport
createSSFrameBuffer();//the frame buffer holding your resulting texture
…
after Render :
renderer.copyFrameBuffer(msFrameBuffer,ssFrameBuffer);
then your texture should be anti aliased.
That’s what is done in the FilterPostProcessor check the reshape method where frame buffers are initialized and postFrame where they are copied.
Note that we check for hardware caps and do this only for prior ogl3.1 version.
Eventually using a FXAA filter will be enough? It is usally mostly equal quality to AA but faster.
@nehon
Thank you for the information nehon!
I will check through my stuff and try to figure it out 
@Empire Pheonix
I am on the fence about FXAA. I find it makes everything a little too blurry. Especially textures that are detailed, like a carpet. But I am considering using it in certain cases. For instance, using it on all lines drawn, and wireframes.
@bigstink said:
I am on the fence about FXAA. I find it makes everything a little too blurry. Especially textures that are detailed, like a carpet. But I am considering using it in certain cases. For instance, using it on all lines drawn, and wireframes.
I'm not sure where you've seen that kind of blurriness but all the results I've seen of FXAA is the same as AA2 but with half the processing requirement.
I'd like to see comparison shots (AA2 vs FXAA) of the blurriness you're talking about. Since I'm using FXAA in my game, this might change things.
Yeah i agree, I currently use fxaa as well, if you can show the difference, i might decide to change that.
RE: FXAA Blur
Here are some screen shots of FXAA vs 2xMSAA in Max Payne 3. I didn’t take these screens shots; I just found them online.
http://www.hardocp.com/image.html?image=MTMzOTQwNjE2N2h1UDVxcXRtOVNfN18zX2wucG5n
I linked to that particular one because I think it’s a pretty good example of the pros and cons of FXAA (other than it’s ridiculously good performance boost). You can press the previous/next links at the bottom if you want scroll for more pics comparing the two methods.
In that picture you can see that the lines in the framework of the building are much smoother than the ones in 2xMSAA. This is the super awesome part of FXAA. However, if you look at the graffiti on the wall, it looks sharper and more crisp in the MSAA one. Unfortunately, the way FXAA samples causes this sort of blur on all textures and in some situations it’s worse than others. For instance, I was working with a scene awhile ago with a carpet which had a carpety design in it (swirls and stuff) and because of the way FXAA sampled it, it didn’t look good enough for what I needed (I was making a video).
So in my opinion, FXAA would work great for wireframes and line drawings. Or maybe even rendering objects without highly detailed textures, running the FXAA, and then rendering the rest. Though, tons of games are using it now and it’s included in nVidia’s drivers so I think most game makers and players feel the trade-off for the results vs performance are completely worth it.
You can google Battlefield 3 or Skyrim boards for FXAA and you’ll find that there are some people that hate it, and others that think it’s great. If you think it makes your game look great, just keep it in!
No filter is perfect and after seeing the pictures in the link above I don’t see why anyone would think FXAA is inferior. Compare the girders and that’s what 99% of games want to avoid. If I get a slight washed-out/blur then it’s something I can and certainly will live with. But that’s a matter of preference as my game supports both, but by default FXAA is enabled.
Correct.
Actually, while searching for FXAA and applying it to an edge map (from an edge detection pass) I came across this post regarding FXAA v4.0.
http://timothylottes.blogspot.ca/2011/12/fxaa-40-will-have-new-spatial-only.html
Sounds like Tim’s found a way to reduce the blur on textures with the upcoming release. Since his stuff is open source (unless nVidia pulled it back… but I don’t think so) we will likely see a port to JME and get that little be closer to perfection
I think both look nice but there is something fishy about this picture from that earlier link:
Where do all of the bumps on the sides of the girders come from in the non-AA pic. That’s not aliasing that’s something else. And regardless of how ugly that picture is, it’s not aliasing and so the MSAA picture is more correct to have kept it. FXAA in this case is blurring the actual image detail (the non-aliased bumps)… which is nice in this case but what if those bumps were important.
I think it’s not a very good example.
Yeeh, I thought the FXAA looked better in those images.
http://www.hardocp.com/images/articles/1339406167huP5qqtm9S_7_6_l.png
Look at the top of the dividing screen at the left of the image and the frame above the door behind it. Both are much better in FXAA. There is also the office organiser on the desk much better in FXAA, even the top of the chairs too.
It might just be down to people’s vision though. People vary a lot in what they notice - so for some people FX will be much better, for other people not so much.
I combine FXAA and an unsharp mask technique along the lines of the to the work @kwando did (Unsharp mask post filter) to help avoid the blurniess.