Since your vertex shader does not need to a projection matrix, you can definitely transform points vertex-points non-linearly. Eg. Instead of a projection matrix, you use an equation.

One problem thou with nonlinear projections is the following:

While you can successfully transform vertices into your desired space, the vertices form triangles.

And here is the issue, to render 3D mesh with a nonlinear perspective, you would like to transform its **triangles** into nonlinear space.

A **triangle** transformed into nonlinear space is most probably **not a triangle anymore**! Yes it can be a shape with 3 vertexes, but the **edges can be curved**!

Thus achieving nonlinear perspective with vertex shader is problematic. It can be achieved approximately if you use small triangles, ideally tesselation to simulate curved triangle edges.

The advantage of this is that it should allow you to render even 360 degree scenes.

An alternative, as you have mentioned, is to apply the nonlinear transform as a post process effect. The advantage of this is that it is probably cheaper than the above and faster to implement. The disadvantage is you are limited to your field of view and as you mention the near and far clipping planes are flat.

Another approach is to use ray tracing.