I think a few dot products can work here (and why I think they are one of the most magic of vector operations). Just need to rework the problem a bit.
First, flip your inVelocity to point out from the surface…
flippedVelocity = inVelocity.multLocal(-1)
Now you have a normal pointing out of 0,0,0 and an inVelocity pointing out of 0,0,0 and we just need to reflect it. If you imagine, there is a vector we need to find that stretches from the tip of inVelocity over to the normal vector “line”. If we find this extra triangle side then we can find the tip of the reflected vector simply by reversing it. But we need to know where along the normal vector the unknown point is…
Dot product to the rescue.
float projected = normal.dot(flippedVelocity);
v1 = normal.mult(projected);
v2 = flippedVelocity.subtract(v1);
Now we have found all of the parts of the right triangle. v1 is the base leg, and v2 is the vector extending from v1… the other leg of the triangle. The flipped inVelocity is the hypotenuse.
If we flip v2: v2.multLocal(-1)
And add it back to v1:
reflected = v1.add(v2)
We have the reflected velocity… because we flipped the triangle.
At least that’s my back-of-napkin level solution. Hopefully I explained it well enough not to require pictures.
The key is to remember that the dot product of some unit vector and some other vector will tell you the cosine of the triangle formed between those two vectors… scaled to the non-unit vector. (ie: if both vectors are unit vectors then you get the actual cosine of the angle between them presuming that in v1.dot(v2) v1 is the base and v2 is the hypotenuse)
