I’ve only used the physics space ray tracing a handful of times, but if I remember correctly, using the physics ray test can have slower performance compared to using the standard ray collision. I should add that in my situation, I was casting multiple rays every frame, so this could be why I was noticing a significant difference.
Edit: Also here’s a link to a thread discussing a few different ways to filter your colllidable objects Selective Raytracing