Physics engines are one of the few places in the universe where spheres actually ARE round.
…so then you have to look at the surface they are rolling on. Or in your case, how you are applying the force.
…because while one giant huge single polygon surface can cause issues for a physics engine’s collision detection, your issue looks like you are applying strange forces as the object rotates. So I’d check the force code first and make sure your applied vectors are consistent with expectations.
Edit: and that you are using the sensible method to do so.