There is a simple way to get a correct selection. The perfect match is more tricky.
You’re trying to translate your selection rectangle from screen space to world space. You will find it much easier to tranlate your unit position from world space to screen space.
Now you just have to check if unit screen projection is inside you 2D box.
Does that help?
You’ll find my implementation for OpenRTS at the bottom of this class
Don’t hesitate to @mention me if you need more RTS code tricks