This is the kind of thing that seems like it would be way easier to do with a UI that is real scene graph objects, like Lemur. There was even another thread on this recently already.
And actually, in the case of Lemur, it wasn’t even using UI components but just the picking and drag handler already built in.