Some things likely use the components directly and would mess up.
You might be better off continuing to use your custom scene graph, really... or write your own Geometry subclass that overrides everything to do with transformation and then only do your odd transforms at the "lowest level", ie: geometry.
When I say non-uniform scaling is only supported at the lowest level I mean it is accumulated separately from rotation and then applied only at the geometry level. So if you give non-uniform scaling to a parent node and then rotate the child you will get very strange results.
Non-uniform scaling is only really supported at the geometry level and I guess only because it's convenient to reuse a box and scale it non-uniformly, etc.