Are you using Batchnode for your objects?
And did you put the ShadowMode on this BatchNode?
I did this and I had a compareable result. So the BatchNode keeps a list of all objects you add to the BatchNode. If you batch it it generate some Gemetries (depending on your amount of materials). Instead of put the ShadowMode to the whole BatchNode, I just put it on this Geometries. After that the amount of Objects in the scene decrease dramaltically