CharaterControl sometimes seeps through RigidBody

So I got my player working with the CharacterControl and All my RigidBody objects which make up the ground. But sometimes the player will start to seep through the ground. I have yet to find a pattern. Sometimes it will happen when I fall onto one, other times it will happen as I am walking.

Any ideas?