Terrain performance issues

I’m having the same problem with falling out of the terrain, I have a terrain that has a heightmap array with 4,096 values, and the world size is set to 17 (2 ^ 4 + 1) and my patch size is 5 (2 ^ 2 + 1). I have a pretty good graphics card (usually about 128 - 234 FPS) and usually near the edge of the map, and usually at a valley or the top of a hill, I will fall out of the map. This also occurs if I slide down a face due to gravity, or occasionally, just for seemingly no reason. Is there a way to fix this?