I did more or less this for smoothing my custom mesh. Basically you calculate a surface normal for the surrounding planes that share that 1 vertex. Take those normals and average them out. I have code that does this and is a bit easier to understand but even with the statement made by @Ogli and others above you should be able to find the equations you need.
It’s the same method I used to generate this smooth rock: (September 2017) Monthly WIP screenshot thread - #17 by thecyberbob
Also, and I can’t stress this enough, look into the Math for Dummies thing at the top there. It’s SUPER helpful when trying to generate a mesh. In particular look at the add, and normalize functions.