So I got the lighting working, shadows still fine.
The only thing thing that is not working is specular highlights, I have no idea what I could try to get them working.
I got this far by using both suggestions but on 2 different instances of the low poly model. One with colorwrite off and one with blendmode modulate.
I have set the material for the modulate model to shine with:
(also did it to the one without colorwrite just in hope)
mat2.setBoolean(“UseMaterialColors”,true);
mat2.setColor(“Ambient”,ColorRGBA.White);
mat2.setColor(“Diffuse”,ColorRGBA.White);
mat2.setColor(“Specular”,ColorRGBA.White);
mat2.setFloat(“Shininess”, 64f);
But shine does not come through with modulate, any ideas?