I guess I need to adapt the lights in the scene for PBR materials and find a new setup. I was now reusing the default light setup I always use for Lighting material based scenes.
Finding a new hierarchy to structure the models/lights in the scene should do the trick I guess.