I'd like to support instancing in my renderer. I was thinking I could go further and just make every object its own scene and use instancing to construct the "global" scene. This would simplify things and also allow me to do deduplication of geometry data on scene construction in order to do a sort of "automatic instancing".
The obvious downside would be doing an extra transform on each ray for every object intersection, but are there any other overheads involved? Would the BVH created from such an organization of the scene be less optimal than just passing all the objects in one scene?
This will definitely cost performance. The extra transformations will cost performance, and the BVH quality will definitely be worse (in particular of geometries overlap much).
I would recommend to instance only geometries that have at least two transformations in the scene graph. Geometries which have a single transformation should get pre-transformed and not instanced. This way the non-instanced geometry will achieve optimal performance. You can get even higher performance for the instanced part of the scene, by clustering all geometries that are transformed the same into one scene to instance.