ProjectiveShading: Inserting 3D Objects into Indoor Images with Complex Shadows

Abstract Realistically inserting virtual 3D objects into real‐world images requires perceptually coherent shadowing of the object and background scene. Achieving this in single‐view indoor scenes with sunlight is challenging due to complex, partially visible occluders and indirect lighting. Environment maps alone cannot produce realistic shadows on virtual objects, and any representation (scene parameters) used for rendering must be practically estimable. We introduce ProjectiveShading, the first automatic method for inverse‐ and re‐rendering that handles bi‐directional shadow interactions for realistic object composition. Our key innovation is the sunlight map, a 2D image encoding direct sunlight and arbitrary occlusions. It is generated from single‐view estimations using off‐the‐shelf models and is compatible with standard rendering engines. We also propose algorithms to estimate sunlight direction and to blend virtual and real shadows while preserving background textures. Experiments on synthetic and in‐the‐wild images show our method outperforms previous approaches.