PhyIR: Physics-Based Inverse Rendering for Panoramic Indoor Images

Figure 1. Given an LDR panorama (a), we estimate geometry (b-c), SV illumination and SVBRDFs, including base color (d), roughness (e) and metalness (f). Our physics-based differentiable renderer can produce detailed specular reflectance (g) on complex material. Based on such physical constraint, our predictions are qualified to produce virtual object insertion (h) with realistic lighting effects, e.g., highlight caused by unseen light source in 1 and specular reflectance on the cabinet in 2.

Figure 2. Overview of our physics-based inverse rendering architecture. The framework consists of four modules (a-d). Given an LDR panorama I, geometry and material estimation module (a) first predicts coarse geometry (D, N ) and SVBRDF (B, R, M ). SC lighting estimation module (b) predicts physically correct per-pixel illumination with physics-based in-network rendering module (c). Finally, trainable guider filter module (d) refines the predicted BRDF.

Table 1. The comparison between previous datasets and our proposed FutureHouse. Our high-quality dataset contains comprehensive annotations.

Figure 3. Qualitative comparison of material estimation on FutureHouse.

Figure 4. Qualitative comparison on real data provided by LRG360 [30]. Although LRG360 with depth as input, our method predicts more detailed geometry.

Figure 5. Qualitative comparison of re-rendered images. With a more physical SVBRDF model and the physics-based differentiable renderer, our approach reproduces realistic lighting effects, especially non-monochrome specular reflectance on complex material, e.g., glossy wall and metal kettle.

Figure 6. Qualitative results of dynamic virtual object insertion. Our approach generate coherent results without any temporal constraints.