Enhancing sparse multi-view super-resolution with unified multi-plane image spatial representation

Abstract The popularization of multi-camera systems and multi-view image capture has led to the emergence of sparse multi-view image super-resolution (MVSR) as a promising research direction. The primary approach to sparse multi-view super-resolution (SR) currently involves extending traditional single-view SR and stereo SR frameworks, i.e., extracting features from each view and performing pixel-domain alignment and fusion to leverage cross-view reference information. However, this straightforward framework has two main drawbacks. First, performing cross-view fusion in the pixel domain disregards the spatial perception information that multi-view images provide. Second, feature alignment and fusion across views introduce considerable redundant and repetitive computations, which hinders further scalability to more viewpoints. This paper proposes a novel sparse multi-view SR framework based on a unified spatial representation reference. Specifically, the proposed method first computes a multi-plane image spatial representation from the multi-view images. This multi-plane image (MPI) representation encapsulates all the information from each view and has spatial perception. Subsequently, an upsampled reference image is rendered from the MPI representation for the low-resolution views. A high-low frequency separation fusion network is then proposed to upscale the input low-resolution images based on the rendered reference. Experimental results demonstrate the effectiveness of the proposed method for recovering high-frequency details.