DiViNeT: 3D Reconstruction from Disparate Views using Neural Template Regularization

NeurIPS 2023

Aditya Vora¹, Akshay Gadi Patil¹, Hao (Richard) Zhang^1,2

¹Simon Fraser University, ²Amazon

Overview

Given a set of sparse view images of a scene from different views and their corresponding camera poses, DiviNet predicts surface priors in the form of templates (3D gaussian functions) which is then used to regularize the surface reconstruction process in the volume rendering framework. This template prediction network is trained across a dataset of objects with sparse point cloud reconstructed from multi views as supervision. Our method can reconstruct the surface under sparse conditions without using any explicit fine-grained ground-truth like depth maps.

DiviNet operates in two stages. In the first stage it learns a template prediction network and in the second stage uses the predicted templates to regularize the reconstruction.

Surface Reconstruction on DTU Dataset

Our method faithfully reconstructs the geometry from under sparse conditions. We use the same set of images which are used by RegNerf.

Results of sparse view (3 disparate views) reconstruction on DTU Dataset.

Surface Reconstruction on BlendedMVS Dataset

We show the surface reconstruction results of objects from BlendedMVS dataset under sparse view scenario. As it can be seen DiviNet can faithfully reconstruct highly detailed surface even on complex real-world dataset.

Results of sparse view (3 disparate views) reconstruction on BlendedMVS Dataset.

Surface Details

Since we do not use explict priors like depth maps which are often smooth and have low resolution, our method is able to reconstruct surface with much better details then previous methods which use explicit priors like depth maps.

DiviNet can reconstruct better surface details compared to previous methods.

BibTeX

@article{vora2023divinet,
      title={DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization},
      author={Vora, Aditya and Patil, Akshay Gadi and Zhang, Hao},
      journal={arXiv preprint arXiv:2306.04699},
      year={2023}
    }