Novel View Prediction Error as a Quality Metric for Image-Based Modeling and Rendering

• Main Author: Waechter, Michael
• Format: Others
• Language: en
• Published: 2017
• Online Access: https://tuprints.ulb.tu-darmstadt.de/6780/1/Dissertation_Michael-Waechter_final-publication.pdf; Waechter, Michael <http://tuprints.ulb.tu-darmstadt.de/view/person/Waechter=3AMichael=3A=3A.html> (2017): Novel View Prediction Error as a Quality Metric for Image-Based Modeling and Rendering.Darmstadt, Technische Universität, [Ph.D. Thesis]

Image-based modeling and rendering (IBMR) is a sub-discipline of visual computing whose objective it is to capture images of a scene in the real world, construct a model of the world using the captured image data, and use this model to synthesize images of the world from previously unobserved viewpoints. This so-called novel (or virtual) view prediction has traditionally been tackled from two sides: On one side the computer vision community has pursued the construction of geometric models from sets of images only. On the other side the computer graphics community has worked on producing photo-realistic renderings from hand-modeled, virtual scenes and has further come up with algorithms that allow for the synthesis of novel views from input photos of real-world scenes either directly without any geometric models or with approximate, hand-modeled geometry models. The wealth of different IBMR systems also brought in its wake various quality evaluation systems that are more or less tailored to the properties of specific IBMR systems. In recent years, computer vision and graphics have grown together, slowly approaching the goal of novel view prediction on scenes without restrictions. However, the fragmentation of evaluation systems has still not been overcome. This thesis makes two main complementary contributions: We first present a novel texture mapping algorithm that assigns a static texture to polygonal 3D models, given images that are registered in the same coordinate frame as the model. Our texturing algorithm takes into consideration real-world scenes' properties such as illumination and exposure changes between images, non-rigid scene parts, unreconstructed occluders such as pedestrians, and images with pixel footprints that vary by orders of magnitude. We address the size (\ie, the number of images and the number of polygons in the geometry model) of real-world datasets with a novel Markov random field solver that solves the main bottleneck of our texturing framework orders of magnitude faster than related work. Conceptually, we can think of our texturing framework as closing the gap between image-based 3D reconstruction and photo-realistic rendering, thereby turning 3D reconstructions into full-fledged IBMR representations. Second, we introduce an evaluation scheme for IBMR methods that is guided by the definition of IBMR: Novel view prediction error evaluates how well an IBMR algorithm predicts novel views by dividing all input images into training and test images, keeping the test images secret, giving the training images to the IBMR algorithm, letting it predict the test images, and comparing its predictions with the actual test images. In this thesis we verify that (if used in conjunction with suitable image comparison metrics) this scheme fulfills a range of basic, intuitive conditions. We further compare our scheme with traditional, geometric 3D reconstruction evaluation schemes, show in a user study how our scheme relates to human judgment of the quality of novel view predictions, and present a new, general IBMR benchmark based on our evaluation scheme.