Position:  PhD Candidate

Current Institution:  Stanford University

Abstract:  Using Generative Deep Learning to Create High-Quality Models from 3D Scans

With recent developments in both commodity range sensors as well as mixed reality devices, capturing and creating 3D models of the world around us has become increasingly important. As the world around us lives in a three-dimensional space, such 3D models will not only facilitate, capture, and display for content creation but also provide a basis for fundamental scene understanding from semantic understanding to virtual interactions which must be formulated in 3D for many applications such as augmented or virtual reality. Leveraging data from commodity range sensors to reconstruct 3D scans of a scene has shown significant promise towards 3D model creation of real-world environments. However, the quality of reconstructed 3D scans has yet to reach that of artist-created 3D models — in particular 3D scans always suffer from incompleteness due to occlusions in real-world scenes as well as physical limitations of range sensors. Such incomplete 3D models are both unsuitable visually and, moreover, provide only a limited basis for higher-level scene reasoning (e.g., virtual interactions will not be accurate in unknown or missing regions). My work introduces a generative formulation for the task of scan completion leveraging deep learning techniques to create high-quality complete models from 3D scans. We approach this problem as a conditional generative task; conditioned on an input partial scan we aim to learn ‘part’-wise similarity between scans to infer the complete model. We begin by focusing on the more constrained problem of completing scans of isolated shapes. We then expand upon this to design a generative approach for completion of general 3D scans directly addressing the challenge of varying scene sizes in 3D. This not only provides scan completion at scale producing geometrically complete 3D models but also provides a basis for higher-level scene reasoning such as that required for virtual interactions or physical simulations.

Bio:
Angela Dai is a Ph.D. candidate in Computer Science at Stanford University advised by Pat Hanrahan. Her research focuses on 3D reconstruction and understanding with commodity sensors. She received her Masters degree from Stanford University and her Bachelors degree from Princeton University. She is a recipient of a Stanford Graduate Fellowship.