Scroll to enter
Morpheus icon

Deep Sketch-Based 3D Modeling: A Survey

Alberto Tono*, Jiajun Wu, Gordon Wetzstein, Iro Armeni, Hariharan Subramonyam, James Landay, Martin Fischer
Stanford University
Paper Supplementary Code arXiv

Figure 1. Illustrates our design space MORPHEUS organized around the Input-Model-Output (IMO) framework used to survey deep sketch-based 3D modeling (DS-3DM) methods. The Input is categorized into the amount of sketches, their type of viewpoint, and sketch styles with their respective subcategories. The Model is categorized based on the architecture type: neural networks, implicit functions, diffusion models, transformers, differentiable renderers, and foundation models. The Output is categorized by the 3D output compositions, such as part-based semantic, number of options, and geometric topology as well as their corresponding subcategories. The background displays the qualitative and quantitative metrics used to evaluate DS-3DM. The figure encapsulates the paper’s goal: enabling a user-controlled process driven by human-centric metrics to support informed design.

Abstract

In the past decade, advances in artificial intelligence have revolutionized sketch-based 3D modeling, leading to a new paradigm known as Deep Sketch-Based 3D Modeling (DS-3DM). DS-3DM offers data-driven methods that address the long-standing challenges of sketch abstraction and ambiguity. DS-3DM keeps humans at the center of the creative process by enhancing the flexibility, usability, faithfulness, and adaptability of sketch-based 3D modeling interfaces. This paper contributes a comprehensive survey of the latest DS-3DM within a novel design space: MORPHEUS. Built upon the Input-Model-Output (IMO) framework, MORPHEUS categorizes Models outputting Options of 3D Representations and Parts, derived from Human inputs (varying in quantity and modality), and Evaluated across diverse User views and Styles. Throughout MORPHEUS we highlight limitations and identify opportunities for interdisciplinary research in Computer Vision, Computer Graphics, and Human-Computer Interaction, revealing a need for controllability and information-rich outputs. These opportunities align design processes more closely with user intent, responding to the growing importance of user-centered approaches.

BibTeX

BibTex Code Here