Robust Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling

摘要

Sketching provides an intuitive user interface for communicating free formshapes. While human observers can easily envision the shapes they intendto communicate, replicating this process algorithmically requires resolvingnumerous ambiguities. Existing sketch-based modeling methods resolvethese ambiguities by either relying on expensive user annotations or byrestricting the modeled shapes to specific narrow categories. We present anapproach for modeling generic freeform 3D surfaces from sparse, expressive2D sketches that overcomes both limitations by incorporating convolutionneural networks (CNN) into the sketch processing workflow.Given a 2D sketch of a 3D surface, we use CNNs to infer the depth andnormal maps representing the surface. To combat ambiguity we introducean intermediate CNN layer that models the dense curvature direction, orflow, field of the surface, and produce an additional output confidence map along with depth and normal. The flow field guides our subsequent surfacereconstruction for improved regularity; the confidence map trainedunsupervised measures ambiguity and provides a robust estimator for datafitting. To reduce ambiguities in input sketches users can refine their inputby providing optional depth values at sparse points and curvature hintsfor strokes. Our CNN is trained on a large dataset generated by renderingsketches of various 3D shapes using non-photo-realistic line rendering (NPR)method that mimics human sketching of free-form shapes. We use the CNNmodel to process both single- and multi-view sketches. Using our multi-viewframework users progressively complete the shape by sketching in differentviews, generating complete closed shapes. For each new view, the modelingis assisted by partial sketches and depth cues provided by surfaces generatedin earlier views. The partial surfaces are fused into a complete shape usingpredicted confidence levels as weights.We validate our approach, compare it with previous methods and alternativestructures, and evaluate its performance with various modelingtasks. The results demonstrate our method is a new approach for efficientlymodeling freeform shapes with succinct but expressive 2D sketches.