ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis

摘要

Manually authoring 3D shapes is difficult and time consuming; generativemodels of 3D shapes offer compelling alternatives. Procedural representationsare one such possibility: they offer high-quality and editable resultsbut are difficult to author and often produce outputs with limited diversity.On the other extreme are deep generative models: given enough data, theycan learn to generate any class of shape but their outputs have artifacts andthe representation is not editable.In this paper, we take a step towards achieving the best of both worldsfor novel 3D shape synthesis. First, we propose ShapeAssembly, a domainspecific“assembly-language” for 3D shape structures. ShapeAssembly programsconstruct shape structures by declaring cuboid part proxies and attachingthem to one another, in a hierarchical and symmetrical fashion. ShapeAssemblyfunctions are parameterized with continuous free variables,so that one program structure is able to capture a family of related shapes.We show how to extract ShapeAssembly programs from existing shapestructures in the PartNet dataset. Then, we train a deep generative model, ahierarchical sequence VAE, that learns to write novel ShapeAssembly programs.Our approach leverages the strengths of each representation: theprogram captures the subset of shape variability that is interpretable andeditable, and the deep generative model captures variability and correlationsacross shape collections that is hard to express procedurally.We evaluate our approach by comparing the shapes output by our generatedprograms to those from other recent shape structure synthesis models.We find that our generated shapes are more plausible and physically-validthan those of other methods. Additionally, we assess the latent spaces ofthese models, and find that ours is better structured and produces smootherinterpolations. As an application, we use our generative model and differentiableprogram interpreter to infer and fit shape programs to unstructuredgeometry, such as point clouds.