MagicClay: Sculpting Meshes With Generative Neural Fields

Amir Barda^1*, Vladimir G. Kim², Noam Aigerman^2,3, Amit H. Bermano¹, Thibault Groueix²

¹Tel Aviv University, ²Adobe Research, ³Université de Montréal
^*Research performed during Adobe Internship

Paper Supplementary Code arXiv

"iron man" "a minotaur" "a bear" "a cowboy riding a horse" "an astronaut riding a horse" "a centaur" "a man holding a knights sword" "a man holding a wizard's staff" "a mermaid" "man with angel wings" "man with airplane wings" "man with bat wings"

Our pipeline's input (shown in the upper left corner) and output are a mesh. Only the yellow region (defined by the user) is allowed to evolve, with unmarked parts of the mesh remain unchanged. We also naturally support the output of textured meshes (see supplementary for more information).

Abstract

Recent developments in neural fields have brought phenomenal capabilities to the field of shape generation, but they lack crucial properties, such as incremental control — a fundamental requirement for artistic work. Triangular meshes, on the other hand, are the representation of choice for most geometry related tasks, offering efficiency and intuitive control, but do not lend themselves to neural optimization. To support downstream tasks, previous art typically proposes a two-step approach, where first a shape is generated using neural fields, and then a mesh is extracted for further processing. Instead, in this paper we introduce a hybrid approach that maintains both a mesh and a Signed Distance Field (SDF) representations consistently. Using this representation, we introduce MagicClay — an artist friendly tool for sculpting regions of a mesh according to textual prompts while keeping other regions untouched.

MagicClay uses a hybrid mesh-SDF representation which allows for non-distructive local editing of the mesh

Application: sequential editing of the same mesh

initial mesh + "devil horns" + "elf ears" + "pig snout"

Application: preserving properties of the original triangulation, such as vertex groups for animations (See supplementary for more information).

BibTeX

@misc{barda2024magicclay,
      title={MagicClay: Sculpting Meshes With Generative Neural Fields}, 
      author={Amir Barda and Vladimir G. Kim and Noam Aigerman and Amit H. Bermano and Thibault Groueix},
      year={2024},
      eprint={2403.02460},
      archivePrefix={arXiv},
      primaryClass={cs.GR}
}