DreamWaltz - Make a Scene with Complex 3DAnimatable Avatars

Aug 25, 2026

Upload by Eliana

DreamWaltz提出一种从文本引导生成并驱动复杂三维可动画头像的新框架。针对现有文本到三维方法难以同时获得高质量、可动画结构一致性的问题，方法引入3D一致、具遮挡感知的Score Distillation Sampling，并以规范姿态优化隐式神经表示。通过3D感知骨架条件提供视角对齐监督，减少伪影与多面问题。动画阶段利用扩散模型在多姿态图像先验上学习可动画头像表征，实现无需重训练的任意姿态变形与场景合成。

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Introduction

Problem and motivation

Proposed framework overview

Contributions

DreamWaltz: Make a Scene with Complex 3DAnimatable Avatars Yukun Huang 1 ,2∗†, Jianan Wang 1∗‡, Ailing Zeng 1 , He Cao 1 , Xianbiao Qi 1 , Yukai Shi 1 , Zheng-Jun Zha2 , Lei Zhang 1 1International Digital Economy Academy (IDEA) 2University of Science and Technology of China (a) Princess Elsa in different poses. (b) Uzumaki Naruto in dance moves. (c) Woody dances with Stormtrooper. (d) Woody sits on: (Left) a chair; (Right) a chair made of cheese. Figure 1: DreamWaltz is a text-to-3D-avatar generation framework, which can (a, b) create complex 3D animatable avatars from texts,(c, d) ready for 3D scene composition with diverse interactions. Abstract We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. While recent methods have shown encouraging results for text-to-3D generation of common objects, creating high-quality and animatable 3D avatars remains challenging. To create high-quality 3D avatars, DreamWaltz proposes 3D-consistent occlusionaware Score Distillation Sampling (SDS) to optimize implicit neural representations with canonical poses. It provides view-aligned supervision via 3D-aware skeleton conditioning which enables complex avatar generation without artifacts and multiple faces. For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses, which could animate complex non-rigged avatars given arbitrary poses without retraining. Extensive evaluations demonstrate that DreamWaltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including avatar-avatar, avatar-object and avatar-scene interactions. See [https://dreamwaltz3d.github.io/](https://dreamwaltz3d.github.io/ for)[ for](https://dreamwaltz3d.github.io/ for)[ ](https://dreamwaltz3d.github.io/ for)more vivid 3D avatar and animation results. 1 Introduction The creation and animation of 3D digital avatars are essential for various applications, including film and cartoon production, video game design, and immersive media such as AR and VR. However, ∗Equal contribution. †Work done during an internship at IDEA. ‡Corresponding author. 37th Conference on Neural Information Processing Systems (NeurIPS 2023) . traditional techniques for constructing such intricate 3D models are costly and time-consuming, requiring thousands of hours from skilled artists with extensive aesthetics and 3D modeling knowledge. In this work, we seek a solution for 3D avatar generation that satisfies the following desiderata: (1) easily controllable over avatar properties through textual descriptions; (2) capable of producing high-quality and diverse 3D avatars with complex shapes and appearances; (3) the generated avatars should be ready for animation and scene composition with diverse interactions. The advancement of deep learning methods has enabled promising methods which can reconstruct 3D human models from monocular images [36, 45] or videos [44, 14, 46, 41, 12, 30] . Nonetheless, these methods rely heavily on the strong visual priors from image/video and human body geometry, making them unsuitable for generating creative avatars that can take on complex and imaginative shapes or appearances. Recently, integrating 2D generative models into 3D modeling [31, 18, 10] has gained significant attention to make 3D digitization more accessible, reducing the dependency on extensive 3D datasets. However, creating animatable 3D avatars remains challenging: Firstly, avatars often require intricate and complex details for their appearance (e.g., loose cloth, diverse hair, and different accessories); secondly, avatars have articulated structures where each body part is able to assume various poses in a coordi

FAQs

DreamWaltz解决了文本生成3D头像中的哪些核心挑战？

它针对复杂外观细节、可关节姿态一致性以及不同姿态下形状与纹理变化带来的动画难题，提出能生成可动画头像的框架。

DreamWaltz在头像生成阶段使用了什么关键方法？

生成阶段采用3D-consistent且aware occlusion的Score Distillation Sampling（SDS），并结合规范姿态与3D感知骨架条件进行视角对齐监督。

DreamWaltz如何实现无需重训练的复杂非rigged头像动画？

通过在姿态先验条件下训练可动画NeRF表征，使生成的头像能在测试时变形到任意姿态，从而实现真实动画而不需要重新训练。

Aug 25, 2026

Upload by Eliana