Personalized Speech-Driven Emotional 3D Talking Face Animation via Context–Emotion Decoupling

When people speak with emotions, facial motions are coupled with both context-driven lip motions and emotion-driven facial expressions. However, existing speech-driven emotional 3D facial generation methods aim to generate directly emotional talking faces from speech, without decoupling these motions. It increases the ambiguity during training regarding whether the current facial motion is related to the context or emotion, hindering effective learning. To decouple them effectively, we introduce the contrastive emotion-face loss that learns the mapping between emotions of speech and expressions in talking lip motion. In addition, since individuals have unique emotional expression styles, we enable personalized emotional expression by utilizing person-specific embeddings. By optimizing the person-specific embeddings, the proposed method can generate emotional talking faces personalized to the target subject. At last, the proposed method generates head motions that align with the context and emotion of speech while maintaining diversity. Through extensive experiments, it is demonstrated that our method achieves 7.18% performance improvement over state-of-the-art methods by animating emotional 3D talking facial animation.