Interactive Conversational Head Generation

by   Mohan Zhou, et al.

We introduce a new conversation head generation benchmark for synthesizing behaviors of a single interlocutor in a face-to-face conversation. The capability to automatically synthesize interlocutors which can participate in long and multi-turn conversations is vital and offer benefits for various applications, including digital humans, virtual agents, and social robots. While existing research primarily focuses on talking head generation (one-way interaction), hindering the ability to create a digital human for conversation (two-way) interaction due to the absence of listening and interaction parts. In this work, we construct two datasets to address this issue, “ViCo” for independent talking and listening head generation tasks at the sentence level, and “ViCo-X”, for synthesizing interlocutors in multi-turn conversational scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the interaction modeling during the face-to-face conversation: 1) responsive listening head generation making listeners respond actively to the speaker with non-verbal signals, 2) expressive talking head generation guiding speakers to be aware of listeners' behaviors, and 3) conversational head generation to integrate the talking/listening ability in one interlocutor. Along with the datasets, we also propose corresponding baseline solutions to the three aforementioned tasks. Experimental results show that our baseline method could generate responsive and vivid agents that can collaborate with real person to fulfil the whole conversation. Project page:


page 2

page 5

page 9


Responsive Listening Head Generation: A Benchmark Dataset and Baseline

Responsive listening during face-to-face conversations is a critical ele...

Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer

This paper reports our solution for MultiMedia ViCo 2022 Conversational ...

Towards automatic estimation of conversation floors within F-formations

The detection of free-standing conversing groups has received significan...

Learning and Evaluating Human Preferences for Conversational Head Generation

A reliable and comprehensive evaluation metric that aligns with manual p...

Head-mounted Displays, Smartphones, or Smartwatches? – Augmenting Conversations with Digital Representation of Self

Technologies that augment face-to-face interactions with a digital sense...

Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglement

Over the last few decades, many aspects of human life have been enhanced...

Keeping Up Appearances: Computational Modeling of Face Acts in Persuasion Oriented Discussions

The notion of face refers to the public self-image of an individual that...

Please sign up or login with your details

Forgot password? Click here to reset