Meta’s Fundamental AI Research (FAIR) team, in collaboration with its Codec Avatars and Core AI labs, has introduced a family of research-focused Dyadic Motion Models that generate realistic facial expressions and body gestures from audio and visual inputs of two-person conversations. These models aim to enhance virtual communication by enabling avatars to mimic human-like behaviors, with potential applications in virtual reality (VR) and augmented reality (AR) telepresence.

Modeling the Nuances of Human Dialogue

Human conversations involve a rich interplay of speech, tone, and body language, such as nods, smiles, or shifts in posture. Meta’s Dyadic Motion Models are designed to capture these dynamics by processing audio from two speakers to produce expressive gestures, active listening cues, and turn-taking behaviors. When visual inputs from one speaker are available, the models can also incorporate visual synchrony, such as smile mirroring or shared gaze, to enhance realism.

These research models can animate avatars in 2D video or as 3D Codec Avatars, offering potential for more immersive virtual interactions. For example, they could visualize a podcast recording by generating gestures and expressions that align with the speakers’ audio. The models include controllability parameters, often guided by speech from large language models (LLMs), allowing researchers to adjust avatar expressiveness for specific contexts, such as emphasizing active listening or speaking behaviors.

The Seamless Interaction Dataset: A Resource for Social AI

The models are powered by the Seamless Interaction Dataset, a comprehensive collection of over 4,000 hours of in-person, two-person interactions involving more than 4,000 participants. This dataset, now publicly available, captures diverse conversational dynamics, including 1,300 naturalistic and improvised prompts, with metadata on participant relationships and personalities.

To ensure authenticity, all interactions were recorded in person. Approximately one-third of the dataset features familiar pairs, such as friends or colleagues, whose interactions reflect natural rapport. Another third includes scripted conversations by professional actors portraying varied emotions and roles, capturing nuanced behaviors like surprise or disagreement. This mix ensures the dataset represents a wide range of human interactions.

Meta is sharing the dataset and a technical report detailing its methodology to support the research community. They’ve also proposed an evaluation framework with objective and subjective metrics to assess audiovisual behavioral models, focusing on speaking, listening, and turn-taking. This open approach aims to advance social AI for applications like virtual agents, telepresence, and video analysis.

Commitment to Privacy and Ethics

Meta prioritized privacy and ethics in creating the Seamless Interaction Dataset. Participants consented to recordings and were advised to avoid sharing personal information. About one-third of conversations were scripted to minimize sensitive disclosures. A multi-stage quality assurance process, combining human reviews and AI-driven analysis, removed any flagged content to protect participant anonymity.

To ensure transparency, Meta uses AudioSeal and VideoSeal to watermark content generated by its models, allowing verification of authenticity. These measures reflect a commitment to responsible AI development.

A Step Toward Human-Centric Virtual Spaces

Meta’s Dyadic Motion Models and Seamless Interaction Dataset represent a significant advancement in understanding and replicating human conversational behaviors. By enabling avatars to reflect the subtleties of face-to-face dialogue, these tools lay the groundwork for more natural virtual interactions in VR, AR, and beyond.

Researchers can access the dataset on GitHub or HuggingFace (specific links available via Meta’s Seamless Interaction website) and explore the technical report for detailed insights. As Meta and the broader research community build on this work, these advancements could enhance how we connect, collaborate, and engage in virtual environments, making digital interactions feel more like real-world conversations.