Creating personalized videos that preserve consistent subject identity across various contexts remains a significant challenge in artificial intelligence. Tencent’s HunyuanCustom, built on the HunyuanVideo platform, introduces a multimodal framework that enables users to generate subject-consistent videos using text, images, audio, and video inputs. Through careful integration of these modalities, the model enhances identity preservation, realism, and alignment with user-specified conditions. Whether it's a girl playing with plush toys or a man interacting with a penguin, HunyuanCustom enables coherent video generation guided by detailed user prompts.

Technical Framework

HunyuanCustom employs several architectural innovations to support flexible and controllable video generation:

1. Image-Text Fusion: A module based on LLaVA integrates identity features from images with text prompts. This enhances multimodal understanding and ensures the generated video reflects both identity and context.
2. Image ID Enhancement: The model concatenates image features across time steps, leveraging HunyuanVideo’s temporal modeling capabilities to maintain identity consistency throughout the video.

Modality-Specific Injection Modules:

1. Audio Integration: The AudioNet module uses hierarchical spatial cross-attention to align audio with character behavior, enabling scenarios such as singing or speaking.
2. Video Integration: A patchify-based feature-alignment network enables conditional injection of video-based prompts, supporting object replacement or addition while maintaining temporal coherence.
3. Decoupled Modality Control: By disentangling image, audio, and video conditions from identity encoding, HunyuanCustom allows for flexible input combinations and enhanced control over content generation.

Applications

HunyuanCustom supports a wide range of applications across single-subject, multi-subject, and audio/video-driven tasks:

1. Single-Subject Video Generation: The model produces coherent and identity-consistent videos for individual subjects.
2. Multi-Subject Video Generation: HunyuanCustom accurately captures interactions between multiple characters in varied settings
3. Audio-Driven Generation: HunyuanCustom can synchronize character behavior with audio input using spatial cross-attention mechanisms. Examples include:
4. Video-Driven Customization: With reference images and source videos, HunyuanCustom enables subject replacement or integration into a pre-existing video scene. This is useful for maintaining visual realism while adapting characters or objects.

Performance Evaluation

Through experiments across various scenarios, HunyuanCustom has demonstrated competitive performance when compared with both open- and closed-source solutions. Evaluation metrics confirm its strengths in:

1. Identity Consistency
2. Text-Video Alignment
3. Realism and Temporal Coherence

While further benchmarking and peer-reviewed results are pending, initial findings suggest that HunyuanCustom is a strong candidate in the space of controllable video generation.