AI正在发生 未来正在更新MeiGen-MultiTalk
> We present **MultiTalk**, an open-source audio-driven multi-person conversational video generation model with the state-of-the-art lip synchronization accuracy. > Key features: > - 💬 Realistic Conversations - Supports single & multi-person generation > - 👥 Interactive Character Control - Direct virtual humans via prompts > - 🎤 Generalization Performances - Supports the generation of cartoon character and singing > - 📺 Resolution Flexibility: 480p & 720p output at arbitrary aspect ratios > - ⏱️ **Long Video Generation**: Support video generation up to 15 seconds This repository hosts the model weights for **MultiTalk**. For installation, usage instructions, and further documentation, please visit our GitHub repository. We propose a novel framework, MultiTalk, for audio-driven multi-person conversational video generation. We investigate several schemes for audio injection and introduce the Label Rotary Position Embedding (L-RoPE) method. By assigning identical labels to audio embeddings and video latents, it effectively activates specific regions within the audio cross-attention map, thereby resolving incorrect binding issues. To localize the region of the specified person, we introduce the adaptive person localization by computing the similarity between the features of the given region of a person in the reference image and all the features of the whole video.
译文尚未就绪，显示来源原文
> We present **MultiTalk**, an open-source audio-driven multi-person conversational video generation model with the state-of-the-art lip synchronization accuracy. > Key features: > - 💬 Realistic Conversations - Supports single & multi-person generation > - 👥 Interactive Character Control - Direct virtual humans via prompts > - 🎤 Generalization Performances - Supports the generation of cartoon character and singing > - 📺 Resolution Flexibility: 480p & 720p output at arbitrary aspect ratios > - ⏱️ **Long Video Generation**: Support video generation up to 15 seconds This repository hosts the model weights for **MultiTalk**. For installation, usage instructions, and further documentation, please visit our GitHub repository. We propose a novel framework, MultiTalk, for audio-driven multi-person conversational video generation. We investigate several schemes for audio injection and introduce the Label Rotary Position Embedding (L-RoPE) method. By assigning identical labels to audio embeddings and video latents, it effectively activates specific regions within the audio cross-attention map, thereby resolving incorrect binding issues. To localize the region of the specified person, we introduce the adaptive person localization by computing the similarity between the features of the given region of a person in the reference image and all the features of the whole video.