# CoVoMix2: Advancing Zero-Shot Dialogue Generation
with Fully Non-Autoregressive Flow Matching

#####

## Abstract

Generating natural-sounding, multi-speaker dialogue is crucial for applications such as podcast creation, virtual agents, and multimedia content generation. However, existing systems struggle to maintain speaker consistency, model overlapping speech, and synthesize coherent conversations efficiently. In this paper, we introduce CoVoMix2, a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. CoVoMix2 directly predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model, eliminating the reliance on intermediate token representations. To better capture realistic conversational dynamics, we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies. Our approach achieves state-of-the-art performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed. Notably, CoVoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and precise timing control, demonstrating strong generalizability to real-world speech generation scenarios.

## System Overview

Figure 1: System Architecture Overview

## Samples

All prompt audios (except for the celebrities) are from the demo page of MoonCast and Soundstorm.

- Zero-shot Dialogue Generation Samples

Transcription | Prompts | Ours | MoonCast | Sesame
--- | --- | --- | --- | ---

- Celebrities Samples

The first pair of celebrity is Donald Trump and Benedict Cumberbatch
The second pair of celebrity is Lex Fridman and Elon Musk

Transcription | Prompts | Ours | MoonCast | Sesame
--- | --- | --- | --- | ---

- Cross-lingual Samples

Transcription | Prompts | Ours
--- | --- | ---

- Overlapping Samples

Transcription | Prompts | Ours
--- | --- | ---