IndexTTS2
bilibili · Emotionally expressive + duration-controlled · Released Sep 8, 2025
IndexTTS2 is the model that made this family go viral internationally — the "AI English dub of Chinese period dramas" videos that circulated on Western social platforms in late 2025 were generated with it. It is the first autoregressive zero-shot TTS model that combines precise duration control with natural, free-running generation, and it decouples emotion from speaker identity so that timbre and emotion can be specified independently. The GitHub repo has passed 10k stars, and weights plus inference code are released.
- Paper
- arXiv:2506.21619
- Architecture
- T2S + S2M + BigVGANv2
- Emotion data
- 55K samples / 135 h emotional speech
- Cloning input
- ~5 s audio, >85% similarity
- VRAM
- ~15 GB
- RTF (RTX 4090)
- 0.3257 fp16 overall
- Community
- Discord + OpenAI-compatible API forks
- Contact for commercial
- [email protected]
Architecture — a three-stage cascade
- Text-to-Semantic (T2S) — an LLM-style autoregressive transformer that takes source text, a style prompt, a timbre prompt and an optional target token count, and emits semantic tokens.
- Semantic-to-Mel (S2M) — a flow-matching module that predicts the mel spectrogram from semantic tokens plus timbre conditioning. Replacing discrete acoustic tokens with flow matching is what preserves fine-grained detail under strong emotion.
- BigVGANv2 vocoder — converts mel spectrograms to waveform.
Innovation 1 — Duration control inside an autoregressive model
Autoregressive TTS is inherently token-by-token, which makes it hard to hit a target duration — a real problem for dubbing, where audio must match picture. IndexTTS2 solves this with a token-count constraint:
- A dedicated duration embedding injects the requested token count into T2S, and the model aligns positional information to that budget.
- Training randomly samples duration-scaling tasks (e.g. 0.75×, 1.25×) so the model stays fluent and emotionally coherent at any requested length.
- Measured token-count error rate: below 0.03%, often below 0.02%.
- Two operating modes: constrained (specify token count → exact duration) and free-running (no token count → faithfully reproduces the prompt's prosody).
Innovation 2 — Emotion/timbre disentanglement, four control modalities
| Control modality | How it works |
|---|---|
| Single audio reference | One clip supplies both timbre and emotion (classic zero-shot cloning) |
| Separate emotion reference | Style prompt can come from a different speaker than the timbre prompt — even a different language |
| Emotion vector | 8 dimensions — happy, angry, sad, afraid, disgusted, melancholic, surprised, calm — with adjustable weights and random sampling |
| Text description ("soft instruction") | A fine-tuned Qwen3 maps natural-language descriptions ("say it with a hint of grievance", "You scared me to death!") into emotion guidance |
The disentanglement is enforced architecturally with a Gradient Reversal Layer (GRL) and separate speaker/emotion perceivers, so the model cannot cheat by leaking identity into the emotion pathway.
Innovation 3 — Clarity under extreme emotion
Shouting and sobbing are where expressive TTS usually falls apart — intelligibility drops. IndexTTS2 incorporates GPT latent representations plus a novel three-stage training paradigm to stabilise generation. Measured WER during shouting: 1.883%, i.e. expressiveness without sacrificing intelligibility.
Reported results
| Metric | Value | Context |
|---|---|---|
| Emotion similarity (ES) | 0.887 | Beats other SOTA zero-shot models |
| Emotion MOS (EMOS) | 4.22 | Natural, well-modulated delivery |
| WER under strong emotion | 1.883% | Shouting / crying scenarios |
| Duration control error | <0.03% | 0.75×–1.25× speed tests |
Real-world inference speed (RTX 4090, kv_cache=True)
| Input length | v2.0 fp16 | v2.0 fp32 |
|---|---|---|
| 7 characters | 0.4004 | 0.3748 |
| 16 characters | 0.3322 | 0.3389 |
| 28 characters | 0.3257 | 0.3480 |
| 80 characters | 0.3229 | 0.3754 |
| 200 characters | 0.3244 | 0.3990 |
| Overall | 0.3257 | 0.3748 |
Independent Docker-based testing on an NVIDIA L40S (80 test cases across 4 versions × 4 scenarios × 5 runs, 100% success rate) reported end-to-end latencies of ~6.4 s (Chinese short), ~28 s (Chinese long), ~7.6 s (English short), ~35.4 s (English long) for the production image. The v2.1-cuda build was fastest for Chinese; v2.0-production was most stable for English.
/v1/audio/speech style) and integration with Voxta, which makes it comparatively easy to drop into an existing pipeline.Use cases: when to pick 2
| Scenario | How to run it | Watch out for |
|---|---|---|
| Anime, motion-comic and short-form emotional dubbing | Control emotion with the 8-dimension vector or a natural-language instruction; timbre and emotion are decoupled, so A's voice can carry B's emotion | ~15 GB VRAM — the highest in this group |
| Emotional passages in audiobooks | Attach emotion instructions to dialogue, fall back to "calm" for narration | Long-form still drifts; you must chunk with overlap validation (see 06.3) |
| Podcasts and two-host shows | Lock one reference clip plus one emotion instruction per host so episodes stay consistent | AI disclosure duties in ad and sponsor segments are tightening (see 06.5) |
| Re-voicing an existing performance | Keep the timbre, swap only the emotion vector — far cheaper than a re-record | Test articulation under extreme emotion (sobbing, shouting); WER rises |
| Video dubbing (first-generation approach) | Duration control holds token-count error under 0.03%, keeping audio aligned to picture | For continuous speed adjustment, 2.5's 0.5×–2.0× is the better tool |
Verdict for international users
Use it if you are doing AI dubbing, audiobooks, character voice work, or any pipeline where audio must match a fixed timeline, or where you need to re-voice a performance into another emotion.
Watch out for the ~15 GB VRAM requirement and the fact that English is a second-language target for this model family — always A/B it against a Western model (Chatterbox, XTTS-v2, F5-TTS) on your own English script before committing.
Where it sits in the head-to-head data
IndexTTS2 also did not take part in CV3-Eval — that table covers the 2.5 generation. Its head-to-head evidence comes mainly from measured RTF on an RTX 4090 (see "Real-world inference speed" above).
| Version | Precision | Overall RTF | Versus 2.5 |
|---|---|---|---|
| IndexTTS2 | fp16 | 0.3257 | ~1.58× slower |
| IndexTTS2 | fp32 | 0.3748 | ~1.82× slower |
| IndexTTS2.5 | bf16 | 0.2065 | Baseline |
An RTF of 0.3257 means roughly 3.1× real-time: one million characters (~18 hours of audio) takes about 5.9 GPU-hours.
What it costs to run yourself
| Setup | RTF | GPU-hours / M chars | On-demand RTX 4090 ($0.74/h) | On-demand A100 ($1.59/h) |
|---|---|---|---|---|
| IndexTTS2 (fp16) | 0.3257 | 5.9 h | ~$4.34 | ~$9.32 |
| Reference: IndexTTS2.5 (bf16) | 0.2065 | 3.7 h | ~$2.75 | ~$5.91 |
Put that next to the commercial API price list:
| Option | Cost / M chars | Versus IndexTTS2 |
|---|---|---|
| ElevenLabs v3 | $60–180 | ~14–41× |
| Cartesia Sonic 3 | $20–39 | ~4.6–9× |
OpenAI tts-1 | $15 | ~3.5× |
| Google / Polly Standard | $4 | ~0.9× — cheaper than self-hosting |
This table also excludes engineering cost: deployment, monitoring, retry logic, chunk stitching, failed re-runs. That is engineer time, and it usually costs an order of magnitude more than the GPU bill. Full model is in 06.4.
Licence and disclosure gate
| Item | Status |
|---|---|
| Code licence | Open source |
| Weights licence | Index Model License |
| Commercial use | Terms must be reviewed; the commercial contact published by the authors is [email protected] |
| Default watermark | Not stated → plan on the assumption there is none |
| Data residency | Runs fully offline |
If not this, then what
- Continuously adjustable speaking rate for dubbing → IndexTTS2.5 (0.5×–2.0× continuous; 2's duration control is fixed steps).
- You don't need emotion control → IndexTTS 1 is lighter, though mind the English version problem.
- The licence has to be clean → VoxCPM2.
- You just want "sounds better" → start with the blind-listening Elo table in 06.1; Step Audio EditX (Apache-2.0) is the highest-ranked open model you can actually ship.
Ready to create with IndexTTS2?
Open Text to Speech with this engine selected and start with your own voice model.