VoxCPM2
ModelBest / OpenBMB / Tsinghua HCSI · Tokenizer-free · Apache-2.0 · Apr 2026
VoxCPM2 is the most internationally friendly model in this set: Apache-2.0 for both code and weights, clean English documentation, and a design decision that sidesteps a common TTS problem. It is tokenizer-free — it generates directly in continuous space rather than quantising audio into discrete tokens first. The rationale: quantisation is lossy, and accent, emotional micro-dynamics, breath and pause rhythm are exactly the details that get flattened by it. The model is 2 B parameters, built on ModelBest's MiniCPM-4 backbone, trained on 2M+ hours of multilingual speech.
- Architecture
- LocEnc → TSLM → RALM → LocDiT
- Backbone
- MiniCPM-4, 2 B total
- Languages
- 30 + 9 Chinese dialects
- Audio VAE
- AudioVAE V2 (16 kHz in → 48 kHz out)
- LM token rate
- 6.25 Hz, max 8192 tokens
- VRAM
- ~8 GB
- RTF (RTX 4090)
- ~0.30 / ~0.13 with Nano-VLLM
- License
- Apache-2.0 (commercial OK)
The four-stage architecture
- LocEnc (Local Encoder) — extracts features from input text and reference audio, producing conditioning for the two language models.
- TSLM (Text–Semantic LM) — "semantic planning": maps text to a coarse speech-semantics representation. This is where content boundaries and the prosodic frame are decided.
- RALM (Residual Acoustic LM) — finer-grained acoustic modelling on top of TSLM's output: timbre, emotional detail and speaker characteristics are added here.
- LocDiT (Local Diffusion Transformer) — flow-matching local diffusion producing continuous audio latents, decoded by AudioVAE V2 into 48 kHz audio.
AudioVAE V2 is asymmetric: the encoder accepts 16 kHz input while the decoder emits 48 kHz, with super-resolution built in — no external upsampler needed. That 48 kHz output is the highest sample rate in this group.
Four usage modes
| Mode | Input | What it gives you |
|---|---|---|
| Basic TTS | Text only | No reference audio and no language tag — the model detects the language itself. Type Chinese, get Chinese; type Thai, get Thai. |
| Voice Design | Natural-language description | Prefix the text with a description in parentheses, e.g. (young woman, warm and gentle voice). Creates an entirely new voice with no reference audio. |
| Controllable Cloning | Reference audio + style instruction | Clone the timbre while steering emotion, pace and delivery ("a little faster, cheerful tone"). The timbre base stays intact. |
| Ultimate Cloning | Reference audio + its transcript | Audio-continuation style cloning — the reference becomes the opening segment. Best reproduction of accent, breathing habits and pause rhythm. Passing the same clip to both reference_wav_path and prompt_wav_path maximises similarity. |
Context-aware synthesis
A capability that is easy to overlook: VoxCPM reads the text and infers appropriate emotion and prosody rather than mechanically converting characters to audio. The same sentence ending in an exclamation mark versus a full stop will get different pacing, energy and stress. Sports commentary, arguments and crying monologues come out with natural prosody rather than the flat "stable but emotionless" delivery typical of older TTS. This comes from TSLM's semantic modelling and the MiniCPM-4 text understanding underneath it.
Benchmarks
| Metric | Value | Notes |
|---|---|---|
| Chinese CER | 0.97% | Public eval set |
| English WER | 1.84% | Public eval set |
| RTF (RTX 4090) | ~0.30 | Standard mode |
| RTF with Nano-VLLM | ~0.13 | Approaching low-latency real-time |
| VRAM | ~8 GB | bf16 |
Reported as state-of-the-art or competitive on Seed-TTS-eval, CV3-eval, InstructTTSEval and the MiniMax multilingual test. Caveat: most numbers are self-reported; independent third-party reproductions are still limited.
Languages
30 languages: Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, Vietnamese.
9 Chinese dialects: Sichuanese, Cantonese, Wu (Shanghainese), Northeastern, Henanese, Shaanxi, Shandong, Tianjin, Hokkien.
Fine-tuning & deployment
- Full SFT and LoRA fine-tuning supported with as little as 5–10 minutes of audio — LoRA is the recommended path. This is the only model in this group that documents a practical personal-voice training workflow.
- Requirements: Python ≥ 3.10, PyTorch ≥ 2.5.0, CUDA ≥ 12.0. Install with
pip install voxcpm. - ModelScope download is available for users in regions where Hugging Face is slow.
- Production serving via Nano-VLLM or vLLM-Omni. Local web demo:
python app.py --port 8808. - Key knobs:
cfg_value(classifier-free guidance strength — how closely output adheres to the prompt) andinference_timesteps(diffusion steps; default 10 is fine for most cases). - Ecosystem traction: 327k downloads in the last month, 21 adapters, 28 community fine-tunes and 10 quantizations on Hugging Face.
Version history
| When | Version | What changed |
|---|---|---|
| Sep 2025 | VoxCPM-0.5B | First release; hit #1 on Hugging Face Trending |
| Dec 2025 | VoxCPM1.5 (~800 M) | SFT + LoRA fine-tuning, 44.1 kHz output, #1 on GitHub Trending; spawned ComfyUI plugins, ONNX exports and a Rust reimplementation |
| Apr 2026 | VoxCPM2 (2 B) | ~2.5× parameters; 2 → 30 languages + 9 dialects; 48 kHz output; Voice Design and Controllable Cloning added; AudioVAE V2 |
Use cases: the only licence-clean option here
| Scenario | How to run it | Watch out for |
|---|---|---|
| Self-hosted public-sector, hospital and financial deployment | Runs fully offline so audio never leaves the network; 48 kHz output with no external upsampling | Plan ~6–8 weeks to production — see the cost table below |
| Online courses and multilingual training | 30 languages plus 9 Chinese dialects, with no language tags required | The European Accessibility Act has turned accessibility into a procurement term — see below |
| Creator voice cloning and personal IP | LoRA fine-tuning on 5–10 minutes of audio bakes your timbre into the weights | Cloning yourself or a released voice is safe; cloning a real person raises personality rights, a separate question from the licence |
| Ad and marketing A/B variants | Batch-generate many variants of one script; marginal cost is near zero when self-hosted | Platform and FTC AI-labelling policies are tightening |
| High-fidelity content delivery | 16 kHz reference audio goes through AudioVAE V2 straight to 48 kHz output | 2B parameters — slower than IndexTTS2.5 |
| Voice Design (no reference audio) | Describe gender, age and pitch in natural language to create a new timbre | Output carries no watermark — add your own where disclosure is required |
Self-hosted deployment: how the maths works
| Item | Typical figure |
|---|---|
| Time to production, self-hosted voice stack | ~6–8 weeks, kickoff to first production call |
| Typical hardware | L40S for ASR + LLM, L4 for streaming TTS; a single dual-GPU box suffices |
| Warm end-to-end latency | ~0.3 s (local inference, no public-internet round trip) |
| Cost structure | Hosted APIs €0.05–0.40 / minute; self-hosted fixed cost from ~€800 / month |
| Break-even | From roughly 50,000 minutes / month self-hosting wins; below that you are buying data sovereignty, not savings |
Publicly reported production outcomes (third-party disclosures, for reference only):
- A European fintech: a 10× reduction in mean time to resolution across support.
- A tier-1 Gulf telecom: cost per contact down 58%; 81% containment on prepaid enquiries with no human handoff; average handle time on outbound collections compressed from 4.2 to 2.1 minutes; CSAT on AI-handled calls 4.3/5 against a 4.1/5 human baseline.
Read-aloud accessibility: get the direction right
WCAG does not require you to provide text-to-speech. It requires that your content be correctly parseable by the assistive technology users already bring — JAWS, NVDA, VoiceOver. What read-aloud genuinely serves is the large middle group: readers with dyslexia, low vision, older users, second-language readers — people poorly served by a full screen reader but still struggling with long text. It is a product feature, not a compliance checkbox.
- SC 1.4.2: audio that autoplays for more than 3 seconds needs a pause/stop control — so read-aloud must never autoplay.
- SC 2.1.1 / 2.4.7 / 2.4.11: every control keyboard-reachable, focus visible, focus never obscured.
- SC 3.1.1 / 3.1.2: language changes must be marked explicitly — in engine terms, SSML
langswitching. - SC 4.1.3: use ARIA live regions so read-aloud keeps up with form errors and loading states.
Cloning: three rules of discipline
Source audio
Record 10 s–5 min of clean single-speaker audio: no music bed, no reverb, no second voice.
5–10 minutes is enough for LoRA fine-tuning to bake your timbre into the weights and reduce run-to-run variance.
Consistency
Lock one reference clip, one seed and one style instruction so episodes and issues never drift.
Voice Design is stochastic — generate one to three candidates before you commit.
Consent
Cloning yourself, or someone who signed a written release, is the safe zone.
Cloning a real person without consent can trigger personality rights, publicity rights and biometric-privacy claims — a completely separate question from the model licence.
Verdict for international users
Use it if you want a commercially clean licence, 48 kHz output quality, fine-tuning on your own voice with minimal data, and a straightforward local deployment on ~8 GB VRAM.
Watch out for the 2 B parameter footprint (slower than IndexTTS2.5 and OmniVoice) and the stochastic nature of Voice Design — it is a creative tool, not a deterministic generator.
Where it sits in the head-to-head data
| Model | Params | Chinese WER / SS | English WER / SS | Spanish WER / SS |
|---|---|---|---|---|
| VoxCPM2 | 2 B | 3.88 / 74.99 | 5.13 / 71.57 | 5.49 / 74.67 |
| CosyVoice3-0.5B | 0.5 B | 3.84 / 80.01 | 4.88 / 74.16 | 4.04 / 78.85 |
| IndexTTS2.5-RL | 0.8 B | 3.93 / 77.92 | 3.89 / 67.79 | 3.33 / 76.68 |
| MOSS-TTS-v1.5 | 8 B | 4.02 / 72.68 | 4.45 / 67.46 | 3.83 / 71.75 |
| Model | zh→en WER / SS | zh→es WER / SS | zh→ja WER / SS |
|---|---|---|---|
| VoxCPM2 | 4.48 / 64.25 | 16.38 / 64.89 | 11.84 / 71.54 |
| OmniVoice | 3.74 / 64.91 | 5.84 / 62.08 | 9.09 / 69.06 |
| Qwen3-TTS | 5.74 / 63.04 | 5.15 / 68.02 | 36.09 / 65.71 |
| Configuration | RTF | Conditions |
|---|---|---|
| VoxCPM2 (standard) | ~0.30 | RTX 4090 bf16 |
| VoxCPM2 (Nano-VLLM) | ~0.13 | After acceleration |
| IndexTTS2.5 (reference) | 0.2065 | RTX 4090 bf16 |
What it costs to run yourself
| Configuration | RTF | GPU-hours / M chars | On-demand RTX 4090 | On-demand A100 |
|---|---|---|---|---|
| VoxCPM2 (Nano-VLLM) | ~0.13 | 2.3 h | ~$1.73 | ~$3.72 |
| VoxCPM2 (standard) | ~0.30 | 5.4 h | ~$4.00 | ~$8.59 |
Licence and disclosure gate
| Item | Status |
|---|---|
| Code licence | Apache-2.0 |
| Weights licence | Apache-2.0 |
| Commercial use | ✓ Yes — the only fully clean dual licence in this set |
| Default watermark | None (the model card states plainly that output carries no AI watermark) |
| What you must do | Add AudioSeal, SynthID or C2PA yourself if your jurisdiction has a disclosure duty (EU Article 50 / Chinese deep-synthesis rules / platform policy) |
| Content moderation | The model card states no production-grade input filtering is provided; that is on you |
| Data residency | Runs fully offline |
If not this, then what
- Heavy compliance pressure, need a default watermark → Chatterbox (Resemble AI, MIT, PerTh embedded by default, 25 languages).
- Chinese emotional performance plus precise duration → IndexTTS2.5.
- 646 low-resource languages, non-commercial → OmniVoice.
- Very low footprint on CPU or edge → Kokoro 82M (outside this set, Apache-2.0, ~2–3 GB, no cloning).
- Top monolingual Chinese similarity → CosyVoice3-0.5B (check its licence first).
Ready to create with VoxCPM2?
Open Text to Speech with this engine selected and start with your own voice model.