Open-Source TTS Model Deep Dive · 2026
04
OmniVoice
Xiaomi / k2-fsa (Next-gen Kaldi) · 646 languages · Released Mar 31, 2026
OmniVoice comes from k2-fsa, the Next-gen Kaldi group led by Daniel Povey — a name that carries considerable weight in the speech-recognition community. It is the broadest language-coverage zero-shot TTS model available, claiming 646 languages trained on roughly 581,000 hours of audio, and it runs entirely locally. The repo accumulated over 12k GitHub stars within weeks of release, and the weights are only ~3.3 GB.
Read this before you plan a product around OmniVoice: the code is Apache-2.0, but the pre-trained weights are CC-BY-NC 4.0 (non-commercial), a constraint inherited from training data such as the Emilia corpus. Personal and research use is fine; any revenue-generating use is not permitted without separate licensing. This is stated explicitly in the model card and is a hard blocker for commercial deployment.
- Paper
- arXiv:2604.00688
- Backbone
- Qwen3-0.6B (~0.8 B total)
- Languages
- 646
- Training data
- ~581,000 h
- Output
- 24 kHz mono
- Weights size
- ~3.3 GB
- Claimed RTF
- 0.025 on H100 with custom kernels
- Code license
- Apache-2.0
- Weights license
- CC-BY-NC 4.0 (non-commercial)
Architecture — a "diffusion language model-style" non-autoregressive design
Most TTS pipelines are two-stage: text → semantic tokens → acoustic features → waveform, and each hop accumulates error. OmniVoice collapses this to a single stage: text → multi-codebook acoustic tokens → waveform.
- The backbone is a repurposed Qwen3-0.6B language model driven in a diffusion-language-model style: rather than emitting audio left-to-right, it refines the whole sequence in a few parallel denoising steps.
- A bidirectional Transformer jointly models an instruction transcript, a prompt segment and the masked target segment.
- Audio is encoded/decoded by a separate tokenizer derived from Higgs Audio v2 (Boson AI, Apache-2.0).
- No error accumulation from an intermediate semantic stage, and far fewer sequential decoding steps — hence the speed.
Five core capabilities
| Capability | Details |
|---|---|
| Zero-shot cloning | 3–10 s of reference audio. Built-in Whisper auto-transcription means you do not have to supply the reference transcript. Handles noisy / reverberant references by extracting a clean speaker embedding. |
| Voice Design | No reference audio required — describe the voice in natural language (male, elderly, low pitch, British accent) and it generates one. Supports gender, age, pitch, accent, dialect and whisper. |
| Fine-grained text control | Inline non-verbal tags: [laughter], [sigh], [breath], [sniff] and 13+ others. |
| Pronunciation override | Chinese via pinyin with tone marks (e.g. ZHE2); English via CMU pronunciation dictionary. |
| Multi-speaker dialogue | [Speaker_N]: tags drive multi-character script synthesis, each speaker with an independent voice. |
Speed — paper vs reality
The headline figure is RTF 0.025 (40× real-time). That measurement was taken on an H100 with dedicated acceleration kernels, not on a workstation. Independent hands-on testing on a Mac Studio / M4 Max reported roughly 4–6 seconds of compute for 5.4 seconds of audio — near real-time, and about half that with --num_step 16 at some cost in clarity. Budget accordingly.
Known issues from third-party testing
- Occasional dropped or mangled words at the start or end of sentences — in one 24-generation test, 5 outputs lost or corrupted a word. It is a known issue; re-running usually resolves it.
- Generating a voice clone and saving it (
model.create_voice_clone_prompt()→.save("voice.pt")) lets you skip the reference clip and transcript in later sessions — useful for batch work. - Fully offline operation works if you pre-download the weights with
hf download k2-fsa/OmniVoice --local-dir ./modeland pass that path to--model. SetGRADIO_ANALYTICS_ENABLED=Falsebefore launching the web UI — it phones home to Gradio on startup.
Ecosystem
- GGUF quantizations exist (
Serveurperso/OmniVoice-GGUF, ~62k downloads/month), ranging from Q4_K_M at ~660 MB to F32. - omnivoice.cpp — a C++17/GGML port that runs on CPU, CUDA, ROCm, Metal and Vulkan, including Apple Silicon. This makes OmniVoice by far the most portable option here.
- Default engine in the popular VoiceStudio local-ElevenLabs-alternative project.
- Apple Silicon supported natively via MPS (~3.3 GB download, no NVIDIA card needed).
Use cases: language breadth and portability
| Scenario | How to run it | Watch out for |
|---|---|---|
| Low-resource and rare languages | 646 languages, covering many with no commercial TTS support at all | Weights are CC-BY-NC — that alone vetoes any commercial use |
| CPU-only / Raspberry Pi / edge devices | omnivoice.cpp runs on CPU, Metal, Vulkan and ROCm; the Q4_K_M language model is under 1 GB | The RTF 0.025 headline was measured on an H100 with custom kernels; locally expect closer to 1× real-time |
| Fully air-gapped deployment | Download the weights once, then no network request is ever made | Technically viable, but the non-commercial licence rules out public-sector and hospital work |
| Research and non-commercial prototyping | Voice Design creates a timbre from a description, with no reference audio | Voice Design is stochastic — generate one to three candidates and pick |
| Running locally on a Mac | Native Apple Silicon support via MPS, ~3.3 GB of weights | Capacity-plan from your own measurements, not the paper numbers |
A counter-intuitive selection conclusion: OmniVoice runs fully offline and ships a CPU build, which makes it look ideal for public-sector and hospital deployments — but its weights are CC-BY-NC, so procurement will not clear it. Those scenarios should use VoxCPM2 instead (Apache-2.0 on both code and weights).
Verdict for international users
Use it if you are doing research, personal projects, or low-resource-language work, or you need the widest possible language coverage and the easiest CPU/Mac path. Voice Design without any reference audio is genuinely useful for prototyping.
Do not use it commercially without negotiating a separate licence — the CC-BY-NC weights are the blocker, regardless of the Apache-2.0 repo badge. For a commercially clean alternative with similar "describe-a-voice" functionality, look at VoxCPM2, Chatterbox (MIT) or CosyVoice 3.
646 languages
Voice Design
CPU / Mac / Vulkan
GGUF + C++ port
NC weights — no commercial use
Where it sits in the head-to-head data
| Model | Params | Chinese WER / SS | English WER / SS | Spanish WER / SS |
|---|---|---|---|---|
| OmniVoice | 0.8 B | 3.41 / 72.99 | 3.62 / 70.13 | 3.52 / 74.14 |
| CosyVoice3-0.5B | 0.5 B | 3.84 / 80.01 | 4.88 / 74.16 | 4.04 / 78.85 |
| Fish Audio S2 Pro | 4 B | 3.62 / 67.79 | 3.83 / 61.66 | 2.93 / 67.44 |
| IndexTTS2.5-RL | 0.8 B | 3.93 / 77.92 | 3.89 / 67.79 | 3.33 / 76.68 |
| Model | zh→en WER / SS | zh→es WER / SS | zh→ja WER / SS |
|---|---|---|---|
| OmniVoice | 3.74 / 64.91 | 5.84 / 62.08 | 9.09 / 69.06 |
| VoxCPM2 | 4.48 / 64.25 | 16.38 / 64.89 | 11.84 / 71.54 |
| Model | RTF | Conditions |
|---|---|---|
| OmniVoice | 0.025 | H100 + custom acceleration kernels (paper figure) |
| OmniVoice (measured) | ~0.9–1.2× real-time | Mac Studio / M4 Max |
| IndexTTS2.5 (reference) | 0.2065 | RTX 4090 bf16 |
The key to reading this: lowest WER does not mean "sounds most like the person". OmniVoice's English WER of 3.62 is among the best in the set, but its speaker similarity is 70.13 — bottom of the leaders. It is clearer, not more identical. That is an advantage for news reading and a real weakness for character dubbing or a fixed narrator across an audiobook.
What it costs to run yourself
| Item | Value | Note |
|---|---|---|
| GPU-hours / M chars | 0.45 h | Paper conditions |
| On-demand H100 ($2.89–3.49/h) | ~$1.30–1.57 | Nominally the cheapest in the set |
| On-demand RTX 4090 | Not applicable | The paper figure depends on H100 plus custom kernels and does not transfer |
| Weight size | 3.3 GB | Under Q4_K_M the language-model portion is under 1 GB |
Do not budget on 0.025. That is a lab figure on an H100 with purpose-built kernels. On a consumer card or Apple Silicon the measured number lands at roughly 0.9–1.2× real-time — one hour of audio takes about an hour. Its nominal cost is the lowest and its hardware bar is the highest. Re-costed on the same on-demand 4090 basis, it is not cheaper than IndexTTS2.5.
But it has a different cost advantage: GGUF plus
omnivoice.cpp lets it run on a machine with no GPU at all (CPU / Metal / Vulkan / ROCm). If you need a TTS inside an edge box, it is the only option in this set — you just have to accept the real-time factor.Licence and disclosure gate
| Item | Status |
|---|---|
| Code licence | Apache-2.0 |
| Weights licence | CC-BY-NC 4.0 (non-commercial) |
| Commercial use | ✗ Not permitted — unless you obtain a separate licence |
| Default watermark | Not stated |
| Data residency | Runs fully air-gapped (pre-download weights, disable Gradio analytics) |
A counter-intuitive conclusion: government and healthcare are exactly the scenarios that look like they need an offline model most — and they are the one place OmniVoice genuinely excels technically. But CC-BY-NC weights kill it at procurement and compliance. Don't burn a proof-of-concept here; go straight to VoxCPM2. The full licence breakdown is in 06.5.
If not this, then what
- Language breadth, but it has to be shippable → Step Audio EditX (Apache-2.0) or Kokoro 82M (CPU, Apache-2.0, no cloning).
- Fully offline and commercially clean → VoxCPM2.
- Research or hobby use only → keep it. Nothing is cheaper for 646-language coverage.
- Real-time interaction on CPU → nothing here. Look at Kokoro in the 06.1 table, or use a closed API.
github.com/k2-fsa/OmniVoice
huggingface.co/k2-fsa/OmniVoice
arXiv:2604.00688
Try it on BiliVoice
Ready to create with OmniVoice?
Open Text to Speech with this engine selected and start with your own voice model.