VoxDance-Lab/VoxMel
Machine LearningA tokenizer-free autoregressive TTS framework that models mel-spectrograms directly and supports English/Chinese zero-shot voice cloning.
Overview
VoxMel is a continuous autoregressive text-to-speech framework that skips a learned acoustic tokenizer and models mel-spectrograms directly. It splits mel features with a fixed, exactly invertible two-axis Haar transform: a decoder-only transformer with a flow-matching head generates Coarse-Mel autoregressively, then two non-autoregressive tiers recover temporal and frequency residuals. A frozen Vocos vocoder converts the reconstructed mel to 24 kHz speech. The intended users are speech researchers and engineers working on zero-shot voice cloning in English and Chinese.
Quick Start
Requirements: Python 3.10–3.12 and a matching PyTorch/torchaudio build (the reference environment uses torch/torchaudio 2.4.1 with CUDA 12.1). Install from the repository root, then place compatible weights before running anything:
pip install -e .
The checkout contains model code and a Qwen tokenizer but no trained VoxMel weights, experiment config, or Vocos weights. Download the checkpoints from VoxMel-0.6B or VoxMel-0.3B and arrange them as:
/path/to/experiment/
├── config.yaml
└── model_L_final.pt
/path/to/vocos-mel-24khz/
├── config.yaml
└── pytorch_model.bin
Minimal Python use, with a reference clip and its exact transcript:
from voxcore import VoxMel
import soundfile as sf
model = VoxMel.from_pretrained(
"/path/to/experiment",
tokenizer_path="examples/voxmel/qwen_tokenizer",
vocos_path="/path/to/vocos-mel-24khz",
device="auto",
)
wav = model.generate(
"这是声音克隆的演示。",
prompt_wav_path="reference.wav",
prompt_text="参考音频中实际说出的内容。",
cfg_text_strength=3.5,
cfg_spk_strength=3.0,
seed=42,
)
sf.write("clone.wav", wav, model.sample_rate)
The same model can be driven from the CLI (voxmel --model ... --prompt-audio ... --output out.wav) or a Gradio demo (pip install -e ".[web]", then python app.py --model ... --port 8808); see the quick start for full argument lists. The demo queues requests one at a time.
Configuration and Controls
device="auto" picks CUDA, then MPS, then CPU; an explicit unavailable device raises an error. Text and speaker guidance are independent (cfg_text_strength default 3.5, cfg_spk_strength default 3.0), speaker guidance only applies when reference audio is supplied, and up_num_steps sets residual solver steps (demo default 32). min_seconds/max_seconds bound coarse-frame generation, while model stop predictions determine the actual duration. For non-CUDA batch evaluation, use DEVICE=cpu or DEVICE=mps with NPROC=1; the batch script accepts GPU, NPROC, MAX_ITEMS, COARSE_STEPS, UP_STEPS, and CFG overrides and skips existing outputs.
Constraints
Zero-shot cloning requires both the reference audio and its exact transcript; without a compatible experiment, Vocos checkpoint, and config, the repository cannot generate speech. MPS operator support and performance still need validation against your checkpoint. The 24 kHz output and the 100K-hour Emilia training setup are the paper's configuration. Code and model weights are Apache-2.0; third-party components keep their own licenses, and the Vocos vocoder is used frozen.
Evaluation Context
On Seed-TTS-eval, the supervised 0.6B model reports 1.51% WER / 0.738 SIM for English and 1.29% CER / 0.755 SIM for Chinese; frozen-head GRPO post-training improves this to 1.15% WER / 0.753 SIM English and 1.17% CER / 0.776 SIM Chinese. Only VoxMel-0.6B received the RL post-training; the 0.3B model did not. These are the authors' reported numbers on that benchmark, not independent reproduction.
HotRepo's take
Coverage is limited: VoxMel entered HotRepo's new-project board on 2026-11-18 for the Python list, gaining 6 stars over roughly 1.5 hours to reach 106, with no all-language daily rank recorded. That record is too short to indicate a trend. The more durable signal is that this is a complete research release — code, two model sizes, a technical report, and a community MLX port for Apple Silicon — rather than a demo-only announcement.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- OpenBMB/VoxCPMA 2B tokenizer-free TTS model for 30-language speech synthesis, voice design, and voice cloning.
- hehehai/AudioSmithA local Mac studio for text-to-speech, voice cloning, audiobooks, dubbing and transcription, with all models running on-device.
- pnnbao97/VieNeu-TTSVietnamese text-to-speech with instant voice cloning that runs on CPU or GPU for real-time speech.
- 0xShug0/audio.cppA C++/ggml runtime for running many audio model families locally without Python, from TTS to speech recognition and music.
- n4ze3m/tiny-vocaA 4.5M-parameter English text-to-speech model with two voices that runs on CPU or in the browser.