sgl-project/sglang-omni
Machine LearningA multi-stage serving runtime for omni, speech, TTS, and music models with OpenAI-compatible APIs.
What It Does
SGLang-Omni is a serving runtime for omni, speech, and text-to-speech models. It targets multi-stage decoding, where generation is split across heterogeneous stages such as preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators. The runtime owns pipeline topology, stage lifecycle, inter-stage transport, model-family integration, and an OpenAI-compatible serving surface, composing with SGLang for autoregressive scheduling and execution where applicable. See the project documentation.
It exposes multimodal chat, speech generation, batch and streaming speech, uploaded voices, transcription, and realtime full-duplex conversation. Supported model families include Qwen3-Omni, MiniCPM-o, Ming-Omni, several TTS models, ASR models, speaker diarization, speech translation, music generation, and image/video generation. Model-specific launch steps are in the Cookbook.
Quick Start
- Install the v0.2.0 prerelease from PyPI:
The installation guide covers the full environment setup.uv pip install --prerelease=allow "sglang-omni==0.2.0" - Launch a model following its cookbook page; model names, launch commands, and card requirements differ per model. For example, start with the Qwen3-Omni cookbook for multimodal chat or the Qwen3-TTS cookbook for speech generation.
- Call the OpenAI-compatible endpoints such as
/v1/audio/speechfor speech and/v1/audio/transcriptionsfor ASR. Exact request formats and model-specific options are documented per cookbook page.
Alternatively, the Voxt Mac voice input tool integrates SGLang-Omni for local speech recognition; a prebuilt Apple Silicon disk image is available from its release download.
Hardware and Compatibility
NVIDIA CUDA is the default backend with full model coverage. Intel Arc GPUs via PyTorch XPU are supported, with a listed set of models serving end-to-end and per-model launch commands in the Intel XPU guide. Apple Silicon is experimental and currently limited to Qwen3-ASR through Torch MPS, installed with the repository's install.sh.
The project is Apache-2.0 licensed. Speech-generation features such as batch, streaming, and voice cloning vary by model, so check the model's cookbook page before relying on a capability.
HotRepo's take
HotRepo recorded SGLang-Omni at 5th place on the Python daily list on 2026-11-11, with 19 stars gained that day against a total of 1,752 stars and 719 forks. The single snapshot shows same-day placement on a language list, not a broader trend; what stands out technically is the breadth of its model coverage, from full-duplex conversation and ASR to music and image generation behind one OpenAI-compatible runtime.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- 0xShug0/audio.cppA C++/ggml runtime for running many audio model families locally without Python, from TTS to speech recognition and music.
- vllm-project/vllmServes and batches LLM inference at high throughput for teams running their own models.
- sgl-project/sglangServing stack for teams that need fast LLM, VLM and diffusion inference at scale on many hardware backends.
- trymirai/uzuRuns LLM and TTS inference locally on Apple Silicon, for app developers who want privacy and no per-token cost.
- OpenBMB/VoxCPMA 2B tokenizer-free TTS model for 30-language speech synthesis, voice design, and voice cloning.