incoai/splash
Machine LearningLocal LLM inference server for Apple silicon Macs, exposing OpenAI- and Anthropic-compatible APIs for coding agents.
What it is
Splash is a local inference engine for Apple silicon Macs. It runs coding agents and applications that speak the OpenAI or Anthropic APIs against models served from the same machine. Its main ingredients are DFlash 2 speculative decoding, model-specific Metal kernels, and automatic memory planning; on top of that it handles vision input, tool calling, prefix-cache reuse, and concurrent request batching. A built-in chat page is served at the same address.
Supported model families include Qwen3.8-27B, Qwen3.6-35B-A3B, Ornith-1.5-122B, Qwen3.8-Flash-Next and Flint, in Unsloth GGUF variants from 1 to 8 bits or MLX 4-bit packages; Prism ML Ternary Bonsai 2 is also supported. The engine pairs each target model with a matching draft model automatically. Whole-BF16/F16 targets are refused.
Quick start
Requirements: an Apple M3 or newer running macOS 26.4 or later. Either install the desktop app from the release DMG, or use Homebrew for the CLI.
brew install incoai/tap/splash
splash serve --model unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
The first run downloads the target model and its draft; later runs reuse them, so leave disk space for both. Once the server prints Ready, open http://127.0.0.1:8000 for the chat page, or launch an installed coding agent in another terminal with splash opencode, splash claude, splash codex, splash hermes or splash pi. If you changed the port, pass the same --port to the agent command. Stop with Ctrl+C.
The OpenAI-compatible endpoint can also be called directly:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M",
"messages": [{"role": "user", "content": "Explain speculative decoding in one sentence."}]
}'
To serve files already on disk, pass the GGUF file with --model-dir and a compatible draft with --draft-model; --model then only names the model in the API.
Configuration and constraints
Memory and context are sized automatically up to the model's native window. Flags such as --max-memory, --max-context, --idle-release, --kv-format and --max-cache-disk override that behavior; the 27B defaults to also using the Neural Engine for prefill, which --disable-ane turns off. By default the model is released from memory after 10 minutes idle, and the server listens on localhost without authentication. LAN access, authentication and cross-origin browser apps need explicit server configuration.
The project is Apache-2.0 licensed; the GGUF kernels include MIT-licensed material from llama.cpp, and model weights keep their own licenses. Builds target Apple silicon only, so Intel Macs and other platforms are not covered.
HotRepo's take
On the 2026-11-18 daily C++ list Splash ranked 8th with 51 new stars, while it has no recorded position on the all-language daily list. The README makes its strongest case through measured comparisons against llama.cpp on Metal: on an M3 Max with 40-core GPU, 92 tok/s decode for the 27B model versus 17 tok/s for llama.cpp, and 209 tok/s versus 66 for the 35B-A3B model, with next-token agreement of 99.30–99.45% and 97.83–98.14% respectively. Those figures come with a specific hardware and settings scope (BF16 KV, ANE disabled) and are author-supplied, so they describe the project's claim rather than an independent result.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- nace-ai/kapitsaA local inference engine for running Qwen, Nemotron and Flint models on Apple silicon Macs, aimed at coding agents.
- ashhart/TensorFoldServe quantized LLMs natively on Apple Silicon and NVIDIA GPUs with an OpenAI-compatible API and no Python.
- trymirai/uzuRuns LLM and TTS inference locally on Apple Silicon, for app developers who want privacy and no per-token cost.
- sybil-solutions/inference-optimisation-skillAn agent skill that guides systematic speed tuning of local LLM servers, from decode and prefill to KV cache and startup.
- magnitudedev/magnitudeRuns open models locally on your own hardware, auto-tuning kernels per model for faster inference with coding agents.