trymirai/uzu

The problem

If you want an LLM inside a mobile or desktop app, the default answer is a hosted API: you send user text to someone else's server and pay per token. For privacy-sensitive apps, offline use, or high-volume products, that is either unacceptable or too expensive. uzu is a Rust inference engine aimed at embedding models directly in your app, using Apple's unified memory to keep latency low and data on-device.

What's different

Most local inference today goes through llama.cpp or MLX, and both are general-purpose runtimes you wire up yourself. uzu's bet is a single high-level API across four languages — Rust, Python, Swift, TypeScript — with model download, configuration, and session handling bundled in, so the same call looks identical in a desktop app and an iOS app. Its second differentiator is explicit Apple Silicon targeting plus a model registry with its own naming scheme (alibaba:qwen3.5:0.8b:mirai:mirai-m:4), which removes most of the file-shuffling that local inference usually requires. The README also claims "traceable computations" to verify results against a reference implementation, though the README is thin on how that works in practice.

Who it's for

App and product developers shipping chat or TTS features on macOS or iOS who want on-device inference behind a stable SDK, and who prefer not to maintain their own llama.cpp bindings. Researchers who want to control quantization, sampling internals, or port arbitrary architectures should look at MLX or llama.cpp instead — uzu is opinionated and the README does not document low-level control. Non-Apple platforms are not the target.

Cost of adoption

You are committing to Apple Silicon, a young vendor, and a pre-1.0 API: the package is versioned 0.6.13 across PyPI, npm and SPM, and the Rust dependency is pulled from the main branch rather than crates.io, which means no semver stability. The license is MIT, which is permissive. Model weights are downloaded at runtime from Mirai's registry, so you depend on that service being reachable. The README is a quick-start and example walkthrough; deeper operational details live in external docs, which makes evaluating failure modes harder before you start.

Verdict

uzu is roughly 16 months old and gained 745 Rust stars in the past month, sitting at #10 on HotRepo's monthly Rust board — that is a sharp recent spike on a project that had a modest 186-point Show HN in mid-2025, not a long-established track record. The cross-language SDK and Apple-first design are a real convenience that generic runtimes do not offer, but the pre-1.0 versioning, git-branch dependency and runtime model registry are meaningful lock-in risks. Worth a prototype if you are building on Apple platforms and want the fastest path to on-device chat; wait if you need API stability or non-Apple targets.