Mesh-LLM/mesh-llm

Mesh LLM lets several machines contribute GPU and memory to serve LLM inference behind a single OpenAI-compatible /v1 API, normally on localhost:9337. It fits people who already have one or more GPU machines and want to run local open-weight models, share them privately over invites or publicly through discovery, and point agent tools at one endpoint without operating a hosted provider.

How the pooling works

Mesh is built on Skippy, which handles model management, the llama.cpp-derived native runtime, split execution and OpenAI serving; Mesh adds peer discovery, transport, routing, plugins and a web console README. Routing is pragmatic: if one node can hold the full model, it serves locally with no stage traffic; otherwise requests are routed by the model field to a peer that has it. For models too large for any single machine, Skippy stage splits load a model as contiguous layer ranges on several nodes, with the coordinator publishing the stage-0 route only after downstream stages are ready split serving guide.

Peer traffic, including inference and split activations, is QUIC end-to-end encrypted, and Iroh relays forward packets without reading them. Private meshes are joinable only by invite token, while published meshes advertise through Nostr discovery. Uninstalling the executable keeps ~/.mesh-llm config and identity unless --purge-config is passed, so a removal can be previewed first README.

Ways to run it

• mesh-llm serve --auto joins the best discovered public mesh, fetches a suitable model, and starts the API plus console.
• --model <catalog-name> starts a private mesh; --publish makes it discoverable; --join <token> joins by invite meshes guide.
• mesh-llm serve --local-model-only --gguf <abs-path> is explicitly a debugging/development topology: it bypasses QUIC, discovery, splits, plugins and the console, does not fall back to distributed serving, and is not the supported production path for OpenAI clients including tool calls. Paths must be absolute non-symlink GGUF files.
• An API-only client mode uses the same mesh without hosting GPU work.

Agent integrations for Goose, Claude Code, OpenCode and Pi, plus a model: "mesh" Mixture-of-Agents fan-out across all models in the mesh, are documented alongside the core flow agents guide. MoA is labelled experimental: its routing, error shapes and tuning may change between versions, and it needs at least two distinct models MoA design. Flash-MoE and DwarfStar are optional plugins rather than base requirements plugin docs.

Requirements and caveats

The author calls the project experimental distributed-systems software. The reviewed Skippy support matrix covers 72 P0/P1 family rows across Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi and others, with split multimodal certified for a shorter list family status. Release bundles exist for macOS plus Linux and Windows on CPU, CUDA, ROCm and Vulkan, with Metal on macOS only install matrix. Source builds need just, cmake, Rust, sccache, a fast linker and Node.js 24 with npm. As of the v0.76.0-rc8 notes, three Windows distribution issues still block the plain native path: the stable Windows bundles fail install verification, runtime install cuda finds no Windows runtimes, and runtime DLLs need an explicit PATH entry Windows guide. The shipped executable's embedded attestation supports provenance and admission checks but is not a runtime integrity proof, and default startup still allows a binary that changed after packaging.

HotRepo's take

Stars stood at 3,619 on a single recorded daily-board appearance on 2026-10-29, when the repository gained 50 stars repository. That one sample is not enough to describe momentum. The project is worth evaluating for a multi-GPU home or lab setup; the stable production path is the mesh inference port, not --local-model-only.