ashhart/TensorFold

Overview

TensorFold is a Zig-based inference engine that loads model checkpoints, tokenizes requests, and runs Metal or CUDA kernels directly, serving an OpenAI-compatible HTTP API. It targets people who want to run quantized open models on their own Apple Silicon Mac or NVIDIA GPU without installing Python or MLX. A native build also covers AMD HIP for one OCR checkpoint, and the project ships a macOS desktop workspace under app/ plus an experimental weight-editing mode called Living Weights.

Quick start

The documented starter path uses Qwen3.5-2B in MLX 4-bit, checked on Metal and a GB10. On Apple Silicon, install via Homebrew, pull a pinned revision, then serve it:

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

On Linux with an NVIDIA GPU, the README downloads a release archive for linux-aarch64 or linux-x86_64, verifies its checksum, extracts it, and defines a tensorfold shell wrapper around tensorfold-native before running the same pull and serve commands (Quick start). The archive requires glibc 2.28 or newer and NVIDIA driver R580 or newer (CUDA 13.0); older drivers are refused at startup. Windows with an NVIDIA GPU follows a separate native build guide.

Leave the server running and send a chat request from another terminal:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

The same endpoint also serves completions, Responses, and Anthropic Messages; /v1/models lists the served name. tensorfold web opens a small local web app for picking, running, and chatting with models (README quick start).

Supported models and hardware

The model table lists native serving paths, download commands, machines checked, and per-model limits, covering Qwen3.5/3.6/3.8, Nemotron 3.5 Lightning and Super VL, GLM-5.3-Flash variants, LFM2.5, and a HIP OCR path. Some entries are explicitly unqualified or experimental: DeepSeek checkpoints are not served by any release binary yet, and single-Mac GLM-5.3-Flash serving is not qualified. Several large MoE checkpoints require two or more machines (for example two M5 Ultras or two/three GB10s), and tensorfold info REPO reports a checkpoint's memory floor before you download and serve it (model table).

Usage constraints

Serving needs the model weights' own licenses (separate from the Apache-2.0 code), enough unified or GPU memory for the checkpoint plus context, caches, and parallel streams, and the platform-specific toolchain described above. Quantization and backend choice affect output, so exact-decoding guarantees are scoped to a fixed checkpoint, backend, settings, and runtime. Compaction, Cortheon web research, Living Weights, and speech features are opt-in flags with their own limits; Living Weights can be undone with tensorfold slide rollback. Building from source requires Zig 0.17.0 and the Xcode Metal Toolchain for .metal kernels (build instructions).

HotRepo's take

On the 2026-11-12 daily all-language board, TensorFold recorded a best rank of 7 with 58 new stars against 1,811 total stars and 385 forks. That single snapshot puts it in the upper tier of GitHub's daily movement, and the repo backs the attention with unusually detailed engineering material: pinned checkpoint revisions, benchmark receipts, qualification notes, and explicit statements about which hardware/model combinations were actually checked. Readers evaluating whether to run it should treat those documented qualification boundaries, not the ranking, as the useful signal.