ranxianglei/billion-context

The problem

Coding agents accumulate history fast. After a few hours, every request replays a huge prefix, and once you cross the window the session either degrades into summarization-by-truncation or dies outright. You pay for the same tokens repeatedly, and the host's built-in summarizer throws away detail you may need later. billion-context sits as a proxy between the agent and the model API and rewrites the stream so consumed history gets folded into summaries instead of being dropped.

What's different

The usual fixes are host-side summarization (lossy, one-way) or just buying a bigger window. billion-context compresses incrementally in layered ranges and exposes decompress and search_context tools, so the model can pull an exact detail back when it matters. It is meant to be prefix-cache friendly — the README claims a 95–97% hit rate with compression costing ≤2% — which matters because naive rewriting of the prefix invalidates provider caching and quietly negates the savings. It also claims to work with a long list of clients (Claude Code, Codex, Gemini CLI, OpenCode, aider, etc.) through one proxy rather than a per-client plugin.

Who it's for

Developers running agentic coding sessions that last days, and anyone on metered API billing who wants one continuous session instead of many short ones. It is also relevant if you must stay on a ~100K-token model. Skip it if you use a single closed IDE assistant you cannot repoint at a custom base URL, or if your sessions are short and already fit comfortably in context — the added proxy hop buys you nothing.

Cost of adoption

Node/npm install (npm install -g billion-context --prefix=~/.local), then point your agent's base URL at the proxy or use the bili launcher/plugin path for supported clients. You are inserting a man-in-the-middle into every model call: model switching and cache TTL behaviour become failure modes, and the README itself lists those as the top causes of poor cache health. The license is NOASSERTION, so check the actual terms before shipping it internally. Compression quality depends on the model following injected tool instructions; a weaker model will produce worse summaries.

Verdict

79 days old, 2,597 stars and +937 in a single week on the TypeScript board — this is an early project riding a real spike, not an established tool. The accompanying paper (a large longitudinal study) is unusually serious for a project this young, but it is self-published and the README is the main source of truth for behaviour. The approach is sound and the pain point is genuine; the risk is that the proxy sits in the most sensitive part of your stack. Try it on a side project with /acp-cache open before trusting it with production work.