Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

sybil-solutions/inference-optimisation-skill

Machine Learning

An agent skill that guides systematic speed tuning of local LLM servers, from decode and prefill to KV cache and startup.

1,035
89
Python
MIT
1 day old

What it is

This repository packages a procedure for making a local LLM server faster: decode throughput, concurrent decode, speculative decoding, prefill, KV cache size, startup time and time to first token. It is written as an agent skill, so Claude Code or another agent that loads SKILL.md-style skills can follow it; a plain PROMPT.md version exists for models without skill support.

The method is the substance, not a flag list: compute the hardware's theoretical ceiling, change one setting at a time, compare candidates against interleaved control runs, log failures, then require a long-context retrieval check and a concurrent full-KV soak before calling a configuration the best. Reference files cover ceiling calculations, measurement traps, individual levers, engine-specific gotchas, and publishing a pinned build (skill files, SKILL.md).

Quick start

For Claude Code, install at user level:

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then prompt it with a goal, for example: "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap".

Without an agent, use the bundled standard-library Python and bash harness: compute ceilings from a bandwidth/FLOPs JSON, copy the example server config and edit base_url/model, run a benchmark, then queue interleaved control/candidate runs.

S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json
python3 $S/scripts/bench.py run configs/base.json --long 131072
bash $S/scripts/queue.sh base cand base
python3 $S/scripts/bench.py show

Results land in ledger.jsonl and are displayed as a table by the show command (scripts).

Constraints

  • bench.py targets any OpenAI-compatible server; token counts are read from server usage stats rather than streamed chunks.
  • The harness is standard-library Python plus bash, so it needs no extra Python dependencies, but it does require a running local inference server and a config pointing at it.
  • The workflow is designed around memory caps: bench.py includes a cap watchdog, and mem.sh reports container footprint including driver residency on unified-memory GPUs.
  • Meaningful comparisons depend on repeated samples and interleaved controls; the README puts measurement noise at ±5–10%, so single-run results are not trustworthy.
  • Reported speedups are hardware-, model-, quantization- and engine-specific; the tables are evidence for the method, not portable expectations.
  • Licensed MIT.

HotRepo's take

The repository appeared on HotRepo's new-project board on 2026-11-17, with 144 stars gained in a window of roughly eight hours and no recorded all-language daily rank. That is a first-day signal only, not a trend. What is notable technically is that the skill documents ceilings and accepts results only after long-context retrieval gates and concurrent KV soak tests, and it ships a harness that enforces interleaving and medians rather than relying on a single benchmark run. The README's before/after tables point to external recipe logs for raw data, so readers comparing hardware should follow those links rather than treat the table as a universal benchmark.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. trymirai/uzuRuns LLM and TTS inference locally on Apple Silicon, for app developers who want privacy and no per-token cost.Rust2,566
  2. kvcache-ai/MooncakeServing infrastructure for LLM inference clusters that need shared KV cache storage, RDMA data movement and disaggregated prefill-decode.C++6,907
  3. Helldez/BigMoeOnEdgeRun MoE GGUF models larger than available memory by streaming experts from flash, with a local OpenAI-compatible server.C++848
  4. eagle-hq/EAGLE-4Speculative decoding drafter and harness that speeds up LLM serving in SGLang, vLLM and TokenSpeed across batch sizes and GPUs.Python967
  5. JakeATX/llamAmpereA llama.cpp fork tuned for NVIDIA RTX 30-series cards, focused on long-context Qwen inference with low VRAM.C++301