All skills
inference-serving
Compares batching, scheduling, KV cache, offloading and speculative decoding for a single-instance inference service.
- Part of bojieli/ai-infra-book
- Machine Learning · Model serving
More Model serving skills
See all- serving-llms-vllm66Serves large language models at high throughput with an OpenAI-compatible API and quantization.
- tensorrt-llm21Runs high-throughput language model inference on NVIDIA GPUs.
- perf-optimization18Routes performance work to the right specialist and applies optimizations or kernels in a safe order.
- inference-optimisation16Measures and tunes local LLM serving speed, testing changes one at a time and keeping a record of results.
Also in this pack
- accelerator-selectionCompares accelerators on capacity, effective compute, memory bandwidth, power and cost for a given model workload.
- agent-runtime-capacityPlans how agent model calls and tool environments are created, kept, released and restored for capacity and latency.
- cluster-network-analysisEstimates cross-server traffic, network bottlenecks and sync waits for multi-node training or inference.
- distributed-inferenceCompares cross-instance inference options like replicas, prefill/decode split, expert placement and shared KV cache.