Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

Vitolearns/KVHarbor

Machine Learning

A C++20 single-GPU LLM serving layer with HTTP/SSE admission control, streaming, and optional speech endpoints.

107
0
Python
NOASSERTION
2 days old

What it does

KVHarbor is a C++20 HTTP service that puts a controlled front end in front of llama.cpp inference on a single NVIDIA GPU. It handles request admission, deadlines, cancellation, streaming backpressure, and worker release, while llama.cpp does tokenization and model execution in process. Optional CPU speech recognition and synthesis run through sherpa-onnx, using Qwen3-ASR for transcription and Kokoro for text-to-speech. The target audience is people who need to serve a local GGUF model and care about overload behavior rather than only raw decode speed.

The repository is explicit about boundaries: it does not implement cross-request continuous batching, combine sequences into one llama_batch, or replace the upstream KV allocator. The engine dispatcher only interleaves serial calls across independent worker contexts, and each context holds one sequence (architecture and implementation notes).

Quick start

The fastest path needs no GPU or model weights: it builds a CPU test backend to exercise the HTTP/SSE path. On Ubuntu 24.04 (native or WSL2), install git, curl, python3, build-essential, cmake, libboost-dev, and nlohmann-json3-dev, then clone the repository and build the CPU serving target:

cmake -S serving -B build-serving-cpu -DCMAKE_BUILD_TYPE=Release
cmake --build build-serving-cpu --parallel 2
./build-serving-cpu/kvharbor-server --test-backend --port 18082

In another terminal, send a streaming chat request with stream: true and max_tokens: 4. The response should contain four content chunks, a final JSON chunk with finish_reason: "stop" and token usage, then data: [DONE] (quick start). Stop the server with Ctrl+C.

For real text generation, download the locked Qwen3.8-27B Q4_K_M GGUF, verify it against the checksum in the text runtime lock, build the llama.cpp runtime and server, then start scripts/native_serving.sh serve with the documented worker, context, batch, and scheduling flags. Health is available at /health, and chat requests go to /v1/chat/completions (real-model instructions). Speech requires running the preparation script and rebuilding with KVHARBOR_AUDIO=1; the transcription and synthesis endpoints are /v1/audio/transcriptions and /v1/audio/speech (speech setup).

Constraints

The reference setup is Ubuntu 24.04 with an RTX 5090 Laptop GPU, and the pinned CUDA build targets SM120; other GPU architectures need a compatible build configuration. Real model serving requires Docker with NVIDIA GPU access, Python 3.12+, Bash, Git, curl, and the Boost and nlohmann-json development packages. The optional speech models also need their pinned runtime assets, which are downloaded and verified by the preparation script.

Request handling has visible limits. Integer request fields such as max_tokens and speech voice IDs must be JSON integers; booleans, strings, and floating-point values are rejected. The current adapter consumes only the last logits row per single-sequence call, so multi-sequence batching would require revisiting that assumption. The project is licensed under PolyForm Noncommercial 1.0.0, and inherited code and model terms are listed in the NOTICE.

HotRepo's take

The only recorded HotRepo activity is a 2026-11-20 new-project snapshot on the Python list: 107 stars, 0 forks, and 4 stars gained over a window of roughly 1.5 hours, with no GitHub daily rank recorded. That is too little history to say anything about sustained interest, so the useful signal is in the repository itself: the README publishes both positive and negative experiment results, including a decode-first policy that lowered p95 time per output token but did not improve SLO goodput (prefill/decode report). For a serving project at this stage, that honesty about end-to-end effects matters more than the star count.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. Neroued/ninferNative C++/CUDA inference engine serving selected LLMs on RTX 5090 and DGX Spark hardware.C++3,280