ai-dynamo/nixl

NIXL (NVIDIA Inference Xfer Library) is a C++-based library for point-to-point data movement in AI inference frameworks such as NVIDIA Dynamo. Its main value is a single abstraction over different memory types (CPU, GPU) and storage types (file, block, object store), implemented through a modular plugin architecture. It fits teams building inference infrastructure who need to connect agents, memory regions and storage backends without writing per-transport code.

How it works

NIXL is organized around agents and interchangeable backends. The Python API wraps the C++ core through pybind11, and backends such as UCX, POSIX, GDS and Mooncake are plugins with their own build notes; the repository also documents a backend development guide for writing new ones overview, backend guide, Python API. For distributed setups, an optional ETCD path exchanges agent metadata and coordinates nodes, exposed through environment variables and demonstrated in an example program README.

Supported ways to use it

• PyPI wheel: pip install nixl provides the Python API with UCX and both CUDA 12 and 13 backends; the right backend is selected at runtime from the CUDA version reported by PyTorch README.
• Source build: Meson/Ninja build with a C++20 compiler (GCC 11+ or Clang 14+), tested against UCX 1.23.x. GDRCopy is optional but described as necessary for maximum performance README.
• Python/Rust bindings: Python bindings can be built from source, requiring uv, tomlkit and PyTorch; Rust bindings are built with a Meson option or cargo README.
• Container and wheel scripts: helper scripts build an Ubuntu 22.04/24.04 container and a wheel README.

Requirements and limits

The library is Linux-only, tested on Ubuntu 22.04/24.04 and Fedora; macOS and Windows are unsupported README. Optional features have their own prerequisites: ETCD needs a server plus the etcd-cpp-apiv3 client, and some plugins (Mooncake, POSIX, GDS) have additional build requirements README, plugin notes. A ROCm path exists, but it is incomplete: the LIBFABRIC plugin is disabled on ROCm and there is no NVSHMEM-equivalent backend yet README. Benchmarks live in NIXLBench and KVBench; these are benchmarking tools, not performance guarantees for a given deployment NIXLBench, KVBench.

Licensing needs attention: the project advertises Apache-2.0, but the PyPI wheels bundle NVIDIA modules under a proprietary license, and the README warns that third-party components are downloaded and installed during use, so downstream users should review those terms README, NVIDIA license.