microsoft/onnxruntime

ONNX Runtime is an execution engine for ONNX-format models: it loads models exported from PyTorch, TensorFlow/Keras, scikit-learn, LightGBM or XGBoost and runs inference across different hardware, drivers and operating systems, applying hardware accelerators plus graph optimizations and transforms to improve performance. A separate training path can accelerate training of transformer models on multi-node NVIDIA GPUs with a one-line addition to existing PyTorch scripts.

How it works

The project does not publish its own model weights; it consumes the ONNX interchange format and executes it. Practically, that means the model you trained in one framework is exported to ONNX once and then served by this runtime, rather than keeping the original framework around at inference time. Speedups come from two mechanisms the README names: targeting available hardware accelerators, and rewriting the computation graph through optimizations and transforms.

The two use paths

The inference path is the primary one: export your model, then run it through ONNX Runtime on the CPU or GPU you have. Documentation and a companion examples repository cover this path (inference docs, inference examples). The training path is narrower: it targets transformer models on multi-node NVIDIA GPU setups and is added to existing PyTorch training scripts rather than replacing them (training docs, training examples).

A 2023 Hacker News commenter described ONNX as the framework-independent interchange format and ONNX Runtime as a performant runtime for running those models (comment); that is a third-party opinion from an older thread, not a benchmark.

Requirements and limits

• Compatibility is stated broadly (different hardware, drivers, operating systems), but the README does not enumerate which accelerators or platform combinations are validated; the linked docs are the place to confirm a specific target (docs).
• The WebGPU backend discussed in 2023 was noted as implementing only a few operations at the time, with browser-based LLM inference not yet feasible (comment); treat browser GPU support as operation-dependent and check the current documentation rather than that old snapshot.
• Training acceleration is scoped to transformer models and multi-node NVIDIA GPU configurations.

Project and license

ONNX Runtime is MIT-licensed (LICENSE). The README says the project may collect usage data and send it to Microsoft, with details in the privacy statement (privacy statement). Release notes are on GitHub (releases) and upcoming release dates and features are on the public roadmap (roadmap).

HotRepo's take

Hacker News discussion in 2023 shows interest in the WebGPU backend and in interoperability; it does not establish current performance or backend completeness. Trending records are too sparse to judge momentum, so they shouldn't drive a choice here.