google-ai-edge/LiteRT-LM

What it does

LiteRT-LM is Google's orchestration layer for running LLMs on edge hardware through LiteRT. It handles model loading and execution across Android, iOS, the web, desktop and IoT boards, and exposes GPU and NPU acceleration, vision and audio inputs, function calling for agentic flows, and support for Gemma, Llama, Phi-4 and Qwen models (README, technical overview). Google reports it powers on-device GenAI in Chrome, Chromebook Plus and Pixel Watch, so it is aimed at teams shipping models inside an app rather than at GPU servers.

Quick start

You can try a model from a terminal without writing code. Install the uv package manager, then install the CLI and run a model pulled from Hugging Face:

uv tool install litert-lm

litert-lm run \
  --from-huggingface-repo=google/gemma-3n-E2B-it-litert-lm \
  gemma-3n-E2B-it-int4 \
  --prompt="What is the capital of France?"

The answer is printed to the terminal. The CLI also supports backend selection, speculative decoding and a Hugging Face repo argument, as shown in the Gemma 4 example with --backend=gpu --speculative-decoding=on (CLI guide). For app integration, use the language guides: Python, Kotlin, Swift, JavaScript, Flutter or C++. Building from source requires checking out the latest release tag and following the build and run guide.

Language and platform support

Python, Kotlin and C++ are marked stable; Swift, JavaScript (web) and Flutter are early-preview or community status (README). The web SDK (@litert-lm/core) stream-loads models, defaults to the GPU backend and accepts image and audio inputs. Speech models including Qwen3-ASR, Kokoro and Qwen3-TTS are supported in v0.19.0.

Constraints to know

• The C API 1.0.0 is a breaking change: fallible functions now return LiteRtLmStatusCode with results in out_* parameters, and only litert_lm_* symbols are exported. Code written against older C bindings needs updating.
• Device backends matter: GPU/NPU acceleration availability and performance depend on the target platform and hardware; the examples choose a backend explicitly with --backend=gpu.
• Models must be in the .litertlm format (or fetched from a prepared Hugging Face repo); arbitrary checkpoints cannot be passed directly.
• Early-preview languages (Swift, JavaScript, Flutter) may change before stabilization.
• The project is Apache-2.0 licensed.

HotRepo's take

In the daily C++ ranking recorded on 2026-11-01, LiteRT-LM sat at position 10 with 6,791 stars and 9 new stars that day. That position is a same-day, same-language figure, and the single snapshot only establishes that date's standing, not a trend. The notable technical signal is the shipping footprint the README names — Chrome, Chromebook Plus and Pixel Watch — which puts this framework in the small group of on-device inference stacks with first-party production use rather than only demos. With stable C++, Python and Kotlin APIs alongside preview Swift and JS bindings, it suits teams that need one runtime across mobile, browser and embedded targets.