orailnoor/cross-platform-llm-client

What It Does

PrivateLM is a Flutter-based chat client that lets one app handle both local and cloud language models. On Android and iOS it loads GGUF models and performs inference on-device through a custom plugin wrapping llama.cpp; on the web it is cloud-only. When you need more capability, the same interface routes requests to OpenAI, Anthropic, Google Gemini, or Moonshot's Kimi.

The client supports text and image messages, with vision available through local Qwen2-VL models as well as cloud providers. Chats and settings persist locally in Hive, and API keys are stored there too, only sent to the provider endpoint you choose. First launch reads device RAM and suggests context size and token limits. A task view for multi-step workflows is described as in development and not yet usable README lines 15–26.

Local Inference Pipeline

On supported devices the app detects Vulkan GPU capabilities to decide how many layers to offload, picks a thread count from a device tier, loads the GGUF model with progress reporting, and generates tokens using native chat templates (ChatML, Llama-3, Gemma, Phi), falling back to manual prompt construction if a template fails. Idle detection after 5 seconds and a 180-second hard timeout keep the UI responsive on weaker hardware README lines 66–77.

Local inference is conditionally compiled: Android uses the full llama.cpp engine, iOS uses it with Metal GPU acceleration, and web swaps in a stub, so local-model UI is hidden where unsupported README lines 89–100.

Quick Start

To run the app from source, install the Flutter SDK 3.3.0 or newer, the Android SDK with API 26+, and JDK 17. Then fetch dependencies and build a debug APK:

flutter pub get
flutter build apk --debug

For a release build, copy android/key.properties.example to android/key.properties, fill in the keystore values, and build split-per-ABI APKs; the signing key must stay consistent across releases and the build number must increase README lines 142–161.

If you only want to try the interface, the project links a hosted web app, which supports cloud APIs but not local inference Live Web App. iOS users can install the sideloaded IPA through AltStore, with the caveat that sideloaded apps expire after 7 days unless AltStore refreshes them README lines 109–132.

Constraints

• Local inference is Android and iOS only; the web build is cloud-only.
• GGUF model performance depends on device RAM and GPU; the README recommends iPad over iPhone for local models.
• Android requires minSdk 28; API 26+ is the build prerequisite.
• Release APKs need a stable signing key, otherwise upgrades over an installed build will fail.
• API keys live in local Hive storage and are sent only to the chosen provider endpoint.
• The GitHub releases page distributes prebuilt artifacts such as the iOS IPA package Releases.
• Licensed under MIT LICENSE.

HotRepo's Take

On the 2026-11-06 C++ daily list, the repository reached first place with 137 stars gained that day, taking it to 1,525 stars and 317 forks. That single snapshot shows strong same-day attention in the C++ list, but no broader trend can be read from one record. The project's combination of on-device llama.cpp inference and multi-provider cloud fallback is the concrete reason it stands out from cloud-only chat clients; whether that holds depends on the shipped binaries matching the described implementation.