zai-org/Open-AutoGLM
AI EngineeringA phone-use agent framework that drives Android and HarmonyOS devices from natural-language instructions via a vision-language model.
What it is
Open-AutoGLM is a phone agent framework built around the AutoGLM vision-language model. It reads the screen, plans the next action, and executes it through ADB (Android) or HDC (HarmonyOS) so a user can describe a task in natural language and let the agent drive the device. It also works with iOS through WebDriverAgent, documented separately.
The loop is: screenshot → model interprets the UI → emits an action (Launch, Tap, Type, Swipe, Back, Home, Long Press, Double Tap, Wait) → the control tool performs it. Login pages and verification codes trigger a human takeover, and sensitive actions can be gated by a confirmation callback. Remote control over WiFi is supported, as is selecting a specific device by ID.
The repository ships the agent and client code, not the model. Two model variants are offered: AutoGLM-Phone-9B tuned for Chinese apps and AutoGLM-Phone-9B-Multilingual for English and mixed-language screens (model links).
Quick start
Prerequisites: Python 3.10+, an Android 7.0+ or HarmonyOS device with developer mode and USB debugging enabled, and a reachable model service. On macOS, install ADB from platform-tools and add it to PATH:
export PATH=${PATH}:~/Downloads/platform-tools
adb devices
The output should list your device; an empty list or unauthorized means the phone has not authorized USB debugging. On Android, install ADB Keyboard and enable it as an input method, otherwise text entry will not work.
Then install the agent:
git clone https://github.com/zai-org/Open-AutoGLM.git
cd Open-AutoGLM
pip install -r requirements.txt
pip install -e .
Point it at a model service. Either use a hosted endpoint (Zhipu BigModel or ModelScope) with an API key, or start your own OpenAI-compatible server with vLLM or SGLang using the parameters in the README. With vLLM locally, the service is served at http://localhost:8000/v1. A deployment check script is provided:
python scripts/check_deployment_cn.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b
A healthy deployment returns a thinking block plus an action such as <answer>do(action="Launch", app="京东"); very short reasoning or garbled output usually means the model service is misconfigured.
Finally, run a task:
python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "打开美团搜索附近的火锅店"
Running main.py without a task enters interactive mode. For HarmonyOS add --device-type hdc; for a hosted API key add --apikey. The same flow is available from Python via PhoneAgent and ModelConfig from the phone_agent package. Use python main.py --list-apps to see the mapped apps (50+ on Android, 60+ on HarmonyOS).
For iOS, follow the iOS setup guide; it requires WebDriverAgent rather than ADB.
Constraints
- Model host is an external dependency. The agent code contains no model; you need either a hosted API or a local GPU deployment (README suggests 24 GB+ VRAM and roughly 20 GB of model download).
- Device permissions matter. Some phones need both USB debugging and "USB debugging (security settings)" for taps to work. A data-capable USB cable is required.
- Sensitive screens. Payment, banking, and password pages typically return black screenshots; the agent detects this and requests human takeover rather than continuing.
- Scope and terms. The project is stated as research and learning use only, with a separate usage terms document; using it to harvest information or disrupt systems is explicitly out of scope.
- Platform coverage. Native support is for Android and HarmonyOS, with iOS through a separate WebDriverAgent path.
HotRepo's take
Open-AutoGLM is the open framework complement to Zhipu's phone-agent model release, and its structure is straightforward: the repo handles device control, prompt templates, and the action loop, while the vision model supplies the reasoning. That split is what makes it usable without a GPU if you are willing to call a hosted endpoint, and it is the first thing to understand before deploying. The README's emphasis on verifying adb devices and installing ADB Keyboard reflects the real failure modes of screen-driving agents on consumer phones: device authorization and text input. No HotRepo ranking or growth records are available for this repository, so no trend claim is made here.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
Hacker News
You might also like
- phone-harness/phone-harnessA Python CLI that lets coding agents tap, type, scroll, and read real iPhones or Android phones.
- tapilot-ai/tapilotAn experimental Android app that executes plain-language phone tasks using a vision model and accessibility control.
- phoneuse-official/phoneuseOpen-source tools to drive a USB-connected iPhone from a Mac with prompts, live video, REST and MCP.
- mylea1/phongentControls multiple Android device agents from an iOS or Android master device with natural-language tasks and live screens.
- droidrun/mobilerunAn LLM-agnostic framework that lets AI agents inspect, tap, swipe, and type on real Android and iOS devices via CLI or Python.