Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

TencentQQGYLab/AppAgent

AI Engineering

LLM multimodal agent framework that controls Android apps by tapping and swiping via ADB, for GUI automation researchers.

6,923
760
Python
MIT
2 years old

What it is

AppAgent is a research framework from Tencent that lets a multimodal LLM drive an Android app like a human user: it takes screenshots, identifies labeled UI elements, and issues tap/swipe actions through ADB. Because it works through the accessibility-free route of screen capture and touch simulation, it does not need app backend access and can be applied to third-party apps.

The key idea is a two-phase workflow. In exploration, the agent either navigates the app autonomously or learns from a human demonstration, producing a documentation base describing UI elements. In deployment, a new task is solved by retrieving that documentation. The paper and demo materials are linked from the project page.

Quick start

Prerequisites: Python 3, ADB installed on the PC, an Android device with USB debugging enabled and connected, or the Android Studio emulator if no physical device is available (prerequisites).

  1. Install dependencies in the repo root:
pip install -r requirements.txt
  1. Edit config.yaml: set the OpenAI API key and the request interval. Optionally, switch MODEL to Qwen and add a Dashscope key to use qwen-vl-max (configuration).

  2. Generate the knowledge base in the exploration phase:

python learn.py

Choose autonomous exploration or human demonstration and provide the app name and task.

  1. Run the task in the deployment phase:
python run.py

Enter the app name, select the documentation base, and describe the task.

Costs and model options

GPT-4V is the primary model; the README notes that each request/response pair costs roughly $0.03, so long explorations are not cheap. qwen-vl-max is offered as a free alternative with reportedly weaker performance. Supporting another model means writing a new model class in scripts/model.py. The request interval in config.yaml throttles calls to stay within account limits.

Usage constraints

  • Android only; the agent controls the device over ADB, so USB debugging must stay enabled. An emulator is supported but must be detectable by ADB.
  • Task success depends heavily on the generated documentation. The tips recommend broader exploration or more demonstrations, and manual correction of inaccurate element descriptions.
  • The project ships an evaluation benchmark used in the paper's testing.
  • MIT licensed.

HotRepo's take

AppAgent packages the exploration-then-deployment pattern into runnable scripts with a config file and a documented evaluation set, which makes the method reproducible rather than only described in a paper. The explicit per-request price note and the fallback to a free Qwen model are practical details that shape how long an exploration run can be before cost becomes the binding constraint. It is aimed at people studying or building GUI agents, not at end users who want a polished phone assistant.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. AAswordman/OperitAn open-source Android AI agent app that connects cloud or local models to tools, terminals, browsers, and device automation.Kotlin9,283
  2. zai-org/Open-AutoGLMA phone-use agent framework that drives Android and HarmonyOS devices from natural-language instructions via a vision-language model.Python26,616
  3. kilolonion/agent-vscreenRun Android apps on an invisible virtual display so phone AI agents can operate them while you keep using your own screen.Kotlin117
  4. AcidGr/agent-mobile-useRuns Android apps on a background virtual display so an agent can operate them without touching the main screen.Java124
  5. X-PLUG/MobileAgentA family of GUI agents and GUI-Owl models that automate desktop, mobile, and web apps via screenshots and instructions.Python9,385