minitap-ai/mobile-use
AI EngineeringAn LLM-driven agent that operates Android and iOS apps via natural language commands and returns structured data.
What it does
mobile-use drives a real phone or emulator by reading the UI accessibility tree and acting on it, so tasks written in plain language (open Gmail, read unread mail, navigate settings) can be executed without hand-written scripts. Beyond acting, it can return scraped output in a caller-specified format such as JSON. The project is a Python 3.12+ LangGraph-based agent system with configurable LLM providers, aimed at developers building mobile automation, QA, or data-extraction workflows rather than end users tapping through a consumer app.
The maintainers report a 100% result on the AndroidWorld benchmark and describe the multi-agent task-decomposition architecture in an accompanying paper Benchmark page paper. These are the authors' claims; the benchmark is theirs to interpret.
Quick start
Fastest path is the Docker-based launcher for Android, which requires Docker, a USB-connected device with USB debugging enabled (or a running emulator), and the device on the same Wi-Fi network as the computer:
chmod +x mobile-use.sh
bash ./mobile-use.sh \
"Open Gmail, find first 3 unread emails, and list their sender and subject line" \
--output-description "A JSON list of objects, each with 'sender' and 'subject' keys"
On Windows the equivalent is powershell.exe -ExecutionPolicy Bypass -File mobile-use.ps1 ... with the same arguments Docker quickstart. Accept the ADB prompts that appear on the phone. If the script cannot resolve the device IP, inspect the interface with adb shell ip addr show up and pass --interface <NAME>.
For development, clone the repo, copy .env.example to .env and fill in API keys, then:
uv venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
uv sync
python ./minitap/mobile_use/main.py "Go to settings and tell me my current battery level"
Model/provider selection is configured through llm-config.override.jsonc; OpenAI-compatible endpoints, MiniMax, Anthropic Claude, and Google Vertex AI credentials are covered local quickstart setup.
Constraints
- Supported targets: physical Android phones and Android emulators, plus iOS simulators on macOS. Physical iOS devices are not yet supported.
- Android needs ADB; iOS simulator automation on macOS needs Xcode and
idb_companion(installable via Homebrew) on the PATH. - Docker launcher is currently Android-only; the device must share the computer's Wi-Fi, and firewall blocking of port 5555 is a common failure.
- UI-aware navigation depends on the accessibility tree, so games and apps with weak accessibility data are explicitly noted as limited.
- An LLM provider key is required; without a configured model the CLI prompts for a choice.
- Licensed Apache-2.0; the maintainers request credit to Minitap and a citation when used in research or commercial products.
HotRepo's take
No HotRepo ranking or snapshot history is recorded for this repository, so there is no growth or leaderboard trend to read here. What stands out in the material is the scope of the claim — a 100% AndroidWorld score from a small team with a self-published benchmark page and paper — combined with a project that is still labeled as rapidly evolving. Treat the benchmark as the authors' result and evaluate it on your own devices and app set; the practical differentiator to check is real-device reliability and the cost of the multi-agent LLM loop.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
Hacker News
You might also like
- droidrun/mobilerunAn LLM-agnostic framework that lets AI agents inspect, tap, swipe, and type on real Android and iOS devices via CLI or Python.
- AcidGr/agent-mobile-useRuns Android apps on a background virtual display so an agent can operate them without touching the main screen.
- mylea1/phongentControls multiple Android device agents from an iOS or Android master device with natural-language tasks and live screens.
- callstack/agent-deviceA CLI, MCP server, and typed Node.js API that let AI coding agents inspect, control, and verify apps on mobile, TV, web, and desktop devices.
- tapilot-ai/tapilotAn experimental Android app that executes plain-language phone tasks using a vision model and accessibility control.