ComputerUseAgent/AutoCua
AI EngineeringDrives real desktops, browsers, Android and iOS devices with an LLM agent team, for automation and exploratory testing.
What it is
AutoCua is a Python framework in which a language model operates actual software rather than an API: macOS, Windows or Linux desktops, your own Chrome, iOS simulators or paired iPhones, and Android emulators or phones. A single mode string selects the surface, and the rest of the call stays the same. Behind a task sits a hierarchy of agents — a parent that owns the screen, a coder that works in a real terminal, and browser, iOS and Android sub-agents — each in its own process, with a memory-compression agent that summarizes the transcript when context grows long.
The desktop path does not ask the model for coordinates. Each step fuses the OS accessibility tree with OCR into a numbered element list plus an annotated screenshot, and rejects numbers the model was never shown. Input is delivered at the OS input layer: kCGHIDEventTap on macOS and pywinauto's real SendInput on Windows, with an optional Interception kernel driver for UAC and UIPI cases. The browser agent is a single Rust crate shipped as a compiled Python extension; with a window it drives your logged-in Chrome through an extension, and headless runs use the DevTools protocol on a browser of the agent's own.
Quick start
Python 3.10+ and a provider API key are required. Install from PyPI (package page):
pip install AutoCua
Then create main.py with one run_agent call, choosing the mode, provider and model (surface selection):
from AutoCua.agent_launcher import run_agent
run_agent(mode="browser", provider="openrouter", model="gemini-3.8-flash",
task="Open wikipedia.org and summarise today's featured article.")
Run it, with your key in a .env beside the script or entered into the app's settings:
python main.py
Model names are case-sensitive and must come from model_list.txt. A full run's state — chats, browser profiles, the credential vault, skills — is written to an AutoCua_data/ folder in the directory you ran Python from. Adding ui=True opens a desktop window instead of running in the terminal. A checkout-based install uses platform setup scripts and follows the same main.py pattern (requirements).
Setup gaps to plan for
Several pieces are deliberately not in the wheel and are fetched only with your consent, or set up by hand:
- Linux desktop use needs GNOME (tested on Wayland; X11 untested) and distro packages for AT-SPI, GTK and the XDG RemoteDesktop portal; the README gives the
aptline (Linux notes). - iOS needs full Xcode and a manually cloned WebDriverAgent at the pinned
v15.1.1tag; iOS automation stays paused until the folder exists (iOS setup). - Android emulator downloads Google's emulator (~2.5 GB) after you accept the SDK terms; Windows needs the Windows Hypervisor Platform and Linux needs KVM access (Android setup).
- Windows UAC handling is the only feature that requires the third-party Interception kernel driver, which you install yourself as Administrator; without it every mode still runs but stops at an elevation prompt (UAC section).
- Browser mode needs Google Chrome installed; AutoCua never downloads it.
Constraints and known limits
Desktop, iOS and Android emulator runs need a logged-in graphical session; headless=True applies to Chrome only. The authors state plainly that this is a language model driving real software: two runs can take different paths, and it is suited to exploratory testing, smoke flows and repetitive back-office work rather than replacing deterministic test suites — there is no assertion framework. The credential vault fills secrets without putting them in the model's context, but the store is plain JSON on disk, and fields that echo their value back on screen are not protected. API keys, a Telegram bot token and settings live in a .env; data location can be moved with AutoCua_DATA_DIR. Code is MIT licensed, while redistributed or fetched components (Electron, WebDriverAgent, Interception, Rust crates) carry their own terms documented in THIRD_PARTY_NOTICES.md.
HotRepo's take
The project's public footprint is thin: a single Show HN thread from November 2026 with 4 points and one comment, in which the submitter highlights local execution, the OCR-plus-accessibility scan, canvas and UAC support, and parallel daisy-chained agents (discussion). No GitHub daily-rank record or star-curve samples are available here, so the 1,282 stars/54 forks cannot be paired with observed growth. What the repository does show is an unusually detailed account of how each surface is driven — input-tap choices, browser extension versus CDP paths, per-platform setup prerequisites — which is more useful for judging fit than any popularity signal. The cost of that breadth is a heavy native footprint: Rust builds for source installs, platform-specific driver and desktop dependencies, and optional multi-gigabyte mobile toolchains.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
Hacker News
You might also like
- vercel-labs/agent-desktopA CLI and C library that lets AI agents read and operate desktop apps via the OS accessibility tree.
- ctoldigital/axuseAn MCP server that lets coding agents drive browsers and desktops through accessibility trees instead of screenshots.
- browser-use/browser-usePython library that lets an LLM drive a real browser via CDP to complete web tasks.
- minitap-ai/mobile-useAn LLM-driven agent that operates Android and iOS apps via natural language commands and returns structured data.
- mylea1/phongentControls multiple Android device agents from an iOS or Android master device with natural-language tasks and live screens.