vercel-labs/agent-desktop

What it does

Agent Desktop gives AI agents a way to read and drive real desktop applications without pixel matching. A snapshot returns JSON describing an app's accessibility tree, and every element an agent can act on carries a ref. The agent acts on a ref, then snapshots again to verify. The control loop stays in your agent; the tool ships as a Rust CLI plus a C library and language bindings for macOS, Windows and Linux.

This suits agent builders and automation developers who need reliable computer use on native apps, especially when screen coordinates are brittle. It is not a vision model or a ready-made assistant: it is the actuator and observer layer that an agent calls.

Quick start

Install the CLI with npm, Bun, Yarn or pnpm, then check the version:

npm install -g agent-desktop
agent-desktop version

On macOS, grant Accessibility permission first:

agent-desktop permissions --guide

Pick an app from agent-desktop list-apps, then run the observe-act-observe loop:

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

A trimmed snapshot result looks like this:

{
  "version": "2.5",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "window": { "id": "w-204749", "title": "Downloads" },
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}

To wire this into an agent, install the bundled skill from skills.sh with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop, or add instructions to AGENTS.md telling the agent to run agent-desktop skills get desktop and agent-desktop skills get platform before first use. The binary also serves version-matched docs through agent-desktop skills get desktop, agent-desktop skills get platform and <command> --help.

For direct execution without the JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

Ref and error semantics

Refs belong to the snapshot that produced them. A bare @e8 is rejected with INVALID_ARGS. If the UI changed, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent must take a new snapshot. Several candidate windows also produce AMBIGUOUS_TARGET; the snapshot lists window ids in error.details, and you pass one with --window-id. Dense apps are easier to map with snapshot --skeleton --app Slack -i --compact, then drilled into with snapshot --root @s8f3k2p9:e3 -i --compact.

Every command prints one JSON envelope containing ok and either data or error. Errors carry a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe. The full contract is in docs/json-output.md. Ref actions are headless by default: they go through the accessibility API without moving the pointer or taking focus. The global --headed flag enables physical input. In PowerShell, quote every ref ('@s8f3k2p9:e8'), because a bare @token is the splatting operator.

Platform requirements

macOS requires Accessibility permission. Windows needs no grant for apps at your terminal's integrity level. Linux needs the user's graphical session with at-spi2-core.

Capability varies by OS: accessibility trees and click/type/keyboard work on all three. Mouse input on Linux is limited to X11 with --headed. Linux screenshots cover X11 windows and a Wayland portal; clipboard on Wayland needs wl-clipboard. App and window management on Linux is launch/list plus X11 window control. Notifications on Linux require GNOME Shell. The macOS shell surface command is planned. Full per-OS details live in the platform notes, and agent-desktop skills get platform prints the capability table for the current machine.

Privacy and data handling

Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider and is separate from the native CLI; the project documents which data can hold private information in its FAQ.

HotRepo's take

On the 2026-11-01 Rust monthly trending record, the repository stood at 5,606 stars and 416 forks, with 3,699 stars gained in that window and a best position of 11. The star gain is large relative to the total, which points to a burst of attention during that period rather than a long accumulation. The accessibility-tree approach is the technical detail worth noting: using stable refs and explicit stale/ambiguous error codes makes retries safer than coordinate-based automation, which is exactly the failure mode agent builders run into. Treat the ranking as a signal of current interest, not as evidence of production reliability across all three platforms.