Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

kilolonion/agent-vscreen

AI Engineering

Run Android apps on an invisible virtual display so phone AI agents can operate them while you keep using your own screen.

117
18
Kotlin
MIT
10 days old

What it does

Agent-VScreen gives an Android AI agent a second, invisible display. The agent launches apps, taps, swipes, types and presses keys on that virtual display, while your own screen stays free. Each action returns the current screen, visible text, tappable elements and a screenshot, so the agent can verify each step README.

The app itself calls no LLM: the caller's model decides what the screen means. It exposes the display through three interfaces — a vscreen CLI, MCP and plain HTTP — so terminal-capable agents, MCP clients and custom code can all use it. A small floating window mirrors the background screen and lets you take over or stop the AI at any time.

Around the display it adds time-aware screen readings, until conditions for waiting on text changes, per-app block/read-only rules, payment confirmations, multi-agent sessions and a "teach by doing" recorder that turns your taps into replayable step segments. The project targets Android phone agents rather than desktop automation.

Quick start

Requirements: an arm64 Android 11+ device (14+ recommended) with Shizuku running, and the APK from Releases.

  1. Install the APK and open Agent-VScreen.
  2. With Shizuku running, tap Grant Shizuku.
  3. Tap One-tap setup to grant overlay, notifications, read-only accessibility, battery optimization and background running. On Honor, Huawei, Xiaomi, OPPO, vivo and similar phones, also allow auto-start and background activity in the system background management.
  4. In Connect agent, tap Copy message for agent and send it to your agent. It installs the vscreen command and saves a short usage guide into the agent's long-term memory.

Then try:

vscreen launch Settings
vscreen act "tap Search; type About phone; key enter until text:About phone"
vscreen help all

For an MCP client such as Operit, the default endpoint is http://127.0.0.1:18765/mcp with httpStream transport and a Bearer token copied from the app; this only works when client and app are on the same phone MCP setup notes. The full CLI, MCP and HTTP reference is in API.md.

Constraints and safety

  • The service listens only on 127.0.0.1:18765 by default. Remote access requires choosing Tailscale (built-in node or the Tailscale app) or SSH port forwarding; there is no public link.
  • The MCP token equals control of the phone. Do not paste it into shared configs, public cards or screenshots.
  • The app UI is currently Chinese, with an English switch available; the button labels in this guide are translated.
  • Privileges come from Shizuku alone; no root, no adb, and no wireless debugging pairing is required.
  • The background display's apps are muted. Vendor background managers may not let exact alarms and wake broadcasts through for frozen apps, and audio muting is not verified on every player and ROM.
  • Permission levels range from "hands-off" (confirm only money-spending steps) to "careful" (also confirm irreversible actions), with per-app rules and per-agent tokens.
  • MIT licensed; the bundled PaddleOCR models keep their own Apache-2.0 license.

Build from source

JDK 17 and the Android SDK (compileSdk 36) are needed:

echo "sdk.dir=/path/to/android-sdk" > local.properties
./gradlew :app:assembleRelease

Built-in Tailscale is an optional Go program; run tsbridge/build.sh first to produce libtsbridge.so, or the APK builds without that feature.

HotRepo's take

No daily-rank or star-history records are available for this repository, so there is no in-site trend to report. What stands out technically is the architecture: the virtual display is delegated through CLI, MCP and HTTP, perception is merged from the accessibility tree and on-device OCR, and every step returns a structured, verifiable result. That makes it a general substrate for Android phone agents rather than a single-agent integration — the Operit notes are explicitly just one example. The release notes also list real-phone regression tasks on an Honor Magic 7, which gives the stability claims a concrete test basis. The trade-offs are the Chinese-only UI and reliance on Shizuku and vendor background policies, both of which affect first-run setup more than the API itself.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. AcidGr/agent-mobile-useRuns Android apps on a background virtual display so an agent can operate them without touching the main screen.Java124
  2. guzhou079-arch/deepseek-harness-androidAn Android APK that bundles DeepSeek Harness with screen automation, virtual display, scheduled tasks and 27 skills, without Termux or root.JavaScript110
  3. KnifeLemon/DroidlineAutomate and test real Android phones from Python, Node.js, C#, CLI, HTTP or Appium without ADB, USB or root.Go547
  4. ybblab/DroidanceAn open-source Android agent that sees the screen and taps through apps for you in a background virtual display.Kotlin151
  5. kuafuai/andeeAn on-device agent for Android and Mac that operates any app through accessibility, operating apps without integrations or root.Kotlin6,150