Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

tapilot-ai/tapilot

AI Engineering

An experimental Android app that executes plain-language phone tasks using a vision model and accessibility control.

138
28
HTML
MIT
57 days old

What it does

Tapilot lets you describe a task in plain language and have an Android phone carry it out: tapping, swiping, scrolling, typing, and switching between apps. It sits on top of Midscene for screenshot-based planning, element locating, action execution and reports, while Tapilot itself provides the native Android UI, the accessibility device adapter, model transport and task lifecycle. The whole orchestration runs in an embedded Node.js worker on the phone, but model inference is remote — on-device orchestration does not mean offline inference.

It is aimed at people who want phone automation without ADB, Shizuku or root. During execution you can add instructions, stop the task, take over manually, and then continue with the saved context; afterwards you can review an HTML replay containing screenshots, plans, actions and errors. Optional voice input/output is available through Volcengine or Deepgram.

Quick start

Prerequisites: an arm64 phone running Android 11 or newer, plus a supported vision model reachable from the phone.

  1. Get the app from the Beta APK or the GitHub releases fallback, and install it.
  2. Open Tapilot and enable Tapilot device control in Android Accessibility settings.
  3. In settings, configure an HTTP or HTTPS vision-model API (or the ChatGPT SIWC option where available) and enter credentials on the phone. For a LAN model, use its OpenAI-compatible base URL such as http://192.168.1.100:8000/v1; localhost refers to the phone, and a non-empty placeholder API key (e.g. local) is still required.
  4. Run a visible task such as “Open Calculator and calculate 12 × 8”, then check the result on the phone screen and open the execution report.

New installations default to automatic execution (YOLO). If you want confirmation for critical or every operation, choose that mode in the composer before sending. Configured credentials do not constitute a successful connection test — the first real task exercises the model, screenshots and actions.

Build from source

Host requirements are Node.js 24, pnpm 12.4.1, JDK 17, Android SDK Platform 35 and Build Tools 35.0.0, with JAVA_HOME and ANDROID_HOME set. The documented flow is:

pnpm install --frozen-lockfile
npm ci --prefix plugins/tapilot
node plugins/tapilot/scripts/patch-midscene-zod.mjs plugins/tapilot
npm ci --prefix tests/midscene
node plugins/tapilot/scripts/patch-midscene-zod.mjs tests/midscene
pnpm build:android
pnpm check:quick

The local APK lands at app/build/outputs/apk/beta/app-beta.apk and uses development signing, so it cannot update a differently signed installation. Full host prerequisites and signing setup are in the contribution guide.

Constraints and data handling

  • Operating the visible screen means the phone is occupied; let a task finish or stop it before using other apps. Interrupted tasks do not replay automatically after an app restart.
  • A model can misread screens, infer unobserved facts or return incorrect completion answers; locked or protected screens, OEM background restrictions and custom-rendered text inputs can block execution. Only arm64 devices are bundled today.
  • Screenshots and task content go to the selected model provider; optional speech features send audio or reply text to the speech provider. Keys and tokens are stored in Android Keystore-backed storage, which does not make inference local.
  • HTTP transmits API keys, messages and screenshots unencrypted, so use it only on a trusted network and prefer HTTPS otherwise. Reports contain screenshots, prompts and model output; the phone keeps at most 20 report files and 200 MB, so export what you need to keep.
  • Wi-Fi control from a computer uses the Tapilot plugin over pinned TLS and needs an unlocked phone with exclusive control.
  • The background-screen mode starting in 0.1.9 is experimental, off by default, and requires explicit shell authorization, phone-side pairing and overlay permission; see its setup and limits.
  • Licensed under MIT, with third-party components under their own licenses (credits, third-party licenses).

HotRepo's take

There is no recorded HotRepo ranking, star-curve sample or discussion thread for this repository, so the available signal is the project materials themselves. What stands out is the honesty of the limits: the README separates orchestration from inference, notes that 23 authored E2E cases do not imply 23 verified passes, and states that historical device runs do not establish a pass for the latest source. Combined with an MIT license and a documented build path, that makes the project worth evaluating on its own docs — but the practical gate is the APK distribution, which the repository materials do not tie back to a specific source build, so check the release artifacts and provenance before installing on a phone you rely on.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. zai-org/Open-AutoGLMA phone-use agent framework that drives Android and HarmonyOS devices from natural-language instructions via a vision-language model.Python26,616
  2. guzhou079-arch/deepseek-harness-androidAn Android APK that bundles DeepSeek Harness with screen automation, virtual display, scheduled tasks and 27 skills, without Termux or root.JavaScript110
  3. phone-harness/phone-harnessA Python CLI that lets coding agents tap, type, scroll, and read real iPhones or Android phones.Python5,051
  4. citrolabs/ego-liteA browser that gives each AI agent its own isolated Space while sharing your real logins.JavaScript22,824
  5. mylea1/phongentControls multiple Android device agents from an iOS or Android master device with natural-language tasks and live screens.Kotlin362