Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

ja1chen/actell

AI Engineering

An experimental iPhone agent that reads the screen and sends taps, drags, and waits through CoreDevice.

124
38
Swift
GPL-3.0
40 days old

What It Does

Actell is a prototype iOS app for building an agent that operates an iPhone. The agent observes the current screen through screenshots, accessibility content, or both, then acts on the device using screenshot, tap, drag, and wait tools. It reaches the device through a Swift/Rust backend called CoreDevice, over either a VPN loopback route (default) or an ESP32-S3 USB NCM relay.

The app also provides a conversation layer: configurable model providers, streaming replies, readable reasoning, attachments, and local text skills that can be imported or created in conversation. File and shell-style tools (read, edit, write, and a bundled bash backed by ios_system, with embedded Python and bundled Pillow/NumPy) run automatically without confirmation prompts command tool scope. Model context is managed separately from full session history with budgeted compaction AgentKit.

The project is explicitly an early vibe-coded prototype. Screenshots and taps have been demonstrated on controlled devices; complete tasks, drag behavior, and long-running background operation still need separate acceptance.

Requirements and Setup

Build requirements are Xcode 27 and iOS 27 or later, plus your own valid signing configuration. Rust targets aarch64-apple-ios and aarch64-apple-ios-sim are needed for the CoreDevice backend. Enable Developer Mode and local network access, complete the app's pairing flow in the system UI, and prepare a matching Personalized Developer Disk Image through Xcode; the backend only checks availability and does not download, personalize, or mount images iOS build instructions.

Select a connection route in Settings → Phone Control → Connection & Validation:

  • VPN loopback (default): run an external VPN that supports local packet reinjection. Default interface/peer addresses are 10.7.1.1 / 10.7.0.1, which must match your VPN setup. The app does not manage the VPN. LocalDevVPN, Shadowrocket, and Clash compatibility has been tested; see the route guide.
  • ESP32 NCM: connect an ESP32-S3 with the current firmware via its native USB data port; check the hardware conditions and firmware README.

Then add a provider, choose a model, and create a conversation. Control tools validate connection and authorization at invocation time; the route cannot change mid-execution. Ordinary chat works without a device connection.

To build the app:

python3 ios-app/scripts/prepare_python_runtime.py
python3 ios-app/scripts/generate_xcode_project.py
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild \
  -project ios-app/PhoneAgent.xcodeproj -scheme PhoneAgent \
  -destination 'generic/platform=iOS' -derivedDataPath .build/DeviceBuild \
  -clonedSourcePackagesDirPath .build/SourcePackages CODE_SIGNING_ALLOWED=NO build

Tests for the model and runtime packages use swift test --package-path ios-app/ProviderKit and swift test --package-path ios-app/AgentKit; core simulator tests need an isolated simulator UDID. The optional ESP32 route requires ESP-IDF 5.5.5 and is built with bash devices/esp32/tools/build.sh ncm (does not flash).

Constraints

  • The app targets iOS 27 and is not production-ready; background execution and full task completion are not guaranteed.
  • On-device control depends on a working loopback VPN setup or ESP32 firmware and hardware; cellular operation and network switching have separate limitations.
  • Screenshots, accessibility content (which may include off-screen text), and explicitly sent attachments are sent to the configured model provider.
  • Tool effects are auto-authorized by the host: file mutations and bundled commands do not prompt for confirmation, and they can reach paths allowed by the app sandbox.
  • The accessibility reader is limited to a verified system build and does not use XCTest Runner accessibility limits.
  • Licensing is GPL-3.0, with third-party components pinned and attributed in THIRD_PARTY.md.

HotRepo's take

There is no HotRepo ranking, snapshot, or Hacker News discussion record for this repository yet, so there is no trend signal to report. The technically notable part is the split between an App-owned Swift agent runtime, a Rust CoreDevice backend, and two physical transport options, together with the project's own honest acceptance boundaries: screenshots and taps have been demonstrated, but end-to-end tasks and background resilience are not. Readers should treat it as a research prototype and an idea-sharing artifact rather than a usable phone automation product.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. zhongerxin/iPhone-useLet Codex, WorkBuddy or Doubao work drive a physical iPhone over USB through WDA and MCP tools.Python5,022
  2. ybblab/DroidanceAn open-source Android agent that sees the screen and taps through apps for you in a background virtual display.Kotlin151
  3. phone-harness/phone-harnessA Python CLI that lets coding agents tap, type, scroll, and read real iPhones or Android phones.Python5,051
  4. ananhq/anan-iphone-useESP32-S3 firmware that turns an M5Stack StopWatch into an MCP server for sending tap, drag, scroll and keyboard input to an iPhone.C160
  5. hridaya423/phonedriverMCP server and CLI that let AI agents read and control Android phones and iPhones with a confirmation step before payments.Rust150