onehumanbeing/esp32-iphone-agent

The problem

Automating an iPhone end to end normally means jailbreaking it, running a Mac as a mirroring host, or paying for a device farm. Anyone who wants an LLM to actually operate a phone — tapping through Settings, setting an alarm, driving an app — hits the same wall: iOS gives you no sanctioned input path and no screen access for third-party code. This project splits the job across two cheap pieces of hardware: the ESP32-S3 receives the screen over AirPlay and impersonates a USB mouse and keyboard, while an app on the same phone does the seeing and deciding.

What's different

Compared with the usual approaches, the trade-off is architectural rather than incremental. Device farms and cloud phone services give you reliable, scriptable access but at per-minute cost and no physical phone. Jailbreak-based automation unlocks the system but breaks on updates and voids the device. Mac-hosted mirroring (the project's own v0.1.0 used UxPlay plus a Python server) works but keeps a computer in the loop. Here the computer disappears: the board is the AirPlay sink and the HID device, the phone is the brain, and nothing else is required at run time. Two details stand out — the board passes H.264 through without decoding, and the loop is closed by re-homing the AssistiveTouch pointer and re-checking the screen after each action, which is what makes relative mouse input usable for taps and swipes.

Who it's for

Hobbyists and researchers who want to experiment with computer-use agents on real iOS hardware and are comfortable with Arduino tooling and Xcode. It suits single-device, personal-scale automation: saved tasks triggered by Siri or Shortcuts, or collecting run traces for agent evaluation. Look elsewhere if you need throughput, parallelism, or unattended reliability — the README itself caps the link at roughly 300–500 kbps and 480×1040 at ≤5 fps, and notes that passcode, Apple Pay and DRM screens are invisible to the agent, as is anything after the phone locks. Teams wanting a supported automation API should not start here.

Cost of adoption

You need an ESP32-S3 with native USB and PSRAM (the tested XIAO ESP32S3 is about $7), an iPhone on iOS 18+ with cellular data, a Mac with Xcode for the one-time flash and app build, and an OpenAI-compatible vision endpoint. Two signing caveats matter: a free Apple ID re-signs every 7 days, and the one-tap hotspot join needs a paid account. The app also uses a private iOS API (suspend) to leave itself, which the README flags as unfit for App Store distribution. Keyboard input is ASCII only and landscape apps are unsupported. MIT licensed, but the README asks that ports and demos credit the author.

Verdict

The repo is 13 days old with about 3,790 stars and roughly +291 per day, so this is a brand-new project riding a launch spike, not an established tool — the Hacker News thread drew only 3 points, which suggests the attention is concentrated on social channels rather than broad validation. The engineering is genuinely interesting: moving the AirPlay sink and the HID device onto a $7 board and letting the phone host the agent is a real simplification of the earlier Mac-based design. But the limitations are structural to the BLE link and iOS itself, not things a few commits will fix. Treat it as a well-documented prototype to build on or learn from; do not build anything you depend on until the landscape, non-ASCII input and App Store questions are settled.