bytedance/UI-TARS
Machine LearningA multimodal GUI agent model that reasons before acting to operate desktop, browser, mobile and game interfaces.
What it is
UI-TARS is ByteDance Seed's research release of a multimodal GUI agent model: it takes a screenshot plus an instruction, reasons about the state of the interface, and emits an action (click, type, scroll, long-press, app switch, and so on). The current open weights are UI-TARS-1.5-7B, built on a vision-language backbone and trained with reinforcement learning so the model can produce a Thought before its Action. A larger UI-TARS-2 is announced in the updates with a separate technical report, while the 7B checkpoint is the one distributed publicly.
The repository is mainly the inference and post-processing side of the research: the model card and weights live on Hugging Face, deployment instructions are in README_deploy.md, prompt templates for desktop, mobile and grounding tasks are in codes/ui_tars/prompt.py, and a small Python package turns raw model text into executable actions. It suits researchers and engineers evaluating or integrating an end-to-end GUI agent, not end users who just want a desktop assistant — that is the separate UI-TARS-desktop.
Quick start
- Deploy the model and run a first prediction following the deployment and inference guide. This covers serving the Hugging Face checkpoint and issuing a prompt.
- Install the action-parsing package:
pip install ui-tars
# or
uv pip install ui-tars
- Parse a model response into executable
pyautoguicode. Thefactorand origin width/height must match the coordinate convention of the model you are using:
from ui_tars.action_parser import parse_action_to_structure_output, parsing_response_to_pyautogui_code
response = "Thought: Click the button\nAction: click(start_box='(100,200)')"
original_image_width, original_image_height = 1920, 1080
parsed = parse_action_to_structure_output(
response, factor=1000,
origin_resized_height=original_image_height,
origin_resized_width=original_image_width,
model_type="qwen25vl"
)
code = parsing_response_to_pyautogui_code(parsed, image_height=original_image_height, image_width=original_image_width)
print(code)
- Pick a prompt template matching the target environment (
COMPUTER_USE,MOBILE_USEorGROUNDING) from the prompt module and feed the model screenshots plus task text. Because the Qwen 2.5-VL-based checkpoints use absolute coordinates, review the coordinate processing guide when writing your own post-processing.
Evaluation and benchmark results
The README reports OSWorld (100 steps) at 42.5, Windows Agent Arena (50 steps) at 42.1, WebVoyager at 84.8, Online-Mind2web at 75.8 and Android World at 64.2 for UI-TARS-1.5, plus ScreenSpot-V2 at 94.2 and ScreenSpotPro at 61.6 for grounding. It also publishes a per-game table on Poki and Minecraft task averages comparing the model with VPT and DreamerV3. These are author-reported numbers on the named benchmarks; the 7B open checkpoint scores lower than the full UI-TARS-1.5 (OSWorld 27.5 vs 42.5), so results obtained with the public weights will differ from the headline table.
Constraints
- The released 7B model is tuned for general computer use and is not optimized for game scenarios, where the closed UI-TARS-1.5 performs substantially better.
- Large-scale or long-horizon tasks still require substantial compute.
- The model can hallucinate UI elements or take wrong actions in ambiguous environments, and the authors note it can complete CAPTCHA-style challenges, which raises misuse and rate-limit concerns when automating third-party services.
- Access to the top-performing UI-TARS-1.5 model is described as early research access via contact rather than public download.
- License is Apache-2.0 for the repository; confirm the checkpoint's own terms on Hugging Face before commercial use.
HotRepo's take
UI-TARS is a heavyweight research repository: roughly 11.7k stars and 889 forks a few months after its January 2025 creation, which fits an actively benchmarked agent model from a major lab. HotRepo has not recorded it on the all-language GitHub daily leaderboard, so there is no ranked hotness signal to cite; the only discussion on file is a small May 2025 Hacker News thread where a commenter described it simply as an open-source multimodal AI agent with broad tool integration discussion. The practical takeaway from the repository itself is the split between the open 7B checkpoint and the stronger unreleased model — anyone benchmarking against the published table should treat it as an upper bound, not the performance of the downloadable weights.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
Hacker News
You might also like
- Supra-3D/PhysAg3DTurns a single image into a rigged, interactive 3D asset by pairing a 3D generator with an agent that builds and tests joints.
- MiniMax-AI/MiniMax-H3-ImageOpen-weights 16-megapixel image generation and editing model with a 33.1B DiT and 6-step sampling.
- solidsf-inc/solidcvA pure-Rust computer-vision crate aimed at replacing C++ OpenCV builds and speeding up vision-language model pipelines.
- ubc-vision/solv3rA pretrained multi-view 3D reconstruction model that solves camera poses and point maps from image or video sequences.
- MUSE-AI-Lab/Concept-BrittlenessDiagnoses and repairs object-dependent concept failures in text-to-image diffusion models using per-step sparse autoencoders.