Open-RUA/uni-robot-agent

What it is

uni-robot-agent is a Python harness for robot-use agents: a language/vision model issues tool calls that act on a robot environment, and the harness schedules, executes, and records those calls. It ships a CLI, a Python SDK, a local web dashboard, and simulator integrations including RoboDojo, RoboUse, and RoboLab. Runtimes for Codex and Claude Code are supported, and the interface lets you swap agent runtime, model provider, tools, environment, or model services.

The project accompanies the paper "Democratizing Robot-Use Agents: A Unified Harness with Open Models" and implements a realistic RGB-only observation setting (with option to drop depth) and a hybrid action mode that can call an action model such as Pi0.5. It is aimed at researchers and engineers working on robot manipulation agents in simulation.

Quick start

Prepare a Python 3.11 virtual environment on a Linux GPU host, then install from PyPI and set up the RoboDojo integration:

pip install uni-robot-agent
uni-robot-agent setup --integration robodojo

Provide a Responses-compatible model endpoint and key, then run a task:

export UNI_ROBOT_AGENT_API_KEY='YOUR_API_KEY'
export UNI_ROBOT_AGENT_PROVIDER_URL='https://provider.example/v1'

uni-robot-agent run --integration robodojo --deploy local \
  --task stack_bowls --model YOUR_MODEL_ID \
  --action-profile direct --no-sam3 --output-dir ./runs --monitor

The command prints a dashboard URL; artifacts go to ./runs. To replay saved runs later, use uni-robot-agent monitor --run-dir ./runs.

For programmatic control, the SDK runs against an already-running environment (for example http://127.0.0.1:18760) and returns a result with agent_finished and task_success:

from uni_robot_agent import AgentConfig, SessionConfig, run

result = run(SessionConfig(
    integration="robodojo",
    env_url="http://127.0.0.1:18760",
    no_sam3=True,
    agent=AgentConfig(model="YOUR_MODEL_ID", provider_url="https://provider.example/v1"),
    output_dir="runs",
))
print(result.agent_finished, result.task_success, result.session_dir)

Environments and hardware

Simulation integrations cover RoboDojo dual-arm manipulation (Direct or Hybrid Pi0.5 control), RoboUse tasks drawn from LIBERO, RoboCasa, MetaWorld, RoboSuite, ManiSkill, Tabletop, Fetch and Kitchen, and RoboLab single-DROID tasks that require a separately prepared Isaac Sim / Isaac Lab runtime. Real-robot support for ARX One and Franka is listed as coming soon; the public package does not yet include a supported physical-robot workflow.

Constraints

• Installation and execution target a Linux GPU host with Python 3.11; the README does not document other platforms.
• An API key and a compatible model endpoint are required for the agent's vision model; placeholder values must be replaced.
• RoboLab depends on a separate Isaac Sim / Isaac Lab setup; check the integration guide before expecting it to run out of the box.
• The harness includes a Depth-enabled default for RoboDojo; use --no-depth for the paper's RGB-only setting while retaining camera calibration.
• Licensed MIT, with upstream code, fonts, and external assets/models keeping their own licenses.

HotRepo's take

The repository entered HotRepo's new-project list on 2026-11-03 with a first recorded snapshot of 100 stars and 6 forks, all gained inside the growth window beginning 2026-11-02. That is a single early snapshot: it establishes initial attention but not a trend. What distinguishes this project from generic agent demos is the combination of a packaged CLI/SDK/dashboard and named simulator integrations, plus a published paper. Its practical scope, however, remains simulation-first; claims about physical robots should be judged by the coming-soon status above.

Project source and documentation