lijiaqi-AI-Product-Manager/dsh-agent-compare

What It Does

AgentCompare is a plugin for DeepSeek Harness that addresses a common problem in agent-assisted frontend work: tests can tell whether a feature exists, but only a person can judge whether an interface is good. The plugin takes one frontend requirement, generates two independent implementations, runs both as live apps, and lets the user inspect and choose before the underlying model or strategy is revealed.

It is aimed at UI and interaction tasks such as search and filtering, form flows, landing pages, dashboard layouts, or component redesigns. For work with a single correct answer, such as bug fixes or refactors, the README suggests a single agent with tests instead.

Two comparison modes are supported:

Mode	Shared	Different
Same model, two strategies (default)	Model, requirement, acceptance	Conservative existing style vs. bolder interaction rewrite
Two models, same requirement	Requirement, acceptance, agent instructions	Model, chosen from models already configured in DSH

The plugin handles the tedious setup: both candidates start from the same baseline (including uncommitted changes), receive isolated git worktree copies, install and build independently, run on separate ports, and are checked against the same baseline-bound acceptance configuration. Adoption only becomes available after verification, and the target repository is re-checked before and during writing.

Quick Start

The following applies to self-run DSH Web profiles. For the desktop desktop profile, use the app's plugin page instead.

1. Install the npm package into the default Web profile:

   dsh plugin --profile web add dsh-agent-compare@1.1.0
   

   For another self-run profile, replace web with its name. Alternatively, build from source and install the tarball:

   git clone https://github.com/lijiaqi-AI-Product-Manager/dsh-agent-compare.git
   cd dsh-agent-compare
   npm ci
   npm pack
   dsh plugin --profile <your-profile> add ./dsh-agent-compare-1.1.0.tgz
   

2. Restart DSH, then click the A|B button at the lower-left of the main conversation input.

3. Choose “same model, two strategies” or “two models, same requirement”, and set a per-candidate generation limit (5–120 minutes).

4. Write the requirement in the original input box and send it.

5. An A/B card appears with two live previews. Interact with both, switch viewport or focus mode as needed, then select one. The strategy, actual model, and token usage are revealed.

6. Expand the code changes and click “Review and adopt”. Confirm the target repository and verification result before writing back. Use “Continue main conversation” to exit comparison mode.

A built-in example (basic search and clear, or full combined filtering and mobile adaptation) can be launched from the settings bar to try the flow in a separate example repository.

Configuration

Optional settings can be appended to the profile's cordis.patch.yml under the plugin id:

- id: agent-compare
  name: dsh-agent-compare
  config:
    generationMinutes: 20

Key	Default	Meaning
generationMinutes	30	Time budget per candidate
maxActiveRuns	1	Concurrent comparison runs
isolation	own-root	shared-root is the experimental legacy mode
home	~/.dsh/agent-compare	Records, patches, screenshots, and copies

Custom Acceptance

A repository-root .agent-compare.json can add scripted browser checks and additional protected files. The configuration is validated before generation; unknown fields, missing scripts, invalid steps, or cross-origin paths are rejected. Acceptance reads the baseline version, so candidates changing it cannot pass. A full example is available in examples/vite-react-demo/.agent-compare.json.

Constraints

• Currently limited to root-level Vite projects in Git repositories with at least one commit.
• Tested primarily on Node 26.13.0, with the latest patches of Node 22 and 24 in CI compatibility checks.
• DSH 0.2.1-rc.3 desktop on Windows 11 and 0.2.1-rc.5 Web UI on Windows 11 are documented as passing cases; Linux core regressions pass in CI, while macOS has not been tested.
• Windows sandbox constraints are reported by DSH as partial; the plugin requires host-provided workspace-write constraints and does not fall back to unconstrained execution.
• Dependencies are installed in frozen mode when a lockfile exists; projects without one are explicitly flagged.
• Browser state is not fully isolated: embedded cookies, external backends, and shared databases are not supported, and network/process visibility is not isolated.
• A single comparison does not prove a model or strategy is generally better.
• MIT licensed.

HotRepo's Take

The project appeared on HotRepo's TypeScript new-project list on 2026-11-05 with 104 stars and 4 net new stars in a roughly one-hour observation window; it has not been recorded on the all-language daily ranking. The README is unusually explicit about what has been verified and what has not, distinguishing real-model runs, scripted-adapter runs, and CI regressions. That makes the repository useful as a concrete reference for how a two-candidate agent workflow can be isolated, bound to a baseline, accepted, and rolled back, even for readers who do not install the plugin itself.