NVIDIA/SkillEvaluator

What it does

SkillEvaluator is part of NVIDIA's Verified Skills pipeline and works on skills defined by the Agent Skills specification — folders of instructions and supporting files that get loaded into agents. It answers three separate questions through independent tiers:

• Tier 1 — Validation: is the skill well-formed and safe? Static checks plus pinned SkillSpector and Gitleaks releases; Semgrep is optional.
• Tier 2 — Deduplication: does the guidance repeat or overlap with itself or other skills, using embeddings and a catalog for inter-skill comparison?
• Tier 3 — Live evaluation: does the skill actually change agent behavior? It runs sandboxed agent tasks through Harbor and compares results with and without the skill.

Reports can go to a terminal, JSON files, or CI. The GitHub/GitLab bundle runs quality and security checks on changed skills and can post reports back on pull requests. The intended audience is skill authors and platform teams who distribute skills to agents such as Claude Code or Codex.

Quickstart

Prepare Git and uv; uv will provision Python 3.13. Then clone the repository and run the keyless quality check on the bundled text-analyzer skill:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

No API key, scanner, GPU, or Docker is needed. According to the keyless walkthrough, the example scores 84.5/100 (B), exits 0, and writes reports/first-quality/skillevaluator-quality.json; the default pass threshold is 70. This is static quality only — publication evidence stays INCOMPLETE, and security or agent-impact conclusions need the other tiers.

For model-backed rubric feedback, enroll in the NVIDIA Developer Program, generate a key at build.nvidia.com, then:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

To run everything, uv run --extra all skillevaluator doctor --env-mode docker checks readiness, then validate ./my-skill runs all tiers with autopilot. Replace the example path with your own skill folder.

Constraints and setup

• Tier extras are opt-in: --extra security, --extra tier2, --extra tier3, or --extra all. Live evaluation needs provider access, a supported agent, and a running sandbox (Docker by default).
• A scanner that cannot run marks the result INCOMPLETE and exits nonzero; --no-llm does not remove scanner requirements. Tier 2 reports when inter-skill comparison did not run.
• Only Tier 1 always gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval.
• Model endpoints and managed sandboxes may incur charges; see the cost notes. Hosted NVIDIA Build prototyping is subject to access terms and rate limits.
• Evaluation data defaults are thin: templates include a starter case, and negatives are omitted unless authored in EVAL.md. Use tier3 create-eval-dataset ./my-skill --full for broader coverage, then review the cases.
• The project downloads and installs third-party open-source scanners; review their licenses before use. Licensed Apache-2.0. Support is labeled Experimental and community best-effort, with no SLA.

HotRepo's take

The project appeared once in the collected record: a Python daily list snapshot on 2026-11-12 with 83 stars gained and a best position of 16, against a total of 756 stars. That single snapshot supports only the fact of that day's showing; there is no multi-day series here to describe a trend. The design choice that matters for adoption is the separation of concerns: cheap deterministic quality checks can run on every commit without credentials, while costly model and sandbox evaluation is gated behind extras and explicit flags. Teams integrating this into CI should budget for the non-LLM tier first and treat live agent impact as a separate, cost-bearing step.