Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

agent-platform-eval-flywheel

Builds an evaluation dataset, runs agent inference, and scores results with Google Cloud's GenAI Evaluation SDK.

  • Part of google/skills
  • 9.8k installs
  • AI Engineering · Evals and monitoring

When to use it

You have an agent or model but no way to tell whether its answers are good, and need a scored dataset before changing prompts or code.

For example

You say

Use agent-platform-eval-flywheel to evaluate my customer-support ADK agent in project my-proj, us-central1: build a small eval dataset from these sample questions and score it for general quality and hallucination.

You get
  • EvaluationDataset script
  • Scored eval report
  • Failure analysis

How it works

  • Runs a five-stage loop: prepare data, run inference, grade, analyze failures, then fix and re-check.
  • Every evaluation run needs a dry-run preview and your explicit approval before it executes.
  • Can synthesize multi-turn user scenarios when you have no data yet.

Before installing

Needs a Google Cloud project and location plus the vertexai, google-genai, pandas, and requests packages installed.

This skill includes 7 scripts your assistant may run on your computer.