All skills
agent-platform-eval-flywheel
Builds an evaluation dataset, runs agent inference, and scores results with Google Cloud's GenAI Evaluation SDK.
- Part of google/skills
- 9.8k installs
- AI Engineering · Evals and monitoring
When to use it
You have an agent or model but no way to tell whether its answers are good, and need a scored dataset before changing prompts or code.
For example
You say
Use agent-platform-eval-flywheel to evaluate my customer-support ADK agent in project my-proj, us-central1: build a small eval dataset from these sample questions and score it for general quality and hallucination.
You get
- EvaluationDataset script
- Scored eval report
- Failure analysis
How it works
- Runs a five-stage loop: prepare data, run inference, grade, analyze failures, then fix and re-check.
- Every evaluation run needs a dry-run preview and your explicit approval before it executes.
- Can synthesize multi-turn user scenarios when you have no data yet.
Before installing
Needs a Google Cloud project and location plus the vertexai, google-genai, pandas, and requests packages installed.
This skill includes 7 scripts your assistant may run on your computer.
More Evals and monitoring skills
See all- ai-prompt-engineering-safety-review10.5kReviews prompts for safety, bias, security and effectiveness, then suggests concrete improvements.
- agentic-eval10.3kSets up evaluation and self-critique loops to measure and improve AI agent output quality.
- prompt-engineer1.5kDesigns effective prompts for LLM applications, covering structure, context, output format and evaluation.
- prompt-engineering1.4kGuides you through prompt patterns and techniques to improve prompts and debug agent behavior.
Also in this pack
- cloud-run-basics19.4kDeploys and manages Cloud Run services, jobs and worker pools.
- gemini-api18.8kWrites code that calls Gemini models for text, images, video, audio, embeddings and more across languages.
- google-cloud-recipe-auth17.1kExplains how to authenticate and authorize users and services to Google Cloud APIs, including IAM and ADC.
- google-cloud-recipe-onboarding16.9kWalks a new developer through creating a Google Cloud account, setting up billing and deploying a first resource.