How do you currently test your AI agents?
If you're like most developers building agentic workflows, you probably rely on the "vibe check" : tweak a prompt, run a couple of manual queries, check if the output looks reasonable, and assume it's good to go.
Here's why that breaks down fast:
Silent Regressions : A teammate submits a PR or tweaks a tool definition, and you have no way to know if performance improved or degraded.
Cost & Latency Spikes : An agent gets trapped in a multi-turn retry loop, quietly burning thousands of tokens.
The "Looks Right" Trap : Output can look perfectly well-written while omitting critical parameters or hallucinating key instructions.
In this episode of the AI Agent Clinic , Google Cloud engineer Dani Zamora and Matthew Feroz (Merge) tackle this challenge live: building an end-to-end, framework-agnostic evaluation pipeline in 60 minutes.
Check out the full walkthrough below:
What We Cover in the Video
The Test Subject (DocsHound) : Matt brings on DocsHound, an open-source LangGraph agent that scans GitHub issues and PRs to find documentation gaps and open automated PRs.
Framework-Agnostic Telemetry (OpenTelemetry + OpenInference) : How to standardize multi-turn traces across any agent framework (LangGraph, CrewAI, AutoGen, ADK) with zero vendor lock-in.
The 3-Tier Metric Strategy : Why you shouldn't start with 50 metrics, and how to combine:
Managed core metrics : Trajectory quality, tool-calling correctness, and groundedness.
Custom LLM judges : Rubrics tailored to formatting, code block enforcement, and actionability.
Deterministic assertions & operational tracking : Syntax checks, latency, token usage, and API cost.
The 0.33 Quality Blind Spot : How the automated scorecard caught a 33% documentation quality failure that manual spot-checks completely missed.
Jump Directly to Key Sections
If you want to skip straight to a specific part of the walkthrough:
00:00 — Why "vibe checking" fails in production
02:11 — Introducing DocsHound (LangGraph agent demo)
03:40 — Standardizing traces with OpenTelemetry & OpenInference
05:51 — The 60-minute eval challenge begins
06:47 — Step 1: Mapping agent execution flow with Antigravity
11:18 — Step 2: Scaffolding the open-source Agent-Eval toolkit
17:14 — Balancing quality vs. latency vs. token cost
18:34 — Step 3: Translating quality definitions into custom metrics
21:40 — Step 4: Inspecting the scorecard & spotting the 0.33 failure
24:08 — Why automated evals change how you build agents
Open-Source Repositories & Resources
Follow along with the exact tools used in the video:
📺 Full Video : Watch on YouTube (25 mins)
🛠️ Agent-Eval Toolkit : github.com/google-cloud-tech/agent-eval
🐶 DocsHound Agent Code : github.com/google-cloud-tech/docshound
📖 Enterprise Agent Platform Docs : g.dev/cloud/agent-evaluation
👉 Watch the full video above to see how to set up the eval pipeline step-by-step. If you have questions about instrumenting your own agent framework or setting up custom metrics, drop a comment below!
