The Six Layers
Golden Sets
Curated input→expected pairs that define "known good" and anchor every future comparison.
Behavioral Coverage
Make sure your tests span every intent, capability, and edge case — not just the happy path.
Error Analysis
Cluster and categorize failures to find root causes instead of chasing individual bugs.
Replay Harnesses
Record real sessions into fixtures and replay them deterministically, cheaply, and offline.
Rubrics
Score outputs across explicit dimensions — accuracy, tone, safety, completeness — with LLM judges.
Experiments
Run A/B comparisons with statistical rigor so prompt and model changes are decisions, not guesses.
How the Layers Build on Each Other
baseline"] --> L2["🧭 2 · Behavioral
Coverage"] L2 --> L3["🔎 3 · Error
Analysis"] L3 --> L4["🎬 4 · Replay
Harnesses"] L4 --> L5["📋 5 · Rubrics"] L5 --> L6["🧪 6 · Experiments"] L6 -.improve.-> L1
Each layer assumes the ones before it. You cannot measure coverage without a dataset, cannot analyze errors without measurable outputs, and cannot run trustworthy experiments without reproducible, multi-dimensional scoring. The loop closes when experiment insights feed back into your golden sets.
A Maturity Path, Not a Checklist
Layers 1–2
Define correctness with golden sets and make sure they cover the behaviors that matter. This alone catches most regressions.
Layers 3–4
Systematically analyze failures and freeze sessions into replayable fixtures so evaluation is cheap, fast, and reproducible.
Layers 5–6
Grade nuanced quality with rubrics and make every change a measured experiment with statistical confidence.