What Is Behavioral Coverage?
Behavioral coverage answers a question your pass rate cannot: "What haven't we tested yet?" A perfect 100% score on a set that only tests refunds tells you nothing about how the agent handles exchanges, complaints, or abuse.
You define a taxonomy of behaviors — intents, capabilities, input types, difficulty levels, languages — then measure how many of those cells your golden set actually touches. Empty cells are blind spots.
It is the natural partner to Layer 1: golden sets give you trusted examples; coverage tells you whether those examples represent the real world.
"Code coverage, but for behaviors — measuring the share of the behavior space your tests reach."
- A map with unexplored regions
- A test matrix / grid
- Branch coverage in unit tests
Why Coverage Matters
Exposes Blind Spots
A high score on a narrow set hides risk. Coverage reveals the behaviors you never tested at all.
Prevents Skew
Stops your metrics from being dominated by whichever category happens to have the most examples.
Directs Effort
Tells you exactly which examples to write next instead of adding more of what you already have.
Real-World Fit
Aligning coverage with production traffic distribution keeps scores honest and representative.
Edge-Case Safety
Forces you to include adversarial, empty, and malformed inputs — where agents most often fail.
Trackable Progress
Coverage percentage is a metric you can grow deliberately, sprint over sprint.
The Coverage Matrix
A behavior taxonomy is a set of dimensions. Their product forms a matrix of cells; each cell should contain at least one golden example.
dimensions:
intent:
- refund_request
- exchange_request
- order_status
- complaint
- abuse # adversarial
difficulty:
- easy
- hard
input_type:
- well_formed
- ambiguous
- malformed
targets:
min_per_cell: 1
priority_intents: [refund_request, abuse]Dimensions
Independent axes of behavior — intent, difficulty, input shape, language, channel.
Cells
Each combination (e.g. refund × hard × malformed) is a distinct behavior to test.
Targets
Minimum examples per cell, with higher targets for high-risk or high-traffic behaviors.
Adversarial Axis
Always include intents and input types designed to break the agent.
How Coverage Analysis Works
for empty cells"] W --> M G -- "No" --> D["✅ Coverage complete"]
| Intent \ Difficulty | Easy | Hard |
|---|---|---|
| refund_request | ●●●● (4) | ●● (2) |
| exchange_request | ● (1) | ○ (0) gap |
| complaint | ●● (2) | ○ (0) gap |
| abuse | ○ (0) gap | ○ (0) gap |
Coverage: 4 / 8 cells populated = 50%. Priority gap: abuse is completely untested.
Step-by-Step Implementation
Declare the taxonomy
Model dimensions and targets as data so coverage is computed, not eyeballed.
from __future__ import annotations
from dataclasses import dataclass, field
from itertools import product
@dataclass(frozen=True)
class Taxonomy:
"""A set of behavioral dimensions and per-cell coverage targets.
Args:
dimensions: Mapping of dimension name to its allowed values.
min_per_cell: Minimum examples required in every cell.
"""
dimensions: dict[str, list[str]]
min_per_cell: int = 1
def cells(self) -> list[tuple]:
"""Return the Cartesian product of all dimension values."""
return list(product(*self.dimensions.values()))
def key_of(self, example) -> tuple:
"""Project an example onto its coverage cell."""
return tuple(getattr(example, dim) for dim in self.dimensions)Compute the coverage report
Bucket examples into cells, then report which cells are under-target.
from collections import Counter
from .taxonomy import Taxonomy
def coverage_report(examples: list, taxonomy: Taxonomy) -> dict:
"""Measure how thoroughly ``examples`` cover ``taxonomy``.
Args:
examples: Golden examples exposing the taxonomy's dimensions.
taxonomy: The behavior space to measure against.
Returns:
A dict with the covered fraction and the list of gap cells.
"""
counts = Counter(taxonomy.key_of(ex) for ex in examples)
all_cells = taxonomy.cells()
gaps = [cell for cell in all_cells if counts[cell] < taxonomy.min_per_cell]
covered = len(all_cells) - len(gaps)
return {
"coverage": covered / max(len(all_cells), 1),
"covered_cells": covered,
"total_cells": len(all_cells),
"gaps": gaps,
}Gate CI on a coverage floor
Fail the build when coverage regresses or a priority cell is empty.
from coverage.taxonomy import Taxonomy
from coverage.report import coverage_report
def test_behavioral_coverage_floor():
"""At least 80% of behavior cells must contain a golden example."""
taxonomy = Taxonomy(
dimensions={
"category": ["refund_request", "exchange_request", "complaint", "abuse"],
"difficulty": ["easy", "hard"],
},
min_per_cell=1,
)
report = coverage_report(load_golden_set("golden/support.jsonl"), taxonomy)
assert report["coverage"] >= 0.80, f"Uncovered cells: {report['gaps']}"Use Cases
Intent Coverage
Guarantee every ticket intent — including rare ones like fraud — has at least one test.
Language × Intent
Add language as a dimension to confirm each intent works across every supported locale.
Capability Matrix
Ensure every tool the agent can call is exercised by at least one scenario.
Malformed Inputs
Track coverage of empty, huge, and adversarial inputs across every intent.
Best Practices & Pitfalls
Do
- Derive from production. Base your taxonomy on real traffic distribution.
- Weight by risk. Require more examples for high-impact or high-frequency cells.
- Always include adversarial. Abuse and malformed input belong in the matrix.
- Report gaps, not just a number. The list of empty cells is the actionable output.
Avoid
- Combinatorial explosion. Too many dimensions creates thousands of impossible-to-fill cells.
- Coverage theater. One token example per cell isn't real coverage of hard cases.
- Static taxonomy. New product features must add new dimensions and cells.
- Confusing coverage with quality. Full coverage says you tested it, not that it passed.
FAQ
How is this different from a pass rate?
Pass rate tells you how well you did on what you tested. Coverage tells you how much of the real behavior space you tested at all. You need both.
How do I discover categories?
Cluster real production inputs (e.g. by embedding + k-means or an LLM classifier), then have a human name and refine the resulting groups into a taxonomy.
Should every cell have equal weight?
No. Weight by production frequency and business risk so a rare-but-catastrophic cell isn't treated like a trivial one.