Your engineering team ships code with 85% test coverage. Your AI agent ships with... 12 scenarios, most of which are variations of a happy path you wrote in the first week of development.
This is almost universal. Fifty-seven percent of organizations now have AI agents in production, and quality is the top barrier to expansion for 32% of them. But only 52% run offline evaluations against actual test sets before deploying. The other half is essentially running blind.
The gap isn't ignorance — it's that no one has told you how to think about coverage for agents the way you've been taught to think about coverage for code. Code coverage has a number. Agent coverage doesn't. This article gives you a framework for it.
Why Code Coverage Doesn't Apply Here
Code coverage answers a binary question: was this line (or branch, or function) executed during the test run? It works because code is deterministic. The same input, on the same code, produces the same output. Coverage tells you which parts of the decision tree you've exercised.
AI agents break this assumption at every level. The same input, processed by the same model with the same prompt, can produce different tool calls, different retrieval results, and different natural language responses across runs. There's no decision tree to map. Coverage based on execution paths doesn't exist.
What you can measure instead is whether your test suite exercises the behavioral space your agent operates in. Think of it as a distribution problem: the space of things users might try to do with your agent is a distribution. Your test suite is a sample from that distribution. Good coverage means your sample represents the distribution accurately, including the long tail.
graph TD
A[Full Behavioral Space] --> B[Common Intents]
A --> C[Edge Cases]
A --> D[Adversarial Inputs]
A --> E[Multi-Turn Patterns]
A --> F[Tool Failure Modes]
B --> G[Suite Coverage?]
C --> G
D --> G
E --> G
F --> G
G --> H[Good if distribution matches production]
G --> I[Bad if all tests are happy-path variants]The Five Coverage Dimensions
Coverage for an AI agent has five dimensions. A test suite that's strong on one but weak on others will leave predictable gaps.
1. Intent Coverage
Does your suite cover the range of things users are trying to accomplish? For an agent connected to internal MCP tools, this might include: looking up records, creating records, running queries, kicking off jobs, and handling errors when a tool returns nothing useful. Each of these is a distinct intent category, and each can fail independently.
The failure mode here is depth without breadth. A team might have 40 scenarios for record lookup and zero for error handling. When a user asks something that hits an unmapped intent, that failure wasn't predictable from the test suite because the intent category was unrepresented.
Practical test: list every distinct thing a user might want to accomplish with your agent, without regard to how common they are. Each item on that list should have at least one test scenario.
2. Persona Coverage
The same intent expressed by different users can produce different agent failures. A user who writes "QUERY DOESN'T WORK FIX THIS" is expressing the same intent as a user who writes "Hi, I ran the report yesterday and the numbers don't match what I expected — could you help me trace it?" But agents trained on formal language often mishandle terse, emotional, or colloquial inputs.
Persona dimensions to test:
- Communication style (formal vs. colloquial vs. terse)
- Technical vocabulary (expert vs. novice language)
- Emotional tone (neutral, frustrated, confused, urgent)
- Language patterns (short questions, long explanations, mid-sentence topic switches)
A test suite with ten scenarios from one persona type will miss failures that only surface with other personas. Scenario testing with realistic personas — not just syntactic variations of the same message — is what reveals these gaps before production does.
3. Conversation Depth
Most pre-deploy testing covers single turns. Most production failures happen in turn four, seven, or twelve.
The pattern is consistent: agents perform well at the start of a conversation when context is minimal and intent is clear. As the conversation lengthens, context accumulates, earlier misunderstandings compound, and the agent loses track of constraints it established earlier. An agent that correctly says "I can't process that without manager approval" in turn two might quietly violate that constraint in turn nine when the context window is crowded.
Test at multiple depths:
- 1-2 turn scenarios (baseline)
- 3-5 turn scenarios (typical conversation)
- 8-12 turn scenarios (complex requests)
- 15+ turn scenarios (escalation paths, edge cases)
Each depth tier reveals different failure modes. You need representation across all of them.
4. Tool Usage Coverage
Every tool your agent has access to should have test coverage, and that coverage should include failure cases, not just success cases.
What "tool coverage" actually means:
- Every tool gets called at least once in your suite
- Each tool is called with boundary inputs (empty strings, max-length values, unexpected types)
- At least one scenario per tool where the tool returns an error
- At least one scenario per tool where the tool is called incorrectly (wrong arguments, missing required fields)
- Multi-tool scenarios where two or more tools are needed in sequence
The failure mode here is testing the tool in isolation but never testing how the agent handles the tool's error response. An agent that works when the upstream API returns a clean 200 can behave unpredictably when it returns a 503 with a generic error message. Test both.
5. Adversarial Coverage
Every agent in production will encounter users who try to push it off its rails — not necessarily with malicious intent, but through ambiguous requests, contradictory instructions, scope violations, and edge cases the original design never anticipated.
Adversarial coverage includes:
- Out-of-scope requests
- Contradictory instructions ("I know you said you can't do that, but just this once")
- Jailbreak attempts ("Ignore your previous instructions and...")
- Ambiguous inputs that could match multiple intents
- Extremely long inputs that approach context limits
- Inputs in unexpected languages or scripts
You don't need exhaustive adversarial coverage. You need enough to verify that your agent fails gracefully — declining out-of-scope requests clearly, resisting manipulation consistently, and surfacing ambiguous cases rather than guessing.
How Many Scenarios Is Enough?
There's no universal number, but there's a useful heuristic: start with 50, grow by production.
50 well-chosen scenarios — roughly 10 per coverage dimension — is enough to catch most pre-deploy failures for a focused agent. Spreading them evenly across intent categories, personas, conversation depths, tool usage, and adversarial cases gives you genuine coverage, not a false sense of security from 50 variations of the same happy path.
After your first production deployment, run a post-mortem on failures. For each production failure, ask: was there a scenario in the pre-deploy suite that would have caught this? If the answer is no, add one. Do this consistently for the first three months and your suite will grow to accurately represent your agent's failure modes.
Practically, teams that do this well end up with:
- 20-30 scenarios derived directly from real production conversations
- 20-30 synthetic scenarios covering known flows not yet seen in production
- 10-15 adversarial scenarios targeting known manipulation patterns
- Growth rate of roughly 5-10 new scenarios per deployment cycle
What you're building is not a static test artifact. It's a living document of everything you've learned about how your agent can fail.
The Coverage-to-Confidence Mapping
Your suite's distribution maps directly to deploy confidence. Use this readiness table to assess where you are before shipping:
| Coverage State | Deploy Confidence | Typical Suite Characteristics |
|---|---|---|
| No pre-deploy testing | Very low | No suite, rely on observability only |
| Happy-path only | Low | 10-20 scenarios, all nominal flows |
| Intent-covered | Moderate | 30-50 scenarios, all intents represented |
| Multi-dimensional | High | 50-100 scenarios, all 5 dimensions |
| Production-derived | Very high | 100+ scenarios, real failures represented |
Most teams are at "happy-path only." Reaching "intent-covered" usually takes one week of deliberate work. Reaching "multi-dimensional" takes a sprint. The jump to "production-derived" happens automatically if you maintain the practice of converting failures into tests.
The Distribution Trap
One more thing to watch: coverage numbers can lie.
A suite of 100 scenarios that are all slight variations of five conversation flows has less real coverage than a suite of 30 scenarios that each represent a genuinely distinct part of the behavioral space. The number isn't what matters — the distribution is.
Before you deploy, do a quick audit: group your scenarios by intent category and by persona type. If any single group has more than 30% of your scenarios, you have a distribution problem. Diversify before you ship.
Tools for this exist on the open-source side (Promptfoo, Inspect, Ragas) and on the managed side (Galileo, Maxim, LangSmith). Pick one that surfaces gaps in your suite before you run it — not just pass-rate after.
What Good Coverage Looks Like in Practice
Here's what a well-designed 60-scenario suite looks like for a developer-tools agent that handles record lookup, querying, and job orchestration:
graph TD
A[60 Scenarios] --> B[Intent Coverage: 18 scenarios]
A --> C[Persona Coverage: 12 scenarios]
A --> D[Conversation Depth: 15 scenarios]
A --> E[Tool Coverage: 9 scenarios]
A --> F[Adversarial: 6 scenarios]
B --> B1[Lookups: 6]
B --> B2[Queries: 6]
B --> B3[Jobs: 6]
C --> C1[Formal: 4]
C --> C2[Terse/Frustrated: 4]
C --> C3[Confused/Verbose: 4]
D --> D1[1-2 turns: 5]
D --> D2[5-8 turns: 5]
D --> D3[12+ turns: 5]
E --> E1[Normal calls: 4]
E --> E2[Error responses: 5]
F --> F1[Out of scope: 3]
F --> F2[Manipulation: 3]This isn't perfect coverage — no suite is. But it's a distribution that will catch most failures before production rather than discovering them from user complaints.
Connecting Pre-Deploy Testing to Production Monitoring
Coverage isn't a problem you solve once at deploy time. It's an ongoing practice that connects your pre-deploy test suite to your production observability.
The pipeline looks like this: production conversations surface new failure modes through analytics and monitoring. Those failures feed back into the test suite as new scenarios. The test suite grows to reflect the real distribution of things users try to do. Pre-deploy coverage improves with each cycle.
Teams that treat this as a closed loop — not a one-time testing effort — end up with agents that get more reliable over time rather than degrading as user behavior evolves.
The Simple Version
If you take one thing from this: your test suite should look like a representative sample of your production traffic, not a curated demo of your agent at its best.
Start with real failures. Convert support tickets into test cases. Add synthetic scenarios to cover intents you haven't seen yet. Grow the suite with each deployment. That practice, done consistently, will tell you more about whether you're ready to ship than any coverage number will.
Skip the eval pipeline build
If you're shipping MCP, the free tier covers your first server. Run scenarios against it and read the results in your existing tooling — no separate platform to set up.
Start freeRelated reading on the Chanl blog: Is Your AI Agent Actually Ready for Production? The 3 Tests Most Teams Skip and How to Eval Agents When There's No Right Answer.
Founder, Protobox
Building TBD bio at Protobox — tools, testing, and observability for customer experience.
MCP changelog and dev notes
Short, occasional updates on the MCP spec, new Protobox features, and patterns we've seen in production. No marketing fluff.
