Running agents in production taught me that the dangerous failures are the quiet ones. A crashed agent gets noticed. An agent that confidently closes the wrong ticket, or summarises a document it only half read, keeps reporting success until a person happens to check. I have written about the supervision layers that catch this in production. This page is about catching it before release, and about the numbers that tell you an agent is ready to act on its own.
Model benchmarks do not answer that question. A model's score on a public leaderboard tells you little about whether your agent, with your prompts, your tools and your data, completes your tasks. You need your own evaluation, and agents need a different one from chatbots.
Why agent evaluation is different
- Many steps, one outcome. An agent plans, calls tools, reads results and adjusts. A wrong turn at step two can still end in a plausible final message, so grading the final message alone misses most failures.
- Non-determinism. The same task can succeed on one run and fail on the next. A single passing run is an anecdote.
- Side effects. Agents write to systems. Testing them needs an environment where those writes are real enough to check and safe enough to throw away.
- Untrusted input. Every document, email and page the agent reads can carry instructions. Safety testing is part of evaluation, not a separate exercise. See prompt injection.
The five layers to test
1. Components. Unit tests for each tool: schema, error handling, permissions, timeouts. Plus tests for any deterministic code around the model. These are cheap, fast and catch the boring failures that otherwise look like model problems.
2. Trajectory. Did the agent call the right tools, with the right arguments, in a sensible order, and did it stop when a tool failed? Compare the recorded trace with an expected path, allowing for valid alternatives. Trajectory checks find agents that reach the right answer by accident.
3. Outcome. After the run, inspect the system: is the order updated, the refund amount correct, the pull request passing its tests, the reply sent to the right person? Grade the state, not the agent's account of it. This is the single most important layer.
4. Policy and safety. Cases where the correct behaviour is to refuse, ask a clarifying question or escalate to a human, plus injected instructions hidden in the inputs the agent reads. Agents fail these more often than task cases, and they are the results your governance and legal teams will ask for.
5. Production. Sample live traces and grade them with the same rubric, track human override rates on approved actions, and alert on drift. Evaluation does not end at launch; it becomes monitoring.
The metrics that matter
| Metric | What it measures | Why it matters |
|---|---|---|
| Task success rate | Share of tasks where the end state matches the expected outcome | The headline number; graded on state, not on the agent's own report |
| pass^k | Share of tasks the agent gets right on all k repeated attempts | Reliability. A 90% single-run agent can fall well below that at pass^5 |
| Tool-call accuracy | Right tool, right arguments, right order | Finds failures that a lucky end state hides |
| Policy compliance | Correct refusals and escalations on cases designed to trigger them | The metric governance and legal will ask for |
| Steps and cost per successful task | Model calls, tool calls, tokens and dollars divided by successes | Catches loops and makes the business case honest |
| Human override rate | Share of proposed actions an approver changes or rejects | The production signal for widening or narrowing autonomy |
The one most teams skip is pass^k. Sierra's tau-bench research introduced it to measure reliability across repeated attempts, and it showed agents that looked strong on a single attempt losing much of that score when asked to succeed consistently. For an agent that will run the same task type thousands of times a month, consistency is the business requirement. Report pass^1 and pass^k side by side.
Cost per successful task closes the loop to the business case. An agent with a high success rate that takes forty tool calls per task can still lose to a simpler workflow. The numbers feed straight into agentic AI ROI.
Building the eval set
Start from real work. Pull 50 recent requests from the logs of the process the agent will take over, write down the correct outcome for each, and get the business owner to agree that those outcomes are right. That agreement matters more than the tooling: it is the definition of done the agent will be judged against.
- Cover the distribution. Mostly common cases, plus the edge cases people remember failing, plus deliberately ambiguous requests.
- Include the no cases. At least one in five tasks should have refusal, clarification or escalation as the correct answer.
- Add every production failure. When the agent gets something wrong live, the case goes into the suite. Grow toward 200 this way.
- Choose the cheapest reliable grader. Code assertions on system state first. A rubric-based LLM judge for open text, calibrated against a sample of human grades. Human review for the cases neither can settle.
Sandboxing agents
The sandbox is a copy of the agent's world where its actions are real enough to grade and cheap to discard. In practice that means staging instances or high-fidelity mocks of every system the agent touches, seeded with synthetic or masked data, with a reset between runs so every attempt starts from the same state.
- Isolate execution. Agents that run code or shell commands belong in containers or microVMs with no access to production credentials.
- Allowlist the network. Egress only to the hosts the task needs. This also turns many injection attacks into harmless failed requests.
- Use the production identity model. Give the agent the same scoped service account it will have in production, so permission failures show up in testing rather than live.
- Record everything. Full traces of model calls and tool calls, so a failed case can be replayed and diagnosed.
From sandbox to production
Passing the suite earns a staged rollout, not a launch.
- Shadow mode. The agent runs on live requests and proposes actions; people still act. Compare its proposals with what people did.
- Approval mode. The agent's actions execute only after a human approves them. The override rate is your best production accuracy signal.
- Canary. A small share of traffic runs with the agent acting autonomously within hard limits, against a control group handled the old way.
- A/B on the business metric. Compare resolution time, accuracy, customer satisfaction and cost per task with the baseline, with guard metrics that stop the test if errors rise.
Widen autonomy one task type at a time, and narrow it automatically when the override or error rate rises. The governance side of that decision, autonomy tiers and who signs off, is covered in the AI agent governance framework on AI Strategy Guide.
Every change is a release
Treat a model upgrade, a prompt edit, a new tool or a changed tool description as a release that re-runs the full suite. Model upgrades are the ones that catch teams out: a newer model can be better on average and worse on exactly the task your agent does. Pin model versions in production and promote a new one only after it beats the current one on your suite, not on a public benchmark.
Tooling
Most teams combine a tracing and evaluation platform with a sandbox runtime. For the platforms, see our LangSmith vs Helicone vs Phoenix vs Langfuse comparison and the LLM observability tools roundup. Buy the tooling after you have written the first 50 cases; the cases decide what you need the tool to do.
Checklist before an agent ships
- Eval set of at least 50 real tasks with agreed outcomes, including refusal and escalation cases.
- Outcome graded on system state; LLM judges calibrated against human grades.
- pass^k reported alongside single-run success, at a k that matches the agent's volume.
- Adversarial cases for injected instructions in every input type the agent reads.
- Sandbox with production identity, network allowlist and reset between runs.
- Shadow and approval stages completed, with override rate tracked.
- Suite wired to run on every model, prompt and tool change.
Back to the agentic AI hub, or see the agentic AI courses that teach this material, including MIT's module on governance, compliance and agent testing.
Frequently asked questions
How do you evaluate an AI agent?
Build a set of 50 to 200 realistic tasks with known correct outcomes, run the agent on each several times in a sandbox, and grade the resulting system state with code where possible and with a rubric-based LLM judge or human review where not. Track task success, pass^k reliability, tool-call accuracy, policy compliance and cost per successful task. Re-run the suite on every model, prompt or tool change, and keep evaluating in production by sampling live traces and tracking how often human approvers override the agent.
What is the difference between testing and evaluating an AI agent?
Testing checks fixed expectations: does this tool return the right schema, does the agent refuse this injected instruction, does the workflow finish. Evaluation measures quality across a distribution of tasks and produces rates: how often the agent succeeds, how reliably, and at what cost. You need both. Tests catch regressions in the plumbing; evaluations tell you whether the agent is good enough to ship.
What is pass^k?
pass^k is the probability that an agent completes a task correctly on every one of k independent attempts, popularised by Sierra's tau-bench. It measures reliability, which is what matters for an agent that will run the same kind of task thousands of times. It is different from pass@k, used for code generation, which counts a task as solved if any one of k attempts succeeds.
Can you use an LLM to grade an AI agent?
Yes, for outputs that code cannot check, such as the tone of a customer reply or whether a summary is faithful. Give the judge a specific rubric, ask for a verdict per criterion, and calibrate it against a sample of human grades before trusting it. Never let the judge grade from the agent's own description of what it did; give it the actual end state or output.
What is an AI agent sandbox?
A sandbox is an isolated environment where an agent can take real actions without real consequences: a copy of the systems it will use, seeded with synthetic or masked data, with network access limited to an allowlist and a reset between runs. Agents that execute code need stronger isolation, such as containers or microVMs. The sandbox is where pre-release evaluation and adversarial testing happen.
How many test cases does an agent eval need?
Start with 50 well-chosen tasks drawn from real requests, with correct outcomes written down, and grow toward 200 as you find failures. Coverage matters more than count: include the common cases, the edge cases you have seen fail, and cases where the correct behaviour is to refuse, ask a question or hand over to a human.