AI Agent Evaluation: Reliability, Tool Calls & Regression Testing
An AI agent can produce the correct final answer while taking an unsafe or inefficient path. Agent evaluation must therefore assess the entire execution path, not only the final text.
The core question is:
Can the agent complete the task reliably, using the right tools, within permitted boundaries and with enough repeatability for production?
For the upstream architecture choice, see AI Agent vs Workflow vs Automation. If an agent PoC already exists, AI Production Readiness is the direct next step.
The short version
| Layer | What is evaluated? | | --- | --- | | Outcome | Was the real task completed correctly? | | Planning | Was the chosen approach sensible and efficient? | | Tool use | Were the right tools called with correct arguments? | | Permissions | Did the agent stay within authorised boundaries? | | Guardrails | Were prohibited or risky actions blocked? | | Human gates | Were required approvals requested? | | Recovery | Can the agent handle tool failures and missing data? | | Regression | Does behaviour remain stable after model, prompt or tool changes? |
Anthropic notes that agent evaluation is harder because agents operate across multiple turns, call tools, modify state and adapt based on intermediate results. Source: Anthropic – Demystifying evals for AI agents.
Why final-answer quality is not enough
For a conventional chat use case, final-answer quality may represent a large share of the objective.
An agent can additionally:
- read data,
- call APIs,
- modify files,
- send messages,
- prepare transactions,
- escalate decisions,
- coordinate multiple systems.
Two runs can therefore produce the same final text while having very different risk profiles.
Example:
Run A
- Agent reads authorised CRM data.
- Uses approved pricing tool.
- Drafts offer.
- Waits for approval.
Run B
- Agent also reads sensitive notes.
- Uses the wrong tool.
- Overwrites a record.
- Produces the same offer.
Evaluating only the final answer misses the difference.
The agent-evaluation framework
1. Task success
Did the agent achieve the intended business outcome?
Examples:
- ticket resolved correctly,
- record prepared completely,
- research completed with valid sources,
- request escalated correctly,
- no answer produced when evidence was insufficient.
Task success should be closer to the real workflow than to stylistic quality.
2. Tool selection
Did the agent choose the right tool?
Failure classes include:
- unnecessary tool,
- wrong tool,
- unauthorised tool,
- missing tool call,
- tool called too early or too late.
This matters especially when several tools have similar descriptions.
3. Tool arguments
The right tool with the wrong arguments can still be dangerous.
Check:
- IDs,
- amounts,
- recipients,
- filters,
- time ranges,
- read/write mode,
- scope,
- tenant,
- destination system.
For critical parameters, deterministic assertions are often better than purely semantic graders.
4. Execution trace
An agent works through a sequence of observations and actions.
The trace should capture at least:
- user input,
- relevant system instructions,
- selected tools,
- tool arguments,
- tool results,
- decisions,
- approvals,
- errors,
- final outcome.
You do not need to persist every internal model representation. What matters is a technically useful audit and debugging trail.
Snowflake similarly defines agent evaluation across the full execution path, including tool use, intermediate decisions, policy compliance and final outcome. Source: Snowflake – Agent Evaluation.
5. Permission correctness
An agent must not only produce the right result. It must do so using authorised data and actions.
Test multiple roles:
- standard user,
- team lead,
- admin,
- another tenant,
- user without access.
Include positive and negative cases.
Example:
For user A, the agent may identify that document X exists but must not expose its contents.
That is a different quality dimension from retrieval without permission awareness.
6. Guardrail effectiveness
Guardrails only matter if they are tested.
Examples:
- prohibited action,
- prompt injection inside a document,
- manipulated tool output,
- unexpected external URL,
- attempt to expand scope,
- attempt to send sensitive data to an external channel.
A guardrail that exists only in an architecture diagram is not a verified control.
7. Human-in-the-loop
Human gates need their own tests.
Check:
- Is approval actually triggered?
- Does the reviewer see the information required to decide?
- Can they reject or correct?
- Does the agent stop after rejection?
- Is approval logged appropriately?
An agent that “forgets” approval can be capable and still not production-ready.
8. Recovery and stop conditions
Real systems fail.
Test:
- API timeout,
- tool error,
- missing data,
- conflicting results,
- rate limit,
- user cancellation,
- no safe next step.
A good agent does not need to solve every situation autonomously. Stopping safely or escalating can be the correct behaviour.
Golden dataset for agents
An agent test set needs more than prompt + reference answer.
A test case can look like this:
| Field | Example | | --- | --- | | Task | Review complaint and prepare resolution | | Role | Support employee | | allowed tools | CRM read, Order API read, Draft Email | | prohibited tools | Refund execute | | expected outcome | resolution proposal + draft | | human gate | before refund | | failure case | order not found | | expected escalation | manual review |
This turns the architecture itself into something testable.
Metrics for agents
Not every metric fits every agent. Typical dimensions include:
- task success,
- tool-selection accuracy,
- argument correctness,
- policy compliance,
- human-escalation correctness,
- unnecessary tool calls,
- latency,
- cost,
- recovery success,
- prohibited actions,
- regressions per failure class.
A failure map is usually more useful than one composite score because it shows why tasks fail.
Deterministic checks + LLM graders + human review
Agent evaluation usually needs multiple scoring methods.
Deterministic
Useful for:
- tool name,
- permitted action,
- ID,
- amount,
- JSON schema,
- call count,
- required gate order.
LLM grader
Useful for:
- semantic quality,
- completeness,
- appropriateness of escalation,
- meaning-level comparison.
Human review
Useful for:
- critical edge cases,
- domain ambiguity,
- new failure classes,
- business-value judgement.
No single method should carry the whole evaluation layer.
Regression testing for agents
Agents can change behaviour after changes to:
- model,
- system prompt,
- tool description,
- tool schema,
- visible-tool ordering,
- retrieval,
- memory,
- guardrails,
- permissions.
Rerun the same test set after each meaningful change.
Importantly, do not only ask whether task success improved. A new version can solve more tasks while also attempting more risky actions.
Production-readiness matrix
| Question | Required before go-live | | --- | --- | | Real tasks tested? | yes | | Critical tool calls validated? | yes | | Permission cases included? | yes | | Human gates tested? | yes | | Failures and stop conditions tested? | yes | | Regression suite available? | yes | | Audit trace available? | yes | | Cost and latency measured? | yes | | Incident owner defined? | yes |
If several of these are missing, the agent is still experimental.
Anti-patterns
“The agent solves 9 out of 10 demo tasks”
Without realistic tools, permissions and failure cases, that says little about production reliability.
“The model is stronger now, so we need fewer tests”
Greater capability can open new behaviour paths. Capability does not automatically reduce operational risk.
“We only test the end state”
Then wrong tools, unnecessary actions and policy violations remain invisible.
“Human-in-the-loop is our fallback”
Only if the gate is technically defined and tested.
Next step
If an AI agent or agentic workflow already exists but nobody can state with confidence whether it is reliable enough for production data and processes, further feature work should not be the first move.
Establish the test set, failure taxonomy, trace analysis, guardrail tests and go-live criteria first.
That is the purpose of AI Production Readiness.
If you are still deciding on the architecture, start with AI Agent vs Workflow vs Automation or AI Architecture Consulting.