An AI agent can look impressive in a prepared demonstration and still fail at the job it was built to do. The demo usually shows one successful path. Production adds incomplete inputs, changing data, unavailable tools, permission boundaries, and users who phrase the same request in many different ways.
Evaluation needs to measure the complete system, not only the final sentence.
Define success at the task level
Start with an observable task outcome. Did the agent gather the required information, use the correct tools, produce the expected artifact, and stop at the correct boundary?
Text quality can be part of the score, but it should not replace task completion. A polished answer is still a failure if the agent used the wrong account, missed required evidence, or claimed that an action succeeded when the tool returned an error.
Build cases from real workflow variation
A useful evaluation set contains more than happy paths. Include:
- normal requests expressed in different language;
- missing or contradictory inputs;
- tool timeouts and partial failures;
- permission-denied scenarios;
- stale or irrelevant retrieved context;
- requests outside the agent's allowed scope;
- cases that require clarification or human review.
Keep the expected behavior explicit. Sometimes the correct result is to abstain, ask a question, or return a structured error.
Inspect the trajectory
For an agentic system, the path matters. Record which tools were selected, the arguments supplied, the order of operations, the evidence retrieved, and the result of each call.
Trajectory checks can catch failures that the final response hides. An agent may eventually produce the right output after making an unnecessary expensive call, retrying a non-idempotent action, or accessing data it did not need.
Use deterministic checks where possible
Structured outputs should be validated with schemas. Tool arguments can be compared with expected constraints. Permissions, required citations, state transitions, and side-effect boundaries can be checked in code.
Model-based grading is useful for qualities such as clarity or whether a synthesis is supported, but it should complement deterministic checks. The evaluator itself needs calibration against human-reviewed examples.
Measure operational behavior
Quality is not the only production constraint. Track latency, token and tool cost, retry rate, timeout rate, and the percentage of runs sent to human review. Evaluate these measurements by task category because averages can hide one expensive or unreliable workflow.
Set budgets before launch. A system that completes the task but regularly exceeds the acceptable response time or cost still needs engineering work.
Test changes against a baseline
Every prompt, model, retrieval, and tool change should run against the same versioned evaluation set. Compare the candidate with the current production baseline and inspect regressions by scenario.
Do not compress the result into one score too early. A candidate may improve writing quality while weakening tool selection or safe abstention. The team needs to see those tradeoffs.
Learn from production carefully
Production feedback can expand the evaluation set, but raw user data should not be copied into tests without review and appropriate handling. Convert recurring failures into sanitized, representative cases with expected outcomes.
The goal is a feedback loop: observe a failure, classify its cause, add a regression case, improve the system, and verify that the change solves the problem without breaking another path.
Agent evaluation is product engineering. It turns an interesting demonstration into a system whose behavior can be measured, changed, and trusted over time.