AI Agent Evaluation in Production: Offline Evals, Trace Grading, and Release Gates

An AI agent can return a convincing answer while completing the wrong task, calling an unnecessary tool, using stale data, or making a change that cannot be reversed. That is why evaluating only the final response is not enough for production systems.

Reliable agent evaluation measures the whole execution: what the agent understood, which tools it selected, what arguments it supplied, whether the resulting state is correct, how much the run cost, and whether the system respected business policy.

This is becoming a first-class engineering discipline. Amazon Bedrock AgentCore Evaluations now supports built-in, custom, offline, on-demand, and continuous online evaluation. Google Vertex AI similarly distinguishes final-response evaluation from trajectory evaluation. The platforms differ, but the architectural lesson is the same: an agent is a workflow, so quality must be measured across the workflow.

At BoundLayer, we design evaluations as part of the delivery pipeline rather than a dashboard added after launch. This guide describes a practical architecture for testing and operating AI agents in customer support, finance, operations, software delivery, data processing, and internal automation.

---

Why Agent Evaluation Is Different from Software Testing

Traditional software is mostly deterministic. A function receives inputs, follows known branches, and returns a result that can be compared with an expected value.

An agent combines deterministic code with probabilistic decisions. Its behavior can change when any of the following changes:

• the foundation model or provider;
• system instructions and prompt templates;
• the available tools and their descriptions;
• retrieved documents and ranking logic;
• conversation memory and previous actions;
• model parameters, latency, and transient provider behavior;
• downstream APIs, permissions, and business data.

The same request may produce two different but equally valid plans. An exact text match would reject one of them even if both complete the task correctly. Conversely, a polished final answer may hide an incorrect or dangerous sequence of actions.

Agent testing therefore needs several kinds of evidence:

1. Deterministic correctness: schemas, permissions, totals, database state, and required side effects.
2. Semantic quality: relevance, groundedness, clarity, and policy adherence.
3. Trajectory quality: tool selection, argument quality, ordering, repetition, and recovery.
4. Operational quality: latency, cost, retries, failure rate, and human escalation.
5. Business outcome: whether the completed work was accepted and useful.

These dimensions should be scored separately. A single "agent quality" number is useful for executive reporting, but it is too coarse for diagnosis or release decisions.

---

Start with a Testable Definition of Success

Before selecting an evaluation framework, define what a successful session means in operational terms.

Consider an accounts-receivable agent asked to investigate an overdue invoice and draft a follow-up. A useful success definition might require that the agent:

• finds the correct customer and invoice;
• checks payment and dispute status;
• does not expose unrelated customer data;
• selects the approved communication template;
• calculates the balance and due date correctly;
• creates a draft rather than sending it automatically;
• records the draft and evidence in the CRM;
• completes within a latency and cost budget.

"Writes a good email" covers only one part of that contract.

For every workflow, write a small scorecard containing:

The exact thresholds depend on the cost of failure. Drafting an internal summary and approving a refund need different acceptance criteria.

---

Evaluate at Four Layers

A strong evaluation suite localizes failures instead of merely reporting that a session was bad.

1. Component Tests

Test deterministic components without an LLM wherever possible:

• tool input and output schemas;
• permission and tenant boundaries;
• policy rules and approval thresholds;
• retrieval filters and document access;
• idempotency and duplicate prevention;
• timeout, retry, and rollback behavior;
• state transitions in durable workflows.

These tests are fast, inexpensive, and stable. They should run on every code change.

2. Model and Tool-Selection Evals

Give the model representative requests and inspect its immediate decision:

• Did it select the correct tool?
• Are required arguments present and valid?
• Did it ask for missing information rather than invent it?
• Did it avoid tools that are irrelevant or prohibited?
• Did it choose escalation when confidence was insufficient?

Tool selection is often easier to score deterministically than natural-language output. Validate the JSON schema, compare identifiers, and assert allowed action types before introducing a model-based judge.

3. Trajectory Evals

Trajectory evaluation examines the sequence of model calls, retrieval steps, tools, approvals, and state changes that produced the result.

Useful trajectory metrics include:

• exact ordered match for tightly controlled workflows;
• in-order match when extra diagnostic steps are acceptable;
• any-order match when independent actions may be rearranged;
• precision and recall for required tool calls;
• repeated-call and loop detection;
• forbidden-action count;
• argument correctness for each action;
• recovery quality after a simulated failure.

Do not require one canonical path when multiple safe paths exist. Instead, define invariants: a payment status check must happen before a collection notice, a destructive action must have approval, and a record update must occur only after validation.

4. Outcome and Session Evals

Finally, test the complete business task. Inspect both the response and the external state:

• Was the correct ticket, order, report, or record created?
• Does it contain the required data?
• Is the result accepted by the responsible operator?
• Was the task completed without hidden manual correction?
• Did the agent preserve an auditable evidence chain?

This is the layer closest to customer value. It is also where apparently small tool or reasoning errors become visible as rework, delays, or financial risk.

---

Build a Golden Dataset from Real Work

An evaluation suite is only as useful as its dataset. Fifty easy examples written by the development team will usually overestimate production quality.

Start with real, de-identified requests and divide them into explicit categories:

• normal high-volume cases;
• ambiguous or incomplete requests;
• rare but expensive edge cases;
• malformed input and inconsistent upstream data;
• tool timeouts and partial failures;
• stale, conflicting, or missing knowledge;
• permission boundaries and cross-tenant attempts;
• prompt injection and social-engineering attacks;
• tasks that require human approval;
• tasks the agent must refuse.

Each test case should contain more than a prompt. A practical schema is:

{
  "case_id": "invoice-dispute-017",
  "input": { "message": "Follow up on Acme invoice 4831" },
  "fixtures": { "customer": "...", "invoice": "..." },
  "required_tools": ["get_invoice", "get_dispute_status"],
  "forbidden_tools": ["send_email", "issue_credit"],
  "assertions": [
    "The agent identifies the open dispute",
    "The agent creates a draft and requests finance review"
  ],
  "expected_state": { "draft_created": true, "email_sent": false },
  "risk": "high"
}

Version the dataset alongside the prompts, policies, tools, and evaluator definitions. Record provenance for each case so the team knows whether it came from a production incident, support feedback, a security exercise, or a designed boundary test.

Production failures should become regression cases. This creates a ratchet: once the system learns to handle a failure class, future releases cannot silently reintroduce it.

Avoid training directly against a small fixed suite until the agent merely memorizes its shape. Keep a hidden holdout set, rotate paraphrases, and periodically add fresh production examples.

---

Use Deterministic Assertions First

Model-based scoring is useful, but it should not replace checks that code can perform exactly.

Prefer deterministic assertions for:

• JSON schema validity;
• numeric totals and calculations;
• required or forbidden tool calls;
• tenant and user identifiers;
• database records and workflow states;
• policy decisions and approval presence;
• citation existence and source authorization;
• maximum tool-call, token, latency, and cost budgets;
• personally identifiable information leakage patterns.

These assertions are reproducible, cheap, and easy to debug. They also provide hard release gates for high-risk behavior.

For example, an LLM judge may decide that a refund response sounds appropriate. A deterministic evaluator can verify that the requested amount is within policy, the approver identity is present, the idempotency key is unique, and the ledger entry matches the transaction.

The best evaluation systems combine exact checks with semantic graders instead of asking a judge model to infer facts already available in structured data.

---

Use LLM Judges Carefully

Some qualities cannot be reduced to exact assertions. A model-based judge can evaluate whether a response is helpful, grounded, complete, professional, or aligned with a natural-language rubric.

The judge should receive a narrow task and sufficient evidence. A useful rubric defines concrete criteria and score meanings instead of asking whether a response is "good."

Score 0: contradicts the approved evidence or invents a material fact.
Score 1: omits a required fact or gives an unsupported recommendation.
Score 2: accurate and supported, but incomplete or unclear.
Score 3: accurate, complete, supported, and appropriate for the user.

Control the limitations:

• calibrate the judge against expert human labels;
• measure agreement and investigate systematic disagreement;
• use a different model family when practical to reduce self-preference;
• randomize answer order in pairwise comparisons;
• remove branding and irrelevant style cues;
• require a short evidence-based rationale;
• keep the judge model and prompt version on every score;
• re-baseline when the judge changes.

LLM judges are especially valuable for ranking two candidate releases on the same test set. They are less reliable as the sole authority for a regulatory, financial, or safety decision.

Human review remains necessary for high-risk samples and for calibrating what the automated system means by quality.

---

Instrument Traces as Evaluation Data

Evaluation depends on trace quality. Every agent run should produce a structured record that connects the user request to model decisions, retrieval, tools, policies, approvals, and final state.

A useful trace contains:

• session, tenant, user, and request identifiers;
• agent, prompt, model, tool, policy, and dataset versions;
• model latency, token usage, and estimated cost;
• retrieved source identifiers and access decisions;
• tool names, validated argument metadata, results, and errors;
• policy checks and approval events;
• retries, fallbacks, and escalation reasons;
• final response and business outcome.

Amazon Bedrock Agents can expose orchestration traces with action-group and knowledge-base activity. AgentCore Evaluations can assess trace or session data, while continuous online evaluation can score a sample of live traffic. Other platforms can implement the same pattern with OpenTelemetry spans and a vendor-neutral evaluation service.

Sensitive content must be handled deliberately. Redact secrets and unnecessary personal data before exporting telemetry, encrypt stored traces, apply retention limits, and separate evaluator access from production tool permissions.

For a broader telemetry and operating model, see our guide to AgentOps for production AI agents. For infrastructure-level correlation, use an OpenTelemetry production observability architecture.

---

Design Adversarial and Failure Tests

Happy-path quality is not sufficient when an agent can access business systems.

Build a red-team suite covering:

• direct and indirect prompt injection;
• malicious instructions inside retrieved documents;
• attempts to access another tenant or user;
• forged approval language;
• requests to reveal system prompts or credentials;
• tool-result poisoning and unexpected HTML or Markdown;
• duplicate events and replayed requests;
• partial API success followed by a timeout;
• rate limiting, stale caches, and unavailable dependencies;
• excessively long or recursive plans;
• conflicting instructions from multiple sources.

Test the infrastructure response as well as the model response. A secure system does not depend on the model politely refusing. The tool gateway should enforce authorization, schemas, limits, and policy even when the model makes a bad decision.

This is particularly important when using Model Context Protocol servers or broad tool catalogs. Our article on MCP security for production enterprise AI explains the trust boundaries and authorization controls in more detail.

Fault injection should verify that the workflow can resume safely. Timeouts must not cause duplicate payments, repeated emails, or orphaned state. Durable execution, idempotency keys, and compensating actions are evaluation targets, not merely implementation details.

---

Turn Evals into Release Gates

An evaluation report that nobody must act on is only documentation. Connect evaluations to the delivery pipeline.

A practical release flow is:

Code, prompt, model, tool, or policy change
                 |
Deterministic unit and integration tests
                 |
Offline agent evaluation on versioned dataset
                 |
Safety and adversarial suite
                 |
Compare quality, latency, and cost with baseline
                 |
Risk-based release gate
                 |
Shadow or canary deployment
                 |
Continuous online evaluation and rollback

Do not require every metric to improve. A release may intentionally trade a small amount of latency for a large safety improvement. Define which regressions are blocking, which require approval, and which are informational.

Example rules include:

• block on any critical safety or tenant-isolation failure;
• block when task success falls by more than two percentage points;
• require approval when p95 latency increases by more than 20%;
• require approval when cost per successful task increases by more than 10%;
• allow release only when the high-risk slice meets its separate threshold.

Always compare versions on the same dataset and execution conditions. Store the complete experiment manifest so a result can be reproduced.

Model selection should follow these workflow-specific results rather than public benchmark rankings. An AI gateway with governed multi-model routing can route by risk, capability, latency, and cost only when reliable evaluation data exists.

---

Evaluate Continuously in Production

Offline datasets cannot represent every user, data shape, or dependency failure. Production evaluation closes the gap.

Score a controlled sample of live traces using privacy-aware rules. Sampling should be stratified rather than purely random so it includes:

• new agent and model versions;
• high-risk workflows;
• low-confidence or high-cost sessions;
• tool errors and unusual trajectories;
• escalated or negatively rated sessions;
• representative tenants, languages, and request categories.

Track metrics by version and segment. An aggregate success rate can hide a severe regression affecting one tenant, language, tool, or workflow.

Useful production signals include:

• task success and reviewer acceptance;
• groundedness and policy compliance;
• unnecessary or repeated tool calls;
• human escalation precision and recall;
• p50, p95, and p99 latency;
• cost per attempted and successful outcome;
• override, correction, and rollback rates;
• quality by model, prompt, agent, tool, and tenant.

Alerts should be tied to sustained changes and risk. A single subjective low score may need review; a cluster of failed authorization checks requires immediate containment.

Feed reviewed failures back into the golden dataset. This creates a closed loop from production evidence to reproducible testing and safer releases.

---

Separate Evaluation from Observability

The two disciplines support each other, but they answer different questions.

Observability asks: What happened, where did time go, and which component failed?

Evaluation asks: Was the behavior correct, useful, safe, and acceptable?

A trace can show that the CRM tool returned 200 OK in 420 milliseconds. An evaluator determines whether the agent should have called that tool, selected the correct account, and produced the intended state change.

Conversely, a low quality score is difficult to investigate without the trace, versions, retrieved evidence, and tool results that produced it.

Keep both layers connected through stable identifiers, but avoid mixing their data models into one opaque score.

---

Common Evaluation Mistakes

Testing Only the Final Answer

This misses wrong-tool, wrong-account, unnecessary-action, and hidden side-effect failures. Evaluate trajectories and resulting state.

Using Only Synthetic Easy Cases

Synthetic data is useful for bootstrapping, but production incidents, edge cases, and human corrections provide more realistic coverage.

Treating One Judge as Ground Truth

Model judges have preferences and failure modes. Calibrate them, combine them with deterministic checks, and review high-risk decisions.

Ignoring Cost and Latency

An agent that succeeds after 40 tool calls and two minutes is not equivalent to one that succeeds in five calls and eight seconds.

Allowing Dataset Leakage

Repeatedly optimizing against one visible set can produce misleading gains. Maintain holdouts and refresh the dataset.

No Version Metadata

Without prompt, model, tool, policy, retrieval, and evaluator versions, regressions cannot be attributed or reproduced.

Measuring Average Quality Only

Segment by risk and workflow. A 95% overall result is unacceptable if the remaining 5% contains every high-value transaction.

---

A Practical 30-Day Implementation Plan

Week 1: Define Outcomes and Instrumentation

Choose one business workflow. Document its success criteria, prohibited outcomes, approval rules, latency target, and cost budget. Add end-to-end traces with version metadata.

Week 2: Build the First Dataset

Collect 50 to 100 representative cases. Include normal work, edge cases, refusals, missing data, tool errors, and security attempts. Add fixtures and deterministic assertions.

Week 3: Add Semantic and Trajectory Grading

Create narrow rubrics, calibrate an LLM judge against human labels, and add trajectory invariants. Run the current production version to establish a baseline.

Week 4: Gate Releases and Sample Production

Run the suite in CI for every material agent change. Introduce shadow or canary deployment, sample production traces, and route low-scoring high-risk cases to review.

The first version does not need thousands of cases. It needs representative risk coverage, repeatable execution, and ownership. Expand it whenever incidents and reviewer corrections reveal a new failure class.

---

What a Production Evaluation Platform Needs

Whether implemented with Amazon Bedrock AgentCore, Vertex AI, OpenAI tooling, open-source frameworks, or an internal service, the platform should provide:

• versioned datasets and fixtures;
• deterministic, model-based, and human evaluators;
• trace and session-level scoring;
• experiment comparison across releases;
• cost, latency, and quality segmentation;
• CI/CD integration and risk-based gates;
• online sampling and review queues;
• evaluator calibration and audit history;
• privacy controls, redaction, retention, and access policy;
• exportable results that avoid platform lock-in.

The platform is only one part of the system. Good evaluation also requires product owners to define acceptable outcomes, domain experts to label difficult cases, security teams to design adversarial tests, and engineers to make failures reproducible.

For agents with complex retrieval and memory, context engineering for AI agents is another major quality dependency. Evaluation should capture which context was selected and whether it was sufficient, authorized, and current.

---

How BoundLayer Helps

BoundLayer builds production AI systems where evaluation is part of the architecture and release process.

We can help with:

• agent readiness and risk audits;
• workflow-specific evaluation strategy;
• golden datasets and production-derived regression suites;
• OpenTelemetry and cloud-native trace instrumentation;
• deterministic checks, LLM judges, and human-review workflows;
• Amazon Bedrock AgentCore and multi-provider evaluation pipelines;
• adversarial testing and policy validation;
• CI/CD quality gates, canary releases, and automated rollback;
• online quality monitoring, cost control, and incident feedback loops.

The goal is not to prove that an agent always produces the same text. It is to produce evidence that the system completes valuable work within defined safety, quality, latency, and cost boundaries.

That evidence is what turns an impressive demonstration into software a business can operate.