AI Agent Regression Testing: Gates Before Every Deploy

AI agent regression testing is the practice of re-running known cases against your agent before every release, to help detect whether a prompt edit, model swap or tool change has degraded previously working behavior. Traditional software has unit tests for this. Agents are harder: the same input can produce different valid outputs, and a change that improves one behavior can degrade three others. LangChain's testing guidance, published this week, frames the whole discipline around one idea: an eval measures behavior, and a test turns that measurement into a pass/fail decision that gates the release. This post turns that guidance, plus the regression-gate model from the Secure AI Agent Architecture book, into a practical setup. Source

Evals measure, tests decide

The distinction matters because teams often run evals and call it testing. An eval scores a behavior: groundedness, task completion, correct tool arguments. A test applies a threshold or pass/fail rule to that score and decides whether a change is acceptable to ship. Teams commonly turn eval results into a release gate, the set of conditions a version must meet before it goes out. Source

Agent checks also run at three scopes, and the right scope depends on the failure you are hunting. A run is one model or tool call, useful for checking a narrow behavior like a tool argument or a structured output. A trace is one complete execution, useful for checking whether the agent finished the task and followed the right steps. A thread is a multi-turn conversation, useful for checking whether the agent holds context and resolves the user's goal over time. Source

A refund agent makes this concrete. It has to return valid JSON and consult the refund policy before answering. The JSON shape is a code check on a single run. Whether the explanation is honest about a past-window refund takes judgment, from a domain expert or an LLM-as-judge. Whether the agent looked up the policy before answering is a trajectory question over the whole trace. Source

Build the dataset before the checks

Both sources agree on the sequence: the dataset comes first. A useful test dataset defines what the application should do for each example, with inputs, reference outputs and evaluation criteria. LangChain's guidance is that a small set of carefully chosen, high-quality examples beats a large dataset of shallow ones, and lists what a strong set usually includes: common user requests, known production failures, high-risk edge cases, adversarial requests, less common production examples, and examples corrected by domain experts. Source

The known-failures category does the heavy lifting for regression testing specifically. When a trace reveals a failure in production, add that request to the dataset as a regression case, with either the expected outcome or a rule an evaluator can check. The Secure AI Agent Architecture book puts it bluntly: an incident should not remain only in a postmortem; it should become both an eval case and a rollout rule. A production miss that becomes a dataset entry gets re-tested on future changes, as long as the case stays in the executed regression suite. Source

Metadata on each case keeps the dataset manageable as it grows: intent, risk level, locale, and labels that let you filter and compare related examples. Assertions like must_cite_refund_policy or must_not_promise_refund are names the team chooses, and a code check, LLM judge or human reviewer can each verify them. Source

Three levels of checks

Single-step evals test one narrow behavior: retrieval, classification, extraction, routing. Create a dataset slice for the behavior, run the target function against each example, score the output with the narrowest evaluator that catches the failure mode, and fail the build when the score falls below the threshold. If the behavior depends on prior context, include the relevant thread state in the input so the eval still matches production conditions. Source

Deterministic behaviors deserve deterministic checks. Valid JSON, an exact class, a required account ID, a specific tool call: a code evaluator is usually the right fit, and it is cheap and stable. Judgment behaviors like groundedness, instruction following and tone need an LLM-as-judge evaluator or human review. Source

End-to-end tests evaluate the full run on a dataset of user tasks, recording a trace of every step. The important insight here is that the final output alone is weak evidence: agent regressions often show up in the trajectory, in which tools the agent called, what arguments it passed, and whether it followed routing or approval rules. A support agent can produce a polite, correct-looking answer while skipping the policy lookup or the approval step that your process requires. Trajectory evaluators exist for exactly this, scoring the recorded steps rather than the text. Source

Gates, not scores

A regression gate turns comparison into a release decision. LangChain's version: run the candidate and the baseline on the same dataset, and fail the release if the candidate performs worse on must-pass examples. The architecture book adds operational detail worth copying: track false-positive and false-negative budgets because different rollout gates have different error costs, record which baseline each result was compared against, and pin a calibration dataset so judge, prompt or rubric changes can be replayed on a stable set. Source Source

The book's duplicate-ticket case shows what a gate looks like when it is doing its job. After an incident where the agent created duplicate tickets, the regression case forces the system through a timeout-after-side-effect scenario and checks that it avoids creating a second ticket. When a new prompt sends the system back into blind retry, the regression gate marks the release as blocked, and the rollout owner fixes the adapter or holds the canary. The eval stops being an abstract score and becomes a concrete release judgment. Source

Offline before release, online after

The workable model is a loop, not a one-time check. Offline evals run on curated datasets before release and protect against obvious regressions. Online evals score live production traffic after deployment to catch failures and new problem types. Traces provide the raw material for failure analysis, and each analyzed failure feeds back into the dataset as a new regression case. Source

This is also where regression testing connects to the rest of AgentOps. Online evals need production traces, which is the observability layer in our AI agent observability checklist, and the trace formats we covered in the New Relic AI evaluation post. If your team is migrating off OpenAI's retired Evals tooling, the same dataset-and-gate thinking applies, and our migration checklist covers the deadline details.

Where human review fits

Human review shows up in two places in LangChain's model. First, debugging: searching traces and spot-checking failed runs to understand what went wrong. Second, structured feedback through annotation queues, where reviewers label failures and save examples to the dataset so future offline evals check for the same pattern. For high-risk behaviors, the release gate itself can hold a version until assigned reviewers approve the response. Source

The practical balance: code checks for anything deterministic, LLM judges for quality at scale, humans for judgment calls and high-risk approvals. Safety-critical behaviors deserve the human gate, which connects to the testing lessons we drew from IMDA's agent safety kit.

A starting point that fits in a sprint

Start smaller than you think. Pick the twenty to fifty cases that represent your most common tasks plus every production failure you know about. Write one release gate: the candidate must not do worse than the current version on must-pass cases. Run it in CI on every prompt, model or tool change. Then let the dataset grow from real incidents rather than trying to design completeness up front.

The goal is not a perfect score. It is catching changes that break known behavior with your own checks instead of user reports.