Short answer
Regression testing for AI agents means re-running an agent on a fixed set of known cases after every change, to catch answers or actions that got worse. Changes include a new model, an edited prompt or procedure, and new knowledge.- Also called
- AI regression testing, agent regression testing, agentic regression testing
- Run after
- Model updates, prompt or procedure edits, knowledge changes
- Test set
- Real past conversations with a known good outcome
- Catches
- A fix in one place that breaks another
In ordinary software, a regression test checks that code which used to work still works. AI agents need the same check for a harder reason: their behaviour can change when nothing in your own code did. A model update ships, someone rewords an instruction, a help article changes, and answers that were right last week drift.
Why AI agents regress
An AI agent has more moving parts than its code.
- The model. A new model version can answer the same question differently.
- Prompts and procedures. Rewording one instruction changes how the model reads the others.
- Knowledge. An article is updated, removed or contradicted by a newer one.
- Tools and data. A new connection, or a change in the data it returns.
- Fixes. A change made for one request type that also affects a similar one.
A store rewords its returns instruction to stress the 30-day window. The agent starts applying it to exchange questions it was getting right, and nobody notices until customers complain.
In Aide, the audit log records every change to procedures, topics, knowledge and agent settings with who made it, when, and a before and after view, so when an answer that was right last week changes, the team can find the edit behind it.
How regression testing for AI agents works
The method is simple. The discipline is running it every time.
- Build a test set from real conversations, including ones the agent once got wrong and was fixed for.
- Write down the good outcome for each: the facts in the reply, the action taken, or the handoff.
- Re-run the set after the change.
- Grade the new outputs with rules, an AI judge or a person, and have a person review every failure.
- Compare with the last run, and hold the change if something that passed now fails.
Because an AI agent can word the same correct answer many ways, the test checks meaning and outcome, not exact text.
What to test in a customer service agent
Each kind of failure needs its own cases.
| What to check | Example failure |
|---|---|
| Facts | Quotes a 14-day return window when the policy says 30 days |
| Actions | Offers to cancel an order that has already shipped |
| Handoffs | Answers a legal threat instead of passing it to a person |
| Scope | A damaged-item procedure starts on a late-delivery question |
| Format and tone | Replies get longer, or drop the order number |
With Aide, some of these checks also run on every reply before it is sent, whatever changed upstream: verification catches a wrong price, stock level or order detail, a promise the policies don't back, a leftover placeholder or the wrong language, rewrites the reply once, and hands it to the team if the rewrite doesn't fix it.
Regression testing with AI, the other meaning
Many searches for AI regression testing mean the reverse: using AI to regression test ordinary software. AI test tools generate test cases from code changes, repair tests when a page layout moves, choose which tests to run first, and check code written by coding agents before it reaches production. That is a QA practice for engineering teams. This page is about testing the AI agent itself, though the grading ideas carry over.
The practice this page rejects is shipping agent changes on faith. A change that fixes one answer and quietly breaks three others is a step backwards, however good the fixed answer looks.
Where Aide fits
Aide tests in two places: each procedure before it goes live, and the whole agent every night.
Before go-live, an edit to a live procedure stays a draft while the live version keeps running. The team checks which recent real conversations the draft would pick up and which it would decline, replays a past conversation with the draft, reads what changed against the live version, and only then sets it live. Every version is kept with who set it live and their note, and an older one can be restored. See testing before go-live.
Every night, Aide replays its eval cases against the agent's current setup and records which pass. The cases come from real work: drafts the team rewrote, replies a teammate marked Wrong, and recurring failures grouped into themes, each written from what the team actually sent. Teammates can edit what a case expects or retire it. When Aide adjusts its own writing style from the team's rewrites, the change is kept only if every case that passed that night still passes, and it is undone on its own if more cases start failing in the days after.