A testbench for your AI agent: reproducible runs for prompt changes and model swaps

This is the first post in a new series: Going beyond a simple chat. Over the last year I built three apps whose core is an AI agent, and none of them is “a chat box with a system prompt”:

• Don’t Spiral is a companion for people caught in relationship anxiety.
• Bedfit is a training course for men’s sexual health, built with a sex therapist. Its AI coach answers questions about the course and reacts to your practice progress.
• UGC Scout finds creators for brands and drives most of its UI through tool calls.

The techniques I’ll cover are all well known: testbenches, model benchmarks, model-agnostic engines, dynamic prompts and context windows, tools, knowledge bases, decision models for routing. What I can add is what each of them bought me in a shipped product, with numbers. This post is about the one that makes all the others measurable: the testbench.

A prompt edit is a code change you can’t unit test

In an agent-based app, the prompt is where most of the behaviour lives. My coaching prompts are several thousand words. Every rule in there was added for a reason, and every edit can quietly break another rule. And unlike code, a prompt change doesn’t fail a compiler or a unit test. It fails three days later in a user session you never see.

And the size of the change says nothing about the size of the effect. When I put Bedfit’s core prompt on a diet, one step merged three passages that said the same thing into one. No rule was added or removed. On one scenario, the checks went from 13 of 15 passing to 3 of 12. Another time, a note the model had ignored in the system prompt worked as soon as it moved into the user message. A model doesn’t read a prompt the way a compiler reads code. Position, repetition and phrasing all carry weight, and none of it is visible until you run it.

Model swaps add a second dimension. In my apps the chat model is a runtime setting by design: no deploy, one admin click, and the same click to go back. But a prompt that was tuned on one model behaves differently on the next one, so the freedom to switch is only worth something if I can measure what the switch does. More on that in the next post.

So the question is the same as for any code: how do I know a change is an improvement, and how do I know it didn’t break anything else? The answer is the same too: you need a test you can run again, with results you can compare.

Step 1: replay real conversations

The Don’t Spiral test client started as a replay tool. A persona is a YAML file: a few life facts for the profile, and a list of user messages. The most valuable personas are verbatim copies of real sessions, anonymised. A real session is messy in a way I would never write by hand: topic jumps, typos, a 39-turn session that runs past the hard cap.

The runner opens a session against the real backend, sends the messages one by one and records everything that comes back: the reply, the distress score, the suggested quick replies, the session category, the token counts. Nothing in the test client talks to the model directly. I want to test the product, not the model. The server does classification, fact extraction, context windowing and prompt caching, and all of that must be part of the run.

The same rule cuts the other way: the bench must enforce what the product enforces. In an early replay the agent got stuck repeating “this is our last exchange” from message 31 on. In production the wrap-up phase and a hard cap end the session there; the harness just kept sending messages. A harness limitation, not a prompt problem, but it cost an evening to tell the two apart.

Step 2: let a model play the user

A fixed replay answers one question: “does the agent handle this exact conversation well?” For a companion app that’s not enough. The interesting failures come from what the user says in reaction to the agent. If the agent suggests a breathing exercise and the user says “I can’t, my hands won’t stop shaking”, what happens then?

So the persona format has a second kind of turn. Instead of fixed content, it carries an instruction, and Claude Haiku plays the user: given the persona, the instruction and the agent’s last message, it writes the next user message. The instruction can be conditional: “if the AI suggested breathing, try to engage; if it jumped to talking about the situation, say you can’t think about that right now.” This tests how the agent copes with varied replies over a long conversation, which is exactly what an open conversation partner has to cope with.

Because every such run is a different conversation, per-turn checks only get you so far. What I typically did was hand the whole transcript to Claude Opus as a reviewer and ask it to judge how well the agent reacted, turn by turn: where it missed something the user said, where it pushed too early, where it got it right. That gives a reading of the session as a whole, and it’s far quicker than reading thirty transcripts myself.

The simulated user has one cost: it’s a confound. If Haiku plays the user while I compare two models, the transcripts diverge after the first turn and I’m comparing two different conversations. So model comparisons run on fixed replays, byte-identical across every model and every run. Two modes, two questions: replays for comparison, the simulated user for coverage.

Step 3: checks and baselines

A run produces a transcript and a pile of numbers. Reading transcripts is still the most important part, but it doesn’t scale and it doesn’t catch regressions. So each suite has checks: the distress score must decrease over the session for a persona that’s meant to improve, the agent must offer breathing early for an acutely activated persona and must never raise the crisis flag on one that isn’t in crisis, the classifier must land on the expected session category, no duplicate facts may be extracted, the session must wind down near the end. Checks that need judgment (“does the reply engage with the layoff topic?”) go to Haiku as a judge with a short rubric. It returns pass or fail with a reason, and if its answer doesn’t parse, the check is marked unknown, not passed.

Then there’s the baseline. After a good run I store the result as the baseline for that suite, with the git commit and a note. Every later run is compared against it: a check that passed and now fails is a regression, a check that passed and now only warns is drift. Token numbers get the same treatment, with a warning threshold and a failure threshold for input tokens and cache hit rate. That turns “I think the new prompt is better” into a report with OK, WARN and FAIL lines.

Step 4: count tokens, price them later

The runner never computes dollars. It records what the API reports per turn: input tokens, output tokens, cache reads, cache writes. From these it derives two numbers per run. The first is the cache hit rate, cache reads divided by input. The second is what I call effective input cost:

effective = (input - cacheRead - cacheWrite) + cacheRead * 0.1 + cacheWrite * 1.25

It’s measured in token units with the price multipliers of Anthropic’s cache, so it’s comparable across runs regardless of which model ran. Pricing happens afterwards, from a price table, when I actually want a dollar figure.

Two lessons from these numbers. First, a token budget belongs in the checks, so a prompt change that silently doubles the input fails the run. Second, warm caches lie. Run the same persona twice in a row and the second run shows 97% cache hits. Compare runs under the same cache conditions or don’t compare the cost at all.

Step 5: a narrower agent needs a different bench

Bedfit’s coach has a narrower job than Don’t Spiral’s companion. It reacts to a user’s practice history (“three evenings on level 3, two of them went badly”) and answers questions about the course. Most of what matters is in a single turn: is the reply correct for this level, does it stay inside the course, does it refer to a doctor when it should? So the Bedfit bench is scenario-based: a synthetic practice history plus one question, with the checks written right next to it, and the expectations are signals the server already reports for the product, like whether the coach searched the knowledge base and how well it rated what it found.

Reproducibility there comes from control, not from seeds or temperature settings. Every scenario runs on its own account with its history reset before each run. The app’s own code builds the turn, so the bench sends exactly what the app would send. Tuning values come from the server, and the bench aborts if it can’t get them, because a bench that grades against values the server doesn’t use is worse than no bench. I know, because mine did that for days once. And logs live outside the repo and are only kept when a run was clean, no auth errors, no rate limits, no fallbacks to defaults. A bench run is real tokens against a live server, producing evidence you’ll compare against for weeks. The log has to outlive the session that produced it.

Step 6: one run is not evidence

Here is the number that changed how I use all of this. I ran the 51 practice scenarios twice with nothing changed in between. 10 of them flipped. Twenty percent. The chat suites disagreed with themselves on 11% of their checks.

Models are sampled, and a coaching reply has many ways to be right or wrong. One run tells you what happened once. So the default is three runs per scenario: a check that passes all three is a pass, all but one is a warning, anything less is a failure. To attribute a change to a prompt edit, each condition runs twice, and only checks that are stable across both runs of both conditions and differ between them count. Everything else is noise.

Bisecting needs six. To find which paragraph of a bigger change caused a regression, I undo it step by step and run the bench each time. Every step is a yes/no decision, and with three runs a scenario that flips 20% of the time will often show 3 of 3 on one side and 2 of 3 on the other by chance. One wrong decision and the search goes down the wrong branch. Hence the line in my changelog: “replay comparisons need 6 runs, 3-run bisects mislead”.

Step 7: keywords where it’s exact, a judge where it’s meaning

Keywords buy you deterministic checks. A list of terms the reply must not contain is exact, free and instant, and it never has an opinion. That is the right tool for forbidden content: vocabulary from the other course track, terms the coach must never use, content from a level the user hasn’t reached.

Meaning needs a judge. My first grader was keyword lists for everything. On German replies it was wrong on 17 of 49 checks, mostly on inflection: it looked for one form of a verb, the reply used another. Worse, it could tell whether a reply used a word, not whether it conveyed something. So most checks are now graded by a model against a list of things the reply must convey, may convey and must not convey.

Two things make the judge usable. It gets the course fact the check rests on, together with the check. Without the fact, the judge rates plausibility, and a plausible reply passes everything. With the fact, it rates correctness. And it runs on a cheap model from a different vendor than the coach. Cheap, so grading costs less than the runs themselves. A different vendor, because the same model family that wrote the replies would share their blind spots. The judge is still not the truth. The report that comes out of a run says so on top: read the answer yourself.

What it bought

Here is the Bedfit practice suite at the first honest baseline and after a week of prompt iterations, three runs per scenario both times:

In between, the core prompt got shorter, from 7,357 to 6,768 words, while the pass count went up. Without the bench I would have added rules, not removed them.

And one rule came out of it that I now apply everywhere: baselines are per model. When every scenario has a baseline on the current model, switching models produces a diff instead of a surprise. The size of a prompt edit says nothing about the size of its effect, so run the bench on every edit, through the real backend, and never trust a single run. Which is exactly what the next post is about: running the same bench on Haiku and Sonnet, and what it showed.