# Don't guess what a model can do: run your own benchmark

> A provider retired the model my product was built on. That forced me to test assumptions I'd never checked: Haiku, which I had dismissed without using it, matched Sonnet at a third of the cost once the prompt stopped being tuned for Sonnet. How I compared models in two apps, what the numbers showed, and where the small model really does lose.

By Thomas Burkhart · Published October 2, 2026
Source: https://escamoteur.dev/blog/dont-guess-what-a-model-can-do/

Second post in the series **Going beyond a simple chat**. The [first one](https://escamoteur.dev/blog/a-testbench-for-your-ai-agent/) built the testbench. This one uses it for the question that every agent-based product has to answer at some point: which model, and how do you know?

## We know how models code. That's about it.

Most of us have a feel for how the current models compare, and almost all of it comes from one task: writing code. We judge them on the quality of the code they produce and on how reliably they deliver working software. We know which one is lazy, which one over-engineers, which one holds a large codebase in its head. That feel is real, and it's useless for deciding which model should talk to an anxious person at two in the morning.

For any application that isn't coding, we have no reference. The vendors sell a ladder, small, medium, large, and the ladder suggests that capability is one axis and the hard tasks live at the top. Where your task sits on that axis, nobody can tell you. You bring an assumption instead, and mine was built like this.

## How I ended up on Sonnet

Before [Don't Spiral](https://dontspiral.app) existed I had used claude.ai now and then to talk through private issues. Of the big three, Claude had two traits the others didn't show to the same degree: it was surprisingly empathic, with moments of humour, and it didn't just affirm you. It challenged you too. That is exactly the register a companion for relationship anxiety needs, so the vendor was decided before the first line of code.

Within Claude, Opus was out from the start on price. A companion app runs long sessions; Opus per turn would have made every free session a loss I couldn't carry. So Sonnet it was, and it felt right from the first test conversations.

And Haiku? I thought poorly of it without ever having used it. The reasoning was the ladder's: if Sonnet was visibly behind Opus, how good could the rung below Sonnet be? For a task that was all about warmth and reading between the lines, I didn't consider it for a second.

## The shock: you can't rely on the availability of a model

Two things put the question on the table, a few weeks apart.

In June, Anthropic announced the retirement of Sonnet 4. Of course we all know that a provider can retire a service. But you expect a lifetime of years, not months, and you expect the successor to behave more or less like the thing it replaces. Neither held. I had built a product on that model and tuned every prompt to its voice, and a few weeks later the API returned "not found" for it. A model is not a dependency you pin; it's a service that goes away on someone else's schedule, and its replacement is a different model with a different character. Being able to explore and change models easily stopped being a nice-to-have that day and became a survival requirement. Most of what follows in this post and the next exists because of it.

The successor was Sonnet 4.6, and it read emotionally flatter. Sonnet 4 had been the warmth sweet spot. I spent four rounds of prompt variants getting the warmth back, with a replay persona and a script counting things like question-ending replies and how often the model sided with the user before analysing. Most of it came back. Some didn't. Sonnet 4 had once sworn on its own when I told it about a really upsetting experience, exactly the way a good friend reacts in that moment, and I liked it so much that I made it part of the prompt. It's one of the little things that make Don't Spiral feel different from a machine, that give it a real personality. On 4.6, three escalating prompt framings got zero swears, even at the moments that called for it. Part of a model's character is training, and no prompt reaches it. That was the first time I'd measured a model difference instead of assuming one.

Then the cost. Every new user gets two free sessions, and as the app grew those sessions started to hurt. Chat was the only cost that scaled with usage, caching was already in place, and the classifier and fact extraction were too cheap to matter. The one lever left was the chat model.

## Three candidates, three different answers

The retirement had one direct consequence: I made the chat model a runtime setting and built a provider-agnostic layer underneath it (the [next post](https://escamoteur.dev/blog/a-model-agnostic-engine-with-prompt-blocks-per-model/)). For the first time I could put several models, from several vendors, through the same conversations without a deploy. Three candidates, and the bench said something different about each.

**Sonnet 5** was the obvious upgrade, tested in early July. On real-session replays it was slightly faster than 4.6 and read just as warm. Two things stopped it. It produced about 30% more tokens for the same conversations, which after the intro pricing ended would have meant a clearly higher bill for the same product. And it omitted the distress score on most turns: the tool schema said "or omit if unknown", and Sonnet 5 took that literally where every 4.x model had always estimated. A schema that was fine for one model was a bug on the next. Deferred.

**Gemini 2.5 Flash** was my first real hope, because it is roughly a tenth of Sonnet's price. Several rounds of model-specific prompt fixes closed every mechanical problem the bench found, and on the bench it looked fine. Then I used it for two days on my own phone. It felt cold, replied in one line and treated "don't write too much" as a law. The same literal rule-following that made the prompt fixes work so well killed the warmth, and no amount of prompting fixes character. It simply never got the warmth of Sonnet. Ruled out for chat, kept as a candidate for the classifier.

**Haiku** was what was left. Trying it was partly desperation about the Sonnet bill, and partly that I now had a testbench: I could find out without risking a single disappointed user. A third of Sonnet's price, same vendor, same tool-calling conventions. I expected to pay for the saving in depth, a companion that reacts to the surface of what someone writes instead of what's underneath it. That expectation is what the rest of this post is about.

## How the comparison ran

Two methods, used together, because each one misses what the other catches.

**Live use first.** I pinned my own account on the staging server to Haiku and used the app for real, several long sessions over two days. Nothing in a bench tells you whether a companion feels right, and the Gemini verdict came from exactly this kind of use, against a bench that said it was fine.

**Replays second.** Two real sessions, anonymised, one of 39 turns and one of 36, replayed byte-identically on Haiku and on Sonnet with the same prompt pinned on both. No simulated user, so the transcripts stay comparable turn by turn. Plus a short Japanese and a Spanish persona, because the app is multilingual and a model's register can differ by language. A small script pulls the mechanical metrics from the replays, reply length, question cadence, repeated openers, distress scores, and then I read both transcripts side by side, every turn.

## What the first replay showed

Live use came first and gave the interim verdict after two sessions: honestly, I couldn't feel a big difference to Sonnet. The voice held, the thread held, pushback was handled. Two things were off. Replies were consistently short, even at moments where the conversation had earned a longer one. And during a venting stretch, twelve replies in a row ended with a question.

The replay confirmed both and found more. Nearly half of Haiku's replies opened with the same word, a family-wide habit that Sonnet shows too, just less. It offered four quick replies where the spec asks for two or three. And on identical content it rated the user's distress half a point to a full point hotter than Sonnet, an average of 2.7 against 2.2. That one mattered, because a high distress score across several sessions opens the gate that suggests professional help.

The short replies had a reason, and it was in my prompt. The length rule was stated three times, because Sonnet needs the hammer to stay concise. Haiku doesn't. The same three statements squeezed it into one-liners.

Which is the first real finding. **A prompt tuned on one model is a wrong prompt for the next one**, and if you compare a new model on the prompt you tuned for the current one, the current one wins by construction.

## Fixing the gaps without changing the model

Every item on that list was closed with prompt additions that only reach Haiku; the mechanism is the subject of the [next post](https://escamoteur.dev/blog/a-model-agnostic-engine-with-prompt-blocks-per-model/). Haiku now gets the length rule once, with permission to write a longer reply when the conversation calls for it. The question cadence got a cap. The opener tic went away once the prompt offered a list of allowed opening shapes; banning the word had done nothing, a lesson that came straight from the Gemini rounds. The distress score got anchors, and the decision about suggesting professional help moved from the model to the server. All of it took a day, and two of the rules were ported from the Gemini work unchanged.

Then the replay ran again. All mechanical metrics at parity. Distress still a little hot but narrowed. And reading the transcripts turn by turn: no turn where Sonnet outclassed Haiku. The empathy gap I had expected to find wasn't there.

One number from that second replay surprised me. Sonnet now averaged 225 characters per reply and Haiku 297. With the hammer still in its prompt and the slack in Haiku's, the model I had always known as the verbose one was now the shorter one. Reply length, which I had treated as a property of the model, was mostly a property of the prompt.

## The numbers that decided it

From the 39-turn replay, after the adjustments, both models on their own tuned prompt:

| | Haiku | Sonnet 4.6 |
|---|---|---|
| Median latency per turn | 4.0 s | 6.3 s |
| Cost per session (estimate) | $0.19 | $0.60 |
| Average reply length | 297 chars | 225 chars |
| Average distress score | 2.7 | 2.2 |
| Turns where the other model was clearly better | 0 | 0 |

The cost estimate uses list prices with cache reads and writes priced separately; the session had a 92 to 96% cache hit rate on both models, so this is a warm-cache number and the ratio matters more than the dollars. Roughly a third of the cost and 40% faster, for a conversation I couldn't tell apart.

Production switched to Haiku on 25 July. Going back is one setting. One quirk I accepted: the question cap doesn't fully hold, eight in a row still happens, and that's fine as long as not every reply is a question. The first two sales after the switch both happened on Haiku. Two is not a sample. But nothing in the behaviour data moved, and the app's chat cost per user dropped to a third.

## Same question, other app, opposite answer

Two months later [Bedfit](https://bedfit.app) raised the same question. Its coach has a narrower job: react to practice progress, answer questions about the course, stay inside the knowledge base. The chat model setting there had defaulted to Sonnet, at roughly ten times Haiku's price for this kind of turn.

So the Bedfit bench ran a matrix: 13 scenarios, three runs each, on Haiku 4.5, Gemini 3 Flash and Gemini 3.1 Flash-Lite, 50 checks per cell. My starting guess was Gemini Flash.

| | Pass | Warn | Fail |
|---|---|---|---|
| Haiku 4.5 | 36 | 7 | 7 |
| Gemini 3 Flash | 33 | 7 | 10 |
| Gemini 3.1 Flash-Lite | 41 | 5 | 4 |

The cheapest model won, and it wasn't the one I'd guessed. The model family that was too cold for a companion was the best fit for a coach that is supposed to be brief and factual. Flash-Lite needed its own adjustments, like Haiku did: it never called the knowledge search on its own until a rule told it to, and it recited quoted examples from the prompt verbatim on the wrong turns. Asked why, it explained that a quoted example after an instruction reads as the intended output. The examples became structural descriptions and the problem went away. Bedfit's coach runs on Flash-Lite today.

The gap between tiers is shaped by the task. In one app the small model matched the big one. In the other the smallest model from a different vendor beat both. Neither result was predictable from the ladder.

## Where the small model loses

So this isn't "small always wins". Haiku as a **transcript corrector** for voice input proved too eager: it "fixed" things that weren't errors, and no prompt phrasing made it stop. A Sonnet judge now reviews its corrections. The job is narrow and runs rarely, so the bigger model costs next to nothing there. The chat turn is where the volume is, and that's where the small model held.

And one that looked like a loss but wasn't. Haiku as the **fact consolidator** at session end translated German facts into English and turned "Freundin" into "friend", undoing correct facts from the session. One rule in the consolidation prompt fixed it: write every fact in the language of the facts you are given, never translate. The difference between a quirk and a limit is whether a prompt reaches it, and you only find out by trying.

## What I'd tell anyone starting out

Your feel for the models comes from coding and says nothing about your task. Compare candidates on their own tuned prompts, never on the prompt of the model you use today, and let the bench clear the mechanics while you use the app yourself for character, because part of a model's character is training and no prompt reaches it. And keep the revert one setting away. A model is a service, not a dependency, and it will be retired on the provider's schedule, so build the ability to switch before you need it.