One engine, any model: a model-agnostic chat engine with prompt blocks per model

Third post in the series Going beyond a simple chat. The last one ended with a rule: a model is a service, not a dependency, and the ability to switch is not optional. This post is about how that ability is built, and about the part people underestimate: the prompt has to switch with the model.

“Model-agnostic” is three problems, not one

When I started making Don’t Spiral independent of a single model, I thought of it as a transport problem: wrap two vendor APIs behind one interface. That is the smallest of the three.

Transport. Each vendor has its own wire format, its own caching, its own tool-calling conventions, its own failure modes.
Roles. A chat app doesn’t use one model. Mine uses five roles: the chat itself, a classifier, fact consolidation, insights, and a utility role for small jobs. Each wants its own model, and the right one differs per role.
The prompt. A prompt tuned on one model is a wrong prompt for the next. If the model is a setting, the prompt can’t be a constant.

The first two are plumbing. The third is where the work went, and it’s what the second half of this post is about.

The model is a setting

Every role has an app setting, read from the database with a one-minute cache. A router looks at the model name and dispatches to the Anthropic or the Gemini implementation. Callers never know which vendor they’re talking to. Changing the chat model for the whole app is one admin edit, live within a minute, no deploy. Going back is the same edit.

On top of that sits a per-user pin: one account can be pinned to a specific model while everyone else stays on the default. That is how I used Gemini and Haiku on my own phone for two days each while production stayed on Sonnet, and how the bench runs two models against the same conversation in one evening.

Structured output through a forced tool call

A chat reply in my apps is never just text. Every turn carries a message plus metadata: the distress score, the quick-reply chips, the crisis and wind-down flags, the facts extracted from this turn. The first version asked the model for that as JSON in its text answer, with the format described in the prompt. It worked most of the time. Then Sonnet 4.5 started producing invalid JSON now and then, an unescaped quote inside a German sentence was enough, and when the parser failed the raw JSON leaked into the chat for the user to read.

The fix, in March, was to stop asking for text at all. Every turn is now a single forced call to a tool named respond_to_user, and the tool’s arguments are the response. The API validates the arguments against a JSON schema before I ever see them, so the output is structurally valid every time. The prompt lost its whole “response format” block and the rule “never break the JSON format” with it; the schema’s field descriptions carry that job now. And the message and its metadata still come from one call, so there is no second extraction pass and no drift between what the model said and what it reported about it.

That decision turned out to be what makes the engine model-agnostic. On Anthropic the forcing is tool_choice set to that one tool. On Gemini it’s function-calling mode ANY with the same declaration, after converting the schema to Gemini’s dialect. Every vendor can be forced to fill a schema, and the whole engine, the Flutter app included, is built on that one structured object. It doesn’t matter which model filled it in.

Provider quirks live in the provider class

The point of the transport layer is that every vendor-specific oddity has exactly one place to live, and that place is not the prompt. A few that live there now:

Caching. Anthropic needs explicit cache breakpoints, so the system prompt is sent as two blocks: the shared “DNA” cached across all users, and the session prompt cached per conversation category. Gemini caches implicitly and reports cached tokens in a different field. The caller sees one cache-read number.
The unescaped quote. Gemini intermittently fails to escape a double quote inside function-call arguments, and its parser cuts the string there. About 3.5% of messages came back truncated mid-sentence. The provider class detects the signature and retries once.
Plain text despite ANY. Once in a while Gemini returns prose instead of the function call. The provider falls back to text extraction instead of failing the turn.
Content policy. Bedfit is a sexual-health coach, and Gemini’s safety filter occasionally blocks a perfectly ordinary turn as prohibited content. The provider stores the blocked conversation as tuning data and hands the turn to Haiku. The user sees a reply, not an error. That fallback only exists because the router can route a single turn to another vendor.

None of these are reasons not to use a model. They’re the cost of using two, and the cost is paid once per quirk.

The prompt follows the model

Here is the part that took the longest to get right. My prompts are long, tuned over months, and they’re A/B tested: at any time two or three variants of the shared prompt are live. When Haiku needs a different length rule, I can’t edit three variants by hand and keep them in sync. And I can’t fork the prompt per model either, because then every content change has to land in every fork.

The mechanism I ended up with has two parts.

Base prompts carry markers. A region that a model might need to see differently is wrapped in a section marker, and there is one slot where model-specific text can be added:

<!-- section: message_length -->
...the length rule as written for the default model...
...

{{MODEL_ADJUSTMENTS}}

A per-model file supplies the differences. For each model family there can be a file named _model.haiku, _model.gemini, _model.sonnet. It contains override blocks that replace a marked section, and free text that fills the slot:

<!-- override: message_length -->
...the length rule as Haiku needs it...
Rules that only this model family needs, appended to the prompt.

At assembly time the server resolves the model’s family, loads the matching file, applies the overrides to whichever variant is active, strips the markers and fills the slot. No file for the family? Markers are stripped, the slot is empty, and the prompt is the same text it always was. One injection fixes all arms of the A/B test at once. The markers are validated when prompts are uploaded, so a typo fails early instead of silently changing nothing, and each per-model file starts with an evidence log: every rule in it has the date and the run it came from.

The first use was mundane. The shared prompt stated the length rule three times because Sonnet needs that. The two extra statements moved into _model.sonnet, so Sonnet’s effective prompt didn’t change by a byte, and Haiku got a _model.haiku with a single statement plus slack for landing moments. That was the “Haiku is the shorter model” mystery from the last post, resolved in the mechanism.

What goes in the blocks

The Gemini file for Don’t Spiral grew to sixteen numbered findings over six rounds in two days. Three of them, so you can see the shape:

It addressed the user by the ex-partner’s name on 6 of 10 turns. Sonnet and Haiku never did. A name rule.
“Never begin with That’s” made the tic worse: 10 occurrences became 13. A negative constraint on a message opening doesn’t bite, the model has committed to the opener before the rule applies. An enumerated list of allowed opener shapes fixed it.
It set the suggest-professional-help flag on 30 of 39 calm turns. Sonnet: 0 of 39. The decision moved to the server, which clamps the flag whatever the model says.

Two things about that list. First, not every fix is in the prompt: the clamp and the retry are server code. When a prompt rule fails twice, the server enforces it. That theme gets its own post later in the series. Second, the findings transfer. The positive-framing lesson from the Gemini rounds was ported verbatim into the Haiku file a day later, where it fixed the same opener tic on the first try.

Ask the model why

The most useful debugging method I found is also the cheapest, and it matters most when the model you’re tuning is a different family than the model that writes your prompts.

My prompts are written with Claude, in Claude’s dialect, with Claude’s conventions about how an instruction and an example relate. A Gemini model reads that text with different conventions. When it misbehaves, Claude guessing why is guessing. The model that misread the prompt is the one expert in the room on how it read it.

So when Bedfit’s Gemini model kept reciting the prompt’s example sentences verbatim, on turns they were never written for, I gave it its own transcript and the system prompt and asked two questions: what did you ignore, and why do you reuse the example sentences?

Its answer to the second: it treats a quoted example after an instruction as a gold-standard completion pattern, the intended output rather than conceptual scaffolding. Its answer to the first was a list: it had opened every reply with a validation formula, used abstract nouns where the coach uses concrete ones, ignored the sentence-length rule, and drifted into counsellor mode instead of training-buddy mode. Then it proposed the fix: write every example as a structural diagram with placeholders, forbid any phrase longer than two consecutive words from an example block, and it rewrote the style block in its own words.

Those proposals went into the Gemini file as the next version, labelled in the changelog as the model’s own. The recitation stopped. One of its proposals, a hard fifteen-word ceiling per sentence, then made the model drop the substance of knowledge-base results, so the following version softened it. Which is the rule: a model’s self-report is a hypothesis, and the bench decides.

Build your own

Everything in this post is a few files: a transport interface, two provider classes, a router, a prompt assembler, and a settings table. It carried Don’t Spiral through three vendors’ models in one week and a production switch to Haiku with revert one setting away. Two months later Bedfit was forked from the same engine and runs on Gemini Flash-Lite with Haiku as per-turn fallback. I didn’t extract a shared package when the second app came along, because abstraction boundaries guessed from one app are partly wrong, and I’m not sure I ever will. The two apps already do several things differently, so what works is evolution with backporting: each app moves on its own, and a fix that makes sense for the other one gets ported.

There are chat frameworks that give you all of that out of the box, and I’d still build it myself, for one reason: every piece of this engine that bought something was a change I made the same afternoon the bench found a problem. The retry for Gemini’s unescaped quote. The per-turn fallback to another vendor. The per-user pin. The injection markers. None of those were on anyone’s roadmap but mine.

A framework hands you its abstractions, and the day your product needs something it didn’t foresee, you’re writing an adapter against the framework instead of code against the API. The vendor APIs are small. A chat call, a tool declaration, a cache marker, a streaming event. The hard part was never the transport; it’s your product’s behaviour, and that part no framework contains.

And with a coding agent at hand, this is not rocket science anymore. The engine described here, including the Gemini provider with its quirks, took days, not months. What you get is an engine that does exactly what your app needs, that you understand to the last line, and that you can bend the moment your users show you where it has to bend.