Building my own multi-model coding harness with pi

In my last post I described how Kimi wrote most of a real feature in my app for about €20, while a DeepSeek model reviewed it and a small Qwen model ran the whole session. Several people asked the same thing: what did you actually run that on?

The answer is pi, a small open-source coding agent, plus about a dozen agent definitions and a handful of extensions. I built it as a backup for the days my Claude subscription runs out. This post is the setup, why I chose pi, and the traps I stepped into on the way.

What I wanted: a backup for when Claude runs out

The main reason was simple. I do almost all my work in Claude Code on a subscription, and when my usage limit ran out in the middle of the week, I couldn’t work at all. As a one-person business, that’s not a risk I want to carry. I wanted a second setup that works without Claude, with models I can pay per token, so a used-up quota means a slower day instead of a lost one.

A backup only helps if it’s good enough to actually work in, not just to fire off the odd job. So once I was at it, I wanted it to be good at something Claude Code can’t do: using a different model for each kind of work.

• Reading a lot of code to find out where something lives needs a big context and a low price, not brilliance.
• Implementing a well-specified slice needs a strong coding model, ideally a cheap one.
• Reviewing needs a model that didn’t write the code, ideally from a different family, because models tend to overlook their own mistakes.
• Design decisions that no test can check need the strongest reasoning I can get. With quota left that’s Claude, and without it, the design has to happen somewhere else.

So I wanted a harness that runs without Claude, where each agent has its own model and its own provider, and where Claude is still one call away as long as my subscription lets me.

Why not just Claude Code?

I live in Claude Code, so that was my first try. But officially, Claude Code only talks to Claude. Anthropic’s docs say plainly that they don’t support routing it to non-Claude models. A subagent can pick a different Claude model, never a different provider. Some open-model vendors offer Anthropic-compatible endpoints, and people do point Claude Code at them. But that’s unsupported, and the whole session then runs on that one model: no Kimi for implementing next to DeepSeek for reviewing. And a backup I can’t rely on is no backup.

My second try was a small MCP server I wrote myself, delegate-mcp, that let Claude Code hand a task to an open model. It worked, and its sandbox passed all 19 escape tests I threw at it. But it missed the main point: Claude Code has to be running to delegate anything, so when my quota is gone, the delegation is gone too. And it was a 470-line agent loop: no way to steer a running agent, no session resume, no context compaction, no diff view. Fine for Claude firing off unattended jobs. No backup.

The mature alternatives were opencode and pi. opencode has native OpenRouter support and a nice terminal UI. I picked pi because it is smaller, and because almost everything in it is an extension:

• Any model per agent, and you can switch the model mid-session with /model.
• Providers are packages. Mine is TensorX, the EU host from my first post in this series.
• Hooks on every provider request, which I use to log exactly what gets sent upstream.
• You can interrupt and steer a running agent, it compacts near the context limit, and it has a headless mode for scripts.
• Subagents and scripted workflows. The workflow extension is a port of Claude Code’s workflow tool, and scripts written for Claude Code run unchanged.

And the part that made the decision: Claude stays reachable in both directions. The pi-claude-bridge package makes Claude a normal provider inside pi, billed to my Claude subscription. And the other way round, I wrapped pi’s agents as an MCP server, so Claude Code can hand a job to a pi agent.

The trade-off, to say it honestly: pi has no permission system, by design. It gives the model a normal shell. There is no “may I run this?” prompt. That’s why the controls from my first post aren’t optional here.

The setup

These are the packages I run on top of pi 0.87:

Agents by kind of work, each pinned to a model

The core of the harness is a folder of agent definitions. Each is a Markdown file with a model, a tool list, a turn budget and a short set of rules. I named them after the kind of work, not after the model:

GLM got the survey job by winning a duel: I gave it and MiniMax-M3 the same brief, and GLM found almost as many settings keys, invented none, and cost about an eighth as much. The details are in the last post.

Naming agents after the work has a nice side effect: when a better model shows up, I change one line in one file, and every workflow that uses implement gets it.

Every agent ends with the same four lines

Every agent has to end its run with a fixed head:

status: done | blocked | needs_input
branch: <its branch>
verify: <exact command> -> <exit code>
report_path: <file with the details>

The details go into a file, not into the reply. The orchestrator pays for every line an agent puts in its answer, on every later turn, so the answer is four lines and a path.

The turn budget is in the prompt

From the cost analysis I knew that cost is turns times context: every turn re-sends the whole transcript. So the agent prompts say that plainly. From implement:

You have a hard budget of 60 turns, and a turn is what costs money — the whole transcript is re-sent on every one, so ten cheap calls in one turn cost far less than ten turns of one call. Batch independent reads and greps into a single turn.

The survey agent gets a sharper version: grep with context in the same call, instead of grepping first and opening the file in a second turn. That habit alone turns one command into three per hit.

Escalate on facts, not on feelings

The escalation ladder is short: implement → implement-strong, survey → judgment, judgment → oracle if I have Claude quota left, and then me. The rule that matters: escalate only on a failed verify, never because the model says the task is hard, and only once. Models are very willing to declare something difficult. A red test is harder to argue with.

Design works without Claude

At first, the design agents ran on Claude Opus through the bridge. That defeats the point of a backup: when the quota is gone, there’s no design step. So judgment and Plan now run on DeepSeek V4 Pro, and only oracle, the rare second opinion, still goes to Claude. If it’s not available, that decision waits for me.

There’s also a second route, and it’s the one I actually used on the one real feature so far. I skipped the design agent and settled the design in a 20-minute chat with the orchestrator. That worked fine, with one miss: we assumed Serverpod merges a device identity into the new account automatically. It doesn’t, so the merge was never called anywhere. The review caught it. To be fair to the chat: when Claude Code wrote the original specification for this task, it made the same assumption. I don’t run a separate design agent in Claude Code either. The lesson isn’t “use a design agent”. It’s “check framework assumptions with a web search before they go into the design”, and that’s now a rule. The harness now offers both: a design document from judgment, or a design chat with me, after which the orchestrator writes the agreed design into a short file that I confirm. Either way, every later agent gets a design document to point at, not a chat log.

Rules files: AGENTS.md vs CLAUDE.md

pi reads the first of AGENTS.override.md, AGENTS.md and CLAUDE.md it finds in a folder. So if a project has an AGENTS.md, pi never sees CLAUDE.md. I use that on purpose: AGENTS.md holds the project truth that every agent needs, whatever the harness, and CLAUDE.md holds the procedures only Claude Code uses.

Skills port unchanged. pi implements the same skills standard, so a symlink from .pi/skills to .claude/skills is enough. It’s the one thing that moves between harnesses without any translation.

The traps

Each of these cost me time, and each is silent: nothing tells you it happened.

• The Explore agent that ships with the subagents package is pinned to Claude Haiku. It copies Claude Code’s own Explore agent, a fast read-only code search, and models trained on Claude Code like to reach for it. With the Claude bridge installed, every one of those searches quietly spends your Claude quota. An agent file with the same name wins, so I override it with my own agents/Explore.md on GLM-5.3-flash.
• Skills and agents vanish in headless mode unless the project is listed in pi’s trust.json. In the interactive UI you get asked; with pi -p they’re just dropped.
• An unquoted colon in an agent’s description: breaks the YAML, and pi skips the whole agent file. Set strictAgentFiles: true, then a broken file stops startup instead of disappearing.
• A typo in an agent type falls back to a general agent with all tools. Set fallbackSubagent: none so unknown types fail. Inside a workflow script that setting doesn’t apply, so spell carefully there.
• Don’t ask a model which tools it has. I did, to check my config. The answers were made up.
• Subagents hit their turn limit and have to be resumed. But once a finished subagent’s context is cleaned up, “resume” fails with “Agent not found”.
• Don’t start pi from a shell that has ANTHROPIC_BASE_URL set. The bridge passes the environment on to Claude Code, and every Claude turn goes to the wrong endpoint.

One fear that didn’t come true: I had read that thinking models lose their reasoning between turns when a harness replays the conversation. I logged the actual requests with pi-reqlog, and it didn’t happen.

What the first real run taught me about the harness

The run from the last post was the first real test: survey, implement server and app in parallel, review, fix. Here is what it said about the harness itself, rather than the models.

Parallel work in one worktree is fine with disjoint files. The server agent and the app agent worked in the same checkout at the same time, because they never touched the same files. That was my idea during the run, not the orchestrator’s.

The orchestrator needed steering. A small model running the session skips steps when nothing forces it. I had to ask why it wasn’t using a survey, and why it hadn’t run a judgment step before silently resolving a contradiction between an old task comment and newer acceptance criteria. And when it was unsure how Serverpod handles account creation, it guessed instead of searching the web. A web search then found the hooks it should have used. Three rules went into AGENTS.md that evening:

• Never silently resolve conflicting records. Ask.
• Design before build, through judgment.
• Never tinker. If you’re not sure, use the web search.

My biggest mistake: no test-first rule. The implement agent requires that its spec has tests or another machine check, runs them, and must “fix the cause, not the test”. Nothing tells it to write the tests first, from the acceptance criteria, before the code. So the same agent wrote the tests after the code, and they tested what the code did, not what the task required. 3 of its 9 new tests were red, and none of them covered the paths Claude Opus later found broken.

So I added a test-first step to the harness. Every slice now starts with a new agent, spec-tests, on GLM-5.3. It writes the tests from the acceptance criteria and the design, never from existing code, and commits them while they’re still red. It has to map every acceptance criterion to a test, and it’s told to test where things broke last time: first run, returning device, inside a transaction, the error path.

Then the rules around it:

• Three model families for three jobs. GLM writes the tests, Kimi writes the code, DeepSeek reviews. Nobody grades their own work.
• The tests are frozen. The implementer may add tests, but it may not edit, skip or delete the spec tests. If it thinks one is wrong, it has to stop and say why. It goes back to the test agent, or to me, never to the implementer.
• The orchestrator checks for itself. Before the implementer starts, it runs the tests and confirms they fail. Afterwards it runs them again, reads the exit code itself, and checks that the frozen test files are unchanged. Kimi’s report had claimed the analyzer passed, but it had measured the exit code of tail. A reported exit code is not evidence.
• The reviewer checks the tests against the requirements, not against the code. A criterion without a test that would fail without the change is a blocking finding.

I haven’t run a full feature through the new flow yet. When I have, I’ll report whether it caught what Opus caught last time.

It’s not only for coding

The cheapest session I had was also one of the most useful. GLM-5.3-flash walked me through 78 video folders for my app, four at a time in the question dialog. On my verdict it moved each folder to _ok or _redo, then built a table of the 22 videos that reference another video, so I know which ones I can reorder safely. Two hours and 13 minutes, $0.85.

For a task like that, a harness where the model can ask you structured questions matters more than the model.

What I take away

• If your work depends on one subscription, build a backup before you need it. A backup that runs inside Claude Code is no backup.
• If you want a different model per agent, and a different provider per model, you need a harness built for it. Officially, Claude Code only runs Claude.
• Name agents after the kind of work, not after the model. Swapping a model is then a one-line change.
• Put the cost model into the prompts: few turns, batched calls, four-line answers, details in a file.
• Escalate on a failed check, never on a model’s own opinion that the task is hard.
• Tests come first, from the requirements, and ideally not from the agent that writes the code.
• pi has no permission prompts. Everything that protects you has to live outside the model: no secrets in reach, no production credentials, a sandbox where it counts.

Next in the series: combining open models with frontier models, including what happened when Claude Opus reviewed the “done” that pi reported.