Different models find different bugs: combining open models with Claude

This is the fourth post in my series on open models next to Claude. In the last one I described the harness I built as a backup for when my Claude subscription runs out. This one is about the other way to use it: not instead of Claude, but together with it.

It comes down to one observation from a week of real work: different models find different bugs. Not “the expensive one finds everything”. Different ones.

“Done” is not done

The story starts where my post on quality and cost left off. Kimi K2.7 Code had implemented a feature across my Flutter app and my Serverpod backend: anonymous device identities that merge into a real account on signup, plus a 30-day purge of identities that never signed up.

The harness had its own review built in. DeepSeek V4 Pro reviewed Kimi’s work and found three real blockers: the app never re-registered the user with the payment provider after login, the account merge was never called anywhere, and a new guard broke two tests. Kimi fixed them, and the orchestrator reported “done”.

Then I had Claude Code look at it. First my Claude session noticed that the server didn’t even pass the analyzer. Kimi’s report had claimed it did, but it had checked the exit code of tail, not of the analyzer. Then an independent Claude Opus review rejected the work with 7 blocking and 4 high findings:

The signup merge could never work. The account-creation hooks received a database transaction and threw it away. The merge ran in a second transaction that couldn’t see the new account.
Every fresh install destroyed its own new identity. A guard against stale sessions signed the brand-new anonymous identity straight out again, so the first session, the one the whole feature was for, recorded nothing.
From the second launch on, a device that never signed up skipped onboarding and the paywall. An anonymous identity counted as “logged in”.
3 of 9 new tests were red.
Anonymous identities could reach things meant for accounts: an AI endpoint and the paid audio library.
The purge deleted real purchase records.

None of these were in DeepSeek’s review. All of them sat at integration points: transactions, auth state, app lifecycle. A Claude fix round fixed everything, all tests green, and it was merged at ten past one in the morning. Most of the code was still Kimi’s: 95% of its lines went into the merge unchanged.

And then the other way round

If the story ended there, the lesson would be “open models write, Claude checks”. It doesn’t end there.

The next feature was account deletion. This time Opus reviewed first and found 3 blocking issues: the change broke the admin app’s build, and two tables still kept the deleted user’s device hash and IP address. They were fixed. Then DeepSeek did the follow-up reviews, five rounds of them. And it kept finding real leaks that the Opus review hadn’t flagged. The admin’s delete path never removed the user’s chat transcripts. Stored payment webhooks still carried the deleted account’s raw ID. A deleted payer’s ID stayed on inactive seats they had sponsored.

On a shared package, it was the reverse again. DeepSeek reviewed a change to how my account-deletion code cleans up across tables: tests green, analyzer clean, no blocking issues. After a small fix round, I asked Opus for a second opinion on top. It found one blocking issue DeepSeek had missed: when a user in a sponsored seat was deleted, the seat was never released, so the payer could never give it to anyone else.

And Opus wasn’t flawless either. It misreported one analyzer result and overstated what one endpoint did.

So the lesson isn’t “Opus good, open models bad”. Each model has blind spots, and they don’t overlap much. Two cheap reviews from different model families plus one strong review make a better net than any single reviewer, however good.

I use the same idea outside code: the LLM that grades my app’s AI coach runs on a different vendor than the coach itself, so both can’t fail in the same way.

Moving work off the Claude quota

The second way to combine models is plainly economic. The night I merged that feature, my Claude session hit its weekly limit. Most of what an agent does isn’t judgment, it’s reading. That doesn’t need my most expensive quota.

Surveys. Mapping which files a change touches cost $0.04 to $0.17 on open models. The result is a report file. The expensive model reads the report instead of searching the code itself.
Routine reviews. An Opus review of that feature took about 217,000 tokens and ten minutes of subscription time. A DeepSeek review costs one to two dollars. So DeepSeek is now the frequent reviewer that runs on every piece of work, and Opus is the rare second opinion before something hard to reverse. Its value is that it comes rarely, with fresh eyes.
Tests and implementation. In my harness, GLM writes the failing tests from the requirements, and Kimi implements against them. Claude only comes in for the rare second opinion and when I want a final look.

The rule that makes this work: hand over files, not content. Every brief names the survey report, the design document and the prepared diff by path. If the expensive model has to re-read everything, the saving is gone. And one rule on turns: the cost of an agent is its number of turns times its context, so cheap models get briefs that let them finish in few turns.

How Claude Code calls the other models

Either side can lead. When I work in pi, Claude is just another provider through the pi-claude-bridge package, so an open-model session can ask Opus one question and go on without it. But most days I still work in Claude Code, so the more interesting direction is the other one: how does Claude hand a job to Kimi or DeepSeek?

One small script

pi can run without its interactive UI: pi --print takes a prompt, runs the agent loop to the end and prints the result. What it can’t do is load one of my agent definitions from the command line. The model, thinking level and tool list in an agent file only apply when pi starts a subagent from inside its own session.

So I wrote a small wrapper, pi-agent. It reads the agent file and builds the pi --print call from it, so the agent file is the only place where “review runs on DeepSeek V4 Pro” is written down. It also removes any environment variables that point Claude tools at a different endpoint (one of the traps from the last post), and gives every run its own folder with the brief, the output and the exit code.

From Claude Code, a delegation then looks like this:

pi-agent run review --prompt-file brief.md --cwd ~/dev/apps/bedfit --fg

Claude writes the brief into a file first: what to review, the path to the prepared diff, the acceptance criteria. The details go in files, not in the command.

Background, no polling

An agent runs between two and ten minutes. Claude Code starts the command in the background and is woken up when it ends. In the meantime it can go on with other work, or just wait without spending anything. At the end, the script prints only the short head every agent has to end with. For a review, that’s three lines, for example:

status: done
report_path: /tmp/review-170.md
issues: 2

Claude reads those lines, not the transcript, and only opens the report if the status says blocked or it needs the details. That keeps the expensive context small, which is the whole point of delegating.

Why not an MCP server?

I also wrapped the script as an MCP server with three tools: start a run, check its status, list the agents. It still exists, and it has advantages: Claude sees the tools on its own without any instructions, and it works in clients that have no shell.

But it has one real cost. The MCP call returns immediately with a run ID, and then Claude has to ask “done yet?” again and again. Every one of those checks is a turn, and turns are what cost tokens. A background shell command needs no polling at all. So in Claude Code, the rule in my global CLAUDE.md is now one line: call pi agents with pi-agent run … --fg in the background, never bare pi. The MCP server stays as the fallback.

Two honest limits: a turn budget in an agent file isn’t enforced in this mode, because pi has no command-line flag for it, so a runaway agent runs until it’s done. And an agent pinned to Claude through the bridge saves no Claude tokens at all. It just takes a detour.

One honest wrinkle: since my last post, DeepSeek also does the design work in the harness, so it works while my Claude quota is gone. That means on a design question, designer and reviewer are now the same model family. For the code it doesn’t matter: GLM writes the tests, Kimi the code, and DeepSeek reviews both. But for designs that are expensive to reverse, that’s exactly where the Opus second opinion earns its place.

Rules of thumb

Cheap models for breadth: surveys, first reviews, test drafts. Strong models for judgment: design, the final review before something hard to reverse. Humans for decisions.
Never merge on one model’s “done”. Not an open model’s, not Claude’s.
Review with a different model family than the one that wrote the code, and on anything that matters, with two.
Check the claims yourself. A reported exit code is not evidence.
Hand over artefacts by path, and keep the number of turns low. That, not the price per token, decides the bill.