How good and how expensive are open models? What I know after a week
Kimi wrote most of a real feature for about €20. Is that cheap? It depends less on the model than on the host, on cache prices and on how many turns your agents take. My measurements, next to Claude at list prices.
In my last post I wrote about running Chinese open models without sending your code to China. The obvious next question: are they any good, and are they actually cheaper than Claude?
Short answer: good enough to surprise me, and cheaper only if you look at the right price. That price turned out not to be the one on the front of the pricing page.
Why this is hard to answer
To really know how good a model is on your work, you have to run the same task on several models. One task isn’t a benchmark. Public benchmarks don’t help much either: none of them covers Dart or Flutter, which is what I write, and many of the open-model numbers are run by the vendors themselves.
So everything below is a handful of measurements from my own work over one week, not a ranking. Take it as field notes.
What the open models did well
Surveying a codebase. I gave two cheap models the same brief: find every settings key the backend reads. GLM-5.3-flash found 52 keys, MiniMax-M3 found 56, neither invented a single one, and GLM’s result was a strict subset of MiniMax’s. GLM cost $0.04 and needed 26 tool calls, MiniMax $0.33 and 72 calls. A regex script Claude wrote afterwards found 39 keys and invented 3. Both models beat it.
Judging German text. My app’s coaching replies are checked by an LLM judge. On 52 hand-labelled replies, Qwen 3.5 122B got 50 right, exactly as many as Claude Haiku 4.5, with the same two misses. The small Qwen 3.5 9B got 48, but it waved through a real defect, which disqualifies it. My old keyword-based grader managed 32 of 49.
One trap on the way: the open models first looked 60 times slower than Haiku and failed on a 400-token limit. Thinking was on by default, and the reasoning tokens count against max_tokens. With reasoning_effort: none both problems disappeared, without losing accuracy.
Implementing a real feature. The big one. I let Kimi K2.7 Code implement a feature across my Flutter app and my Serverpod backend: anonymous device identities that merge into an account on signup. Two Kimi agents worked in parallel, one on the server, one on the app, while a DeepSeek model reviewed their work. I skipped the usual design stage with Claude. Instead, GLM-5.3-flash surveyed the relevant code, and I settled the design in a 20-minute chat with Qwen 3.8 27B, the model running the whole session.
The structure Kimi built held up. Of the roughly 1,690 lines it added, 95% are in the final merged commit unchanged. Three quarters of the merged production code is Kimi’s.
While the agents were still working, I asked Claude in a side question what it thought of the code so far. It answered, clearly surprised, that the code was really well organized. Coming from the model I’d have trusted to write it, that was the moment I started taking open models seriously.
What they didn’t do well
The same feature is also the counterexample. When Kimi and DeepSeek reported “done”, I had Claude Opus review the result, and it found 7 blocking issues. They all sat at integration points: a transaction that wasn’t passed on, a fresh install that destroyed its own new identity, a data purge that deleted purchase records. Kimi’s report also claimed the code analyzed cleanly, but it had measured the exit code of tail, not of the analyzer.
And the tests were weak. 3 of Kimi’s 9 new tests were red, and none of them covered the paths Opus found broken. Only 39% of the tests in the merged commit are Kimi’s. Part of that is on me: my harness had no rule that tests come first, written from the requirements before the code. More on that in a later post.
One more observation, and it’s not about code quality: these models don’t hesitate. In my runs, they never stopped to ask before doing something risky. With Claude Code I’m used to a safety net that has two layers. In auto mode, a separate check judges risky actions before they run. And even without that, Claude typically stops and asks before it does something risky. Not always, but often enough that I had come to rely on it. My harness, pi, has no permission prompts by design, so nothing stood in the way.
If you switch, you lose that judgment, and you have to replace it with hard limits: no production credentials anywhere the agent can read them, and where it needs a database at all, a staging copy or a read-only database user. How I keep passwords out of files and chats is in a password vault for AI agents. Keep in mind that this protects the password, not the database: an agent that is allowed to run the injecting command can still use it.
So my honest summary: good code, not yet correct code. Better than I expected, and not something I’d merge on its own word. That’s true for any model, and it’s why I review with a different model than the one that wrote the code.
What it cost
The whole evening cost $23.53, about €20:
| Stage | Model | Cost |
|---|---|---|
| Orchestration and design chat | Qwen 3.8 27B | $2.10 |
| Two codebase surveys | GLM-5.3-flash | $0.10 |
| Implementation plus one fix round | Kimi K2.7 Code | $17.77 |
| Three reviews | DeepSeek V4 Pro | $3.56 |
| Design for the task I started with | Claude Opus 5 | subscription |
| Total | $23.53 |
That’s 726 model turns. Now look at the tokens: 70.3 million cache-read tokens against 0.41 million output tokens. The models barely wrote anything, relatively speaking. What they did was read their own growing context again on every turn.
That’s the cost law of agentic work: cost is turns times context. The implementation agents cost about 15 times as much as the survey agents, on a model only 2.5 times as expensive per token. The difference was almost entirely the number of turns.
The price that matters is the cache read
Every provider caches the part of your prompt that didn’t change since the last turn, and bills it at a discount. In an agent loop almost the whole context is cached, so the cache-read price decides your bill. The input and output prices on the front of the pricing page barely matter.
And cache-read prices vary a lot, even for the same model. Here is my evening’s token mix, 70.3M cache reads and 0.41M output, priced as if it had all run on one model, at list prices as of September 2026:
| Model | Host | Same tokens would cost |
|---|---|---|
| DeepSeek V4 Pro | DeepSeek’s own API, off-peak (China) | ~$2.40 |
| Kimi K2.7 Code | DeepInfra (US) | ~$11.00 |
| GLM-5.3 | Mistral | ~$11.70 |
| Kimi K2.7 Code | Moonshot’s own API (Singapore) | ~$15.00 |
| Kimi K2.7 Code | Inceptron (Sweden) | ~$15.50 |
| Claude Sonnet 5 | Anthropic | ~$18.20 |
| Claude Opus 5.5 | Anthropic | ~$22.30 |
| Kimi K2.7 Code | TensorX (EU, my host) | ~$23.60 |
| GLM-5.3 | TensorX | ~$32.80 |
| DeepSeek V4 Pro | TensorX | ~$36.80 |
| Claude Opus 5 | Anthropic | ~$45.40 |
This table assumes equal tokens, which is a simplification: a different model would take a different number of turns and read a different amount of context. But it shows the point clearly.
At my host, Kimi costs about as much as Claude Opus 5.5 for the same tokens, and more than Sonnet 5. TensorX bills cache reads at 25% of the input price, Anthropic at 10% (Opus 5.5 even at 5%). The same Kimi model on DeepInfra or Inceptron would have cost half to two thirds as much. And DeepSeek’s own API is about five times cheaper than the next option, which is exactly the trade-off from my last post: that price comes with your data stored in China.
So “open models are cheap” is only half true. The host decides the bill more than the model does. I’m looking at switching hosts, and a later post in this series will compare them properly.
Why Claude still melts like snow in the sun
If Sonnet costs about the same for equal tokens, why does paid extra usage on my Claude subscription disappear so fast? Because a Claude Code session isn’t at equal tokens.
- Claude Code defaults to Opus, not Sonnet.
- Cache writes. Writing to Anthropic’s cache costs 1.25 times the input price for a 5-minute cache and 2 times for a 1-hour cache, and most of my cache writes are the 1-hour kind. In the run that ate my usage limit, one lead session wrote 21.8 million tokens to the cache. At Opus 5.5 prices that’s about $109 for the writes alone at 1.25x, and about $174 at 2x.
- The agents come on top, each with its own context.
Reads or writes: why it’s different
So with open models the cache reads cost me, and with Claude the cache writes do. That’s not because I worked differently. Every token that’s new in the context has to be processed once, and from then on it’s read from the cache on every turn. The ratio was about the same on both sides: in my open-model evening about 3 million new tokens against 70 million reads, on a normal Claude Code day now about 11 million writes against 300 million reads.
The difference is the price of each kind of token:
| New token (write) | Cached token (read) | One write costs as much as | |
|---|---|---|---|
| Anthropic | 1.25x or 2x input | 0.1x input (Opus 5.5: 0.05x) | 12 to 40 reads |
| TensorX | 1x input, no surcharge | 0.25x input | 4 reads |
At Anthropic, reads are very cheap and writes carry a surcharge, so 11 million writes cost roughly as much as 300 million reads. At my open-model host, reads are relatively expensive and writes cost nothing extra, so the sheer volume of reads decides the bill.
The surcharge has a reason. Anthropic’s cache is explicit and guaranteed: you pay for your prefix to be kept for 5 minutes or an hour. Most open-model hosts cache automatically and on a best-effort basis. If the cache is gone, or your request lands on a different server, your text is simply processed again as normal input.
Claude Code also produces some extra writes on top: every subagent starts with a fresh context of its own, and a compacted or expired context gets written again from the start.
The subscription is still the cheapest option, while it works
To keep this fair: I measured my own Claude Code usage over the last weeks. Priced at API list prices, my days range from about $115 to over $1,300. My subscription costs a small fraction of that. As long as the subscription lets me work, no pay-per-token option comes close, open or not.
After my context rules, my usage dropped clearly while I worked about as much as before. Cache writes per message fell by 40 to 70% on every model, the most on Fable 5.1 (-72%), the most expensive one.
So open models aren’t a replacement for my subscription. They’re what I reach for when:
- the subscription limit runs out in the middle of the week, or the limit accounting suddenly misbehaves, as it did for me last week;
- the task is breadth, not judgment. A codebase survey for $0.04 is cheaper than any subscription quota. I also had GLM-5.3-flash walk me through 78 video folders for two hours, sorting them on my verdict. Total: $0.85;
- I want a second opinion from a different model family. Different models find different bugs. That gets its own post.
What I take away
- Open models are better than I expected: good structure, good surveys, a judge as accurate as Haiku. But their “done” still needs a review by a different model.
- One task isn’t a benchmark. If you want to know, run the same task on several models.
- For agent work, the cache-read price decides the bill, not the input or output price.
- The same open model can cost two or three times as much depending on the host. Compare hosts before you compare models.
- While your subscription works, it’s the cheapest option. Open models are the backup and the second opinion.