Keep the context small: what the model sees on each turn

Six months ago I asked Claude whether Don’t Spiral could run on a sliding window: only the last few messages of the conversation instead of the full transcript. Neither of us believed it would work. A therapy-style conversation without the history sounded like a companion with amnesia. Today the app runs on four messages, and this week I measured that on a small model the window does more than save money. This is the fourth post in Going beyond a simple chat, after the testbench, measuring models instead of guessing and an engine where the model is a setting. It is about what the model gets to see on each turn, and how we got from “that can’t work” to four.

The default hurts in long conversations

A chat app’s default is to send the whole conversation on every turn. For a support tool that looks like the only honest option: how can the agent respond well to turn 40 if it can’t see turn 3?

Don’t Spiral sessions are long, thirty to forty exchanges is normal. With a system prompt of several thousand words in front, the full transcript makes every turn more expensive and slower than the last, in a conversation that is supposed to calm someone down. What I wanted was a bounded context: a fixed cost per turn whatever the session length, and the model’s attention on what the user just said. Three things together got there, and they arrived in this order.

The prompt follows the session

Keeping the context small starts with the system prompt, not the transcript. Don’t Spiral handles ten kinds of session, from an acute spiral in the middle of the night to someone preparing a difficult conversation, and the agent has to behave quite differently in each. Someone spiralling needs a companion that mostly listens, stays close and slows things down. Someone reflecting on a recurring pattern needs one that challenges, asks the uncomfortable question and helps them see what they keep doing. Those are not the same prompt with a different tone. They are different rules about when to speak, when to push and when to hold back.

One prompt for all ten would carry, on every turn, nine sets of rules that don’t apply to this session. At best that costs tokens. At worst it confuses the agent: the challenging stance leaking into a night where the user needs to be held, or the holding stance into a session where the user asked to be pushed.

So each kind of session has its own prompt, with only what it needs, on top of a shared core. A session starts under an opening prompt whose job is to understand the situation. Once the server knows what kind of session this is, it replays the conversation so far under that session’s prompt. The user notices nothing. The model has just been handed a focused brief instead of a general one, and everything irrelevant has left its context. Because the first understanding can be wrong, and because people sometimes reveal the real issue late, the model can ask for one more switch per session.

Facts came first

The second piece was built for a different reason: memory across sessions. A companion that forgets the partner’s name between Tuesday and Thursday isn’t a companion. So the model extracts facts on every turn, inside the same structured call that carries its reply: the partner’s name, a pattern the user described, something they said they can’t do. What is known about this person goes into the next session’s context as a short list, instead of everything they have ever typed. A cheap model tidies the list at session end, and facts that stop coming up fade out.

Once that worked, it raised a question I hadn’t planned to ask. If the facts carry what matters about this person, how much of the current transcript does the model actually need?

Four messages and an anchor

I had started to worry about what a long session does to the model. The longer it ran, the dumber the model might get, the response times grew with every turn, and so did my token bill. That is when I asked Claude about the sliding window, and we tried it despite our doubts, because the bench made it cheap to find out.

We started cautiously, with a window of six or eight messages, I don’t remember which, and then tested how far down we could go. It held up as well as the full transcript all the way down to four. Two messages lost the thread too often. Four is where it has stayed, plus the first assistant message as an anchor so the conversation keeps its opening frame. Everything older is gone from the context, and the facts extracted from it stay. The saving grows with the session length, and in a product like Bedfit, which runs one endless conversation per user, the difference is a bounded context versus an unbounded one.

One side effect I didn’t expect: the window improved the facts. With the full transcript in view, the model treats “extract new facts” as “summarise what I know” and re-emits the same facts every turn, hundreds per session. With the window it extracts what’s new. A smaller context made the memory cleaner.

The one downside users felt

Windowed mode had one failure that reached users: the agent re-asking a question the user had already answered. It happens when a short answer leaves no fact behind and then falls out of the window, so nothing remembers it. A rule in the prompt and better handling of short answers fixed most of it. If you adopt a window, watch for re-asking. It is the symptom that the facts aren’t covering what the window dropped.

Does a small model need the window more?

The window was tuned on Sonnet. After the switch to Haiku I had a guess: a smaller model should lose the thread in a long context sooner, so the window should pay out more for it, and for similar small models. This week I verified that with the bench: two real sessions replayed in both modes on Haiku and on Sonnet, three paired runs each, every turn pair judged blind by Opus.

The guess was right, and by more than I expected. On Haiku the window doesn’t just save money, it fixes the model. With the full transcript in view, Haiku re-lists everything the user is dealing with instead of answering the last message, repeats itself, and rates distress higher and higher as the session goes on. The window removes all of that, and the feared recall loss didn’t show: the facts carried the state. On Sonnet there is nothing to fix. It handles the full transcript fine, and the window is a cost saving with no quality gain.

What held on both models: about a third of effective input cost saved, and response times that no longer grow with the session.

What I’d tell anyone starting out

Give each kind of conversation its own prompt and switch rarely. Let facts, not the transcript, be the memory, and once they carry the state, start wide and test how far down the window can go. Then measure it on your model. The same four messages are a cost saving on a big model and a fix on a small one.