Book a call

AIEngineering

Giving your agent tools: chat as the control surface

UGC Scout drives most of its UI through tool calls: 21 tools, a loop capped at ten rounds, and a pane that shows what the chat decided. How a tool call actually works, what a dead tool, a missing tool and a dropped connection taught me, why the descriptions do the prompt's job, and what it costs: one heavy account, 293 calls, $2.46 in a week.

Fifth post in the series Going beyond a simple chat. The first four were about a companion that talks. This one is about an agent that does things.

Imagine your agent could do more than write a reply. The user says “find me creators for this product”, and the agent starts a search. “Show me the ones with over 50k followers”, and the list on the screen changes. “Create a campaign with these five”, and a form appears, filled in, waiting for a click. That is UGC Scout, the third of my three apps, where the chat is the control surface of the whole product. The mechanism that makes it possible is tool calling, and it is simpler than it sounds.

How a tool call works

A model can’t run code, call an API or touch a database. What it can do is ask for it. You send the model, next to the prompt, a list of tools: each one a name, a description in plain language, and a JSON schema for the arguments. When the model decides a tool would help, it answers not with text but with a structured request: the tool’s name and the arguments, filled in according to the schema. Your server runs the tool, sends the result back as the next message, and the model continues. It can ask for the next tool, or write its reply.

The tool-call round trip: the user sends a message, the model answers with a tool call, your server runs the tool and returns the result, the model continues until it writes a reply. User Model Your server runs the tool message reply tool call launch_search({ angle: … }) result as JSON the model repeats this until it answers with text instead of a tool call

Two things about this are easy to miss. The model never executes anything; every tool call passes through your code, so you decide what runs, with what checks. And the model picks a tool based on the description you wrote. The description is the only thing it knows about the tool, which is why the descriptions carry most of the behaviour, as you’ll see further down.

In Don’t Spiral and Bedfit I use this for exactly one thing: the reply itself is a forced tool call, so that the message and its metadata come back as a validated structure (the engine post). UGC Scout is the app where the agent gets real tools, 21 of them, and that is a different kind of agent to build.

Why a chat and not a dashboard

UGC Scout finds creators for brands that want user-generated content. The pitch: a UGC agency costs $10,000, UGC Scout costs $49. A brand describes a product or pastes a URL, the agent proposes keyword angles, launches creator-discovery searches, shows the results, digs deeper into the promising ones, finds contact emails, drafts outreach and tracks the campaign.

Every one of those steps could be a form on a dashboard, and I thought about building it that way. Three things made me go with chat:

  • The model writes better search keywords than users do. The quality of a creator search depends on the angles you search for, and a brand owner typing into a keyword box produces worse angles than the model does from the product description. In a chat the model proposes and the user picks.
  • A long search is a broken spinner in a dashboard and a conversation in chat. A discovery search runs between two and thirty minutes on an external API. On a dashboard that’s a progress bar nobody trusts. In a chat it’s “I’ve started the search, meanwhile let’s talk about the second angle”.
  • Refinement is conversational. “More like this one, fewer of those” is a sentence, not a filter UI. And more than that: the user can say what they want in vague terms, with conditions nobody predefined. “Creators who look like they’d actually use this, not the ones who promote anything” is not a filter I could have built a checkbox for, and the agent can act on it.

But not everything is chat. A list of two hundred creators doesn’t belong in a chat bubble. So the app has a rule that shaped everything after it: verbs belong in the chat, nouns belong in the pane. The chat is where the user says what to do. The right pane is where the things are: creator cards, forms, campaign lists. The agent drives the pane through tool calls, and the pane never tells the agent what to do.

Three kinds of tools

Sorting the 21 tools into kinds was the single most useful design step, because each kind has its own rules:

  • Read tools fetch from our database. They cost nothing, so the model can call them freely, and the description says so.
  • UI tools only change the pane: render a product form, show a creator profile, apply a filter. They mutate nothing.
  • Act tools change state or spend money: create a product, launch a search, find emails, look something up on the web.

The rule for act tools is show, then confirm. The agent never changes state on its own. It renders a form in the pane with the values it extracted, the user clicks confirm or says “looks good”, and only then does the act tool run. Every state change has a moment where the user saw what was about to happen. That is not only a safety measure; it is how the user learns what the agent understood, and it catches the mistakes before they become database rows.

Not every context gets every tool. A user in onboarding, who has no product yet, gets three: import a product from a URL, render the product form, create the product. Fewer tools means fewer wrong choices, and the onboarding conversation has one goal.

The loop needs a cap and a wrap-up

One user turn can take several rounds through the diagram above, and the loop needs a limit, because a model that cannot finish a task does not stop trying. Mine allows ten rounds per turn. The first live test on production hit that limit on two turns in a row, and both died mid-flow: the streamed text ended with “Let me try each group separately:” and nothing came after it. No answer, no suggestion chips. To the user it looked like a dropped connection.

The fix is one extra call after the tenth round, with tools disabled: the server tells the model the budget is spent and asks it to write its final reply, say what it couldn’t finish and why, and offer next steps. A capped turn now ends with an honest answer instead of silence.

Tool errors go back to the model as data, with instructions in them. When a campaign id no longer exists, the result says so and adds “ask the user which campaign to use, never retry the same id”. Without that sentence the model retries, because from where it sits, retrying is the reasonable thing to do.

What the first weeks taught me

The agent went live in late July, and the backlog from the first two weeks reads like a list of tool-design rules. Three of them:

A dead tool misleads the model. One tool was registered and described as “check hashtag performance before launching”, because that step was planned. Its handler was a no-op. The model called it, and then told the user about a validation step that had never happened. The description said the tool exists, so the model used it. If a tool isn’t real yet, it isn’t in the list.

A missing tool burns the whole budget. A user asked for an analysis of the twenty creators they had deep-dived. No tool could fetch full data for a list of named creators. The model spent all ten rounds on workarounds and died at the cap with a table full of dashes. It was doing its best with the tools it had, and its best was ten expensive rounds of nothing. One new tool, a bulk fetch by handle, and the same request takes one call.

A dropped connection runs the tool twice. On a day with network trouble, one account created four identical products in seven minutes. The stream had died after the create tool ran on the server but before the tool call was saved to the conversation. The next turn held no record that the product existed, so the model, correctly from its point of view, created it again. Two fixes, and I’d do both again: the server logs sensitive tool calls before it sends the result, and surfaces that log on the next turn if the stream died. And the create tool checks for the same name on the same account within ten minutes and returns the existing product. Assume the connection drops at the worst moment, and make the expensive tools safe to run twice.

The descriptions are the prompt

In an agent with twenty tools, the tool descriptions carry more of the behaviour than the system prompt does. The prompt says what the product is and how the conversation should go. Which tool gets called, in what order, with how much caution, is decided by the descriptions. Mine carry four things:

  • Cost. “Free, reads from our database” versus “costs credits”. The model treats the two differently once it knows which is which.
  • Order. The cheap semantic search over the creator pool says “prefer this first” over a fresh discovery search.
  • Preconditions. The web lookup says to use it only after the cheap email lookup came up empty.
  • When not to call. “Add them to the campaign” once created a new campaign. The create tool’s description now says never to call it for that request when campaigns already exist.

And a confirmation threshold: before running the web lookup for more than about five creators in one go, confirm with the user. Without that rule, “find emails for everyone on the list” spends credits silently.

A tool description is a prompt, and it gets edited like one: after a live incident, with a test that checks the model now does the right thing.

Test the payload

The testbench post was about replaying conversations and checking the text of the reply. Once the reply is a tool call, the bench can check the payload, and that is a far better test than reading prose. Did the model call the search? Did it set the crisis flag? Which facts did it extract this turn? None of that needs a judge model; the value is either there or it isn’t.

That is how the fact extraction from the last post got good: a set of scenarios prints every extracted fact as a line, and I could watch the counts and the share of lasting facts move across prompt revisions. It is also how I caught the user’s own name being filed as a fact about another person on 13 of 30 facts, which no amount of prompt wording fixed and a check on the server did.

UGC Scout has no scenario bench yet. What it has is the production log: every request records its rounds, tokens, cost and duration, and every tool call is a line I can read. For an agent whose output is actions, that log is the bench.

What it costs

First production week, the heaviest account: 69 sessions, 293 model calls, $2.46. The whole AI bill for the app that week was about $2.74.

The reason is in one number: 93% of chat input tokens are cache reads. Twenty tool definitions are a lot of tokens, and they go out on every round of every turn. With the cache they cost a tenth. Even the wrap-up call keeps the tools in the request and disables them through tool_choice instead, because dropping them would invalidate the cached prefix on the largest request of the turn.

What I’d tell anyone starting out

Sort your tools into read, show and act before you write the first one, and give act tools a show-then-confirm step. Put cost, order and “when not to” into the descriptions, because that’s where the model reads them. Cap the loop and end a capped turn with a wrap-up, not silence. Make the expensive tools safe to run twice. And once your replies are tool calls, test the payload; it’s the cheapest test you’ll ever write for a model.

Next in the series: Bedfit answers questions about a course, and a model that doesn’t know the answer makes one up. How a knowledge base and a forced “how well does this fit” field stopped that.