Sweet PalAlmost all of the cost of an AI chat is decided before the model writes a single word. In a typical turn the model reads tens of thousands of tokens of instructions and writes a few dozen tokens of reply; if it works in several rounds, it rereads everything each time. So the way to lower the bill is to make it read less, read it once, and reuse from cache whatever doesn't change. The rest, picking the right model for each job, comes after.
We'll tell it with Sweet Pal's numbers. Sweet Pal is a companion for people living with type 1 diabetes, and the chat is the heart of the app. These are small, recent numbers, taken from our logs and our test bench: treat them as orders of magnitude, not as law.
Sweet Pal's chat prompt holds who Sweet Pal is, the safety rules, baseline clinical knowledge and the manual for 31 tools: log a meal, show a curve, read the diary, set a reminder. Every message from the person reaches the model wrapped in all of that. The reply, on the other hand, is short: in our measurements a voice reply writes a median of between 57 and 84 tokens. The ratio between read and written is nowhere near one to one, it's tens, sometimes hundreds, to one. Even though a written token costs more than a read one (on the main model we were using at the end of September, list price was 66 cents per million output tokens against 22 for input), the bulk sits on the input side.
So trimming reply length helps the person reading, but barely moves the bill. The bill moves when you cut what the model has to read, and the quickest way to make it read too much is to make it work in rounds. A chat that uses tools reads everything, decides to call a tool, gets the result, and to decide the next step it rereads everything from the start, plus the result. Our chat can do up to three tool rounds, plus a final writing-only round, and two or three rounds per turn were normal.
Measured on our test bench, the same message had the model read about 70,000 prompt tokens over the classic chat's two rounds. With the new architecture, where specialised layers decide first and the model writes once, with no tools, the voice reads about 2,800: roughly twenty-five times less, for the same reply. Not all of it comes from removing rounds. Part of it comes from the script: instead of receiving the entire persona and working out what to ignore, the model receives only the sections that turn uses. Over twelve fixed sentences, introducing the script took the voice's prompt tokens from 4,290 to 2,805 at the median, about a third less, and the complete reply from 2.73 to 1.61 seconds. Less to read also means an earlier first word.
What prompt caching changes
Model providers let you mark part of the prompt as a stable prefix: if the next turn starts with the same prefix, that part isn't processed from scratch, it's read from cache, at a fraction of the price. In our chat the prefix is the system prompt: persona, rules, knowledge, tools. It's identical for every turn of every person, so it's the perfect candidate.
Two details matter. The first is prompt order: caching works on the prefix, so everything that changes from one turn to the next, like the day's context or the person's memory, has to come after the fixed part. A timestamp at the top is enough to make the cache useless. The second is the key: some providers let you set one, and we use a single bucket for the whole of production, so every turn finds the prefix warm from the previous turn, whoever it belonged to. In our logs each call records input tokens, tokens read from cache and output tokens separately, because they have three separate prices. If the cache column stays empty, that's the first place to look.
Caching doesn't remove the problem of rounds, though: it makes rereading cheaper, it doesn't eliminate the rereading. And it does nothing for the time a model spends reasoning. Models that "reason" before answering write reasoning tokens, and those are billed as output tokens, the most expensive line. On our bench, a model set to no reasoning, which on its provider doesn't really switch off, spent 1,786 reasoning tokens and 16.7 seconds before its first word, for a reply of a few sentences. That's why the voice in our new architecture is flat by default: the layers before it have already made the decisions. It goes deep, with the reasoning level the subscription provides, only when it's really needed: the person asks to go deeper, the message is ambiguous, or there's a calculation on their data to explain. It's a quality choice before it's a cost choice, but the cost follows.
The same thinking applies to models. Not every job in an AI app is the same, and the app has three roles: main, the voice the person reads every turn; secondary, sixteen background jobs such as extracting memories, compacting the conversation, writing summaries and the daily analysis; vision, which reads meal photos. There will be two subscriptions, Expert and Pro. The first idea was one model for Expert and a stronger one for Pro, but it doesn't hold: taking the secondary role away from the first subscription would mean taking away its memory and summaries. So every role has a model for every subscription, chosen from the admin panel: Pro can get a stronger voice without Expert losing its background work, and the cost of each subscription is controlled role by role.
What does a subscriber really cost?
To find out we read the call logs, person by person: uncached tokens, cached tokens, output tokens, and cost. The cost is divided by the number of days the person was present in the window, never fewer than seven, so someone who arrived yesterday doesn't count their first day seven times over. People are sorted by usage and split into five bands.
With data from the end of September, very few people and a ten-day window, repricing everything at the main model's list price, the spread between bands is what stands out: the heaviest cost about five times the median. These are numbers from an app still in beta and they will change, but they're enough to know where to look when reasoning about margins: not the average person, but the top bands.
Those numbers are clean only because we removed the dirty part. Between late August and mid-September an automated test user had burned through around a hundred million tokens, and expensive models were being tried. With those days included, the average rose almost twentyfold, past the price of a subscription. Before you do any maths on cost per user, separate the people building and testing from the people using the app. And two last things make the calculation realistic: model cost is paid for every active subscriber and for everyone in the trial period, because they use the app too; and from a subscription's price, before you reach the margin, you take off VAT and the store's commission, so model cost has to be compared with what's left, not with the list price.
In short:
- In an AI chat, cost lives in the tokens read, not the ones written: cut it by having the model read less.
- Every tool round rereads the whole prompt: two rounds can mean reading twenty-five times the tokens of a single round with a focused prompt.
- Prompt caching only works if the fixed part sits at the top and never changes; your logs should say how much really comes from cache.
- Reasoning is billed as writing: switch it on when it's needed, not always.
- One model per role and per subscription, and cost per subscriber measured from logs, excluding test accounts.