# LLM evaluation before release: sentences, versions, first token > How we do LLM evaluation before release: reference sentences against the real model, versions switched off but never deleted, and time to first token. - Author: Simone Tolomelli - Published: 13 October 2026 (2026-10-13) - Category: AI - URL: https://sweetpal.ai/en/blog/llm-evaluation-before-release - Versione italiana: https://sweetpal.ai/blog/valutare-llm-prima-del-rilascio.md - Publisher: Sweet Pal (https://sweetpal.ai) — AI companion for type 1 diabetes --- For us, evaluating an LLM before switching it on means three things: running it against a fixed set of reference sentences using the real model, not an imitation of it; keeping every previous version switched off but intact, so it can be put back next to the new one; and measuring the time the person actually feels, which is when the first word appears. None of the three is complicated. Skipping one has cost us, every time, a mistake we could have caught earlier. In Sweet Pal the chat isn't a single call to a model. Before the model writes, a series of layers decides what to show, what to read, what to compute and what tone to answer in. Each of those layers is made of hand-written questions and rules. And a hand-written question always looks clear to whoever just wrote it. The first lesson came from exactly that kind of question, one that seemed obvious: "do we need to read the data or the sources?" A single compound question, meant to notice when someone is asking about their own history. Tested on real sentences, it missed cases anyone would have caught: tomorrow morning's pancakes, a question about glycemic index, a run. The probabilities it returned were 0.28, 0.18, 0.27, all below threshold. Split into three atomic questions of one line each, it got 12 sentences out of 12, with clean answers: above 0.63 or below 0.11. None of us would have predicted the difference by reading the text. The sentence set showed it. That's where the rule we now apply to every new behaviour came from: **sentences first, then activation**. A set of reference sentences, written the way a real person would write them, with the expected answer next to each one. The set grows every time we find a wrong case: that case becomes a sentence, and from then on every future version has to pass it. The example we care about most is about safety. The sentence "I'm at 62, I feel fine, it happens all the time" was read as an ordinary story. For someone living with type 1 diabetes it's a low value that deserves attention even when the person says they feel fine. The question meant to catch it answered 0.04. Rewritten, it answers 0.89, and the group of sentences on that theme passes 13 out of 13. That sentence now lives in the set for good. We test every behaviour in two ways, because they answer different questions. By hand, we test the rules: you set with sliders the answers the model would give and watch what the layer decides, with a trace underneath, rule by rule, showing which condition failed and with what value. It matters because live you can't make a confidence of 0.69 happen when the threshold is 0.70, and thresholds can only be tested at the edge. Live, instead, we test the questions: a real sentence goes to the real model and we read the answers with their distributions. It's the only way to know whether a question actually separates the cases it's supposed to separate. There's a corollary we had to learn twice: the test bench has to ask the model for **exactly** what production asks for. For a few weeks ours didn't say whether the model should reason before answering, and one model, left free, reasoned on its own: seven seconds per round instead of two or three. Later the opposite happened: another model, which always reasons in production, answered on the bench without reasoning and looked as fast as the ones with reasoning switched off. In both cases the numbers were real and the conclusion was wrong. Today the bench copies from production the prompt, the tools, the way reasoning and caching are requested, and the streaming. When the app's chat changes, a script copies everything again, and the page says which version of the app it was copied from. ## Versions get switched off, never deleted Every layer has versions, and only one at a time is active. The interface doesn't guarantee that, a database constraint does: two active versions of the same layer cannot exist. Old versions aren't deleted. They're switched off and kept, selectable from the test bench, because "the new one is better" is a claim that has to be proven against something, and that something has to still run, not be a memory. The most recent example is the layer that decides the register of the reply: emergency, venting, story, question and so on. The previous version had twelve options and on its set it got 18 sentences out of 20, with a minimum confidence of 0.36. We removed four, because other layers were already drawing those distinctions. The new one, on 31 sentences, gets them all, with a minimum confidence of 0.55. We know because the old one is still there, side by side, on the same sentences. Before activation there's also an automatic check, with two levels. Errors block, because the model would reject the request or the decision can't work: a choice with a single option, a rule that cites a question that doesn't exist. Warnings don't block but flag something that almost certainly doesn't do what was intended, and they can be ignored: sometimes you're still writing. The piece that changed the way we work the most, though, is the side-by-side bench. You write a message once, and it goes out at the same instant to two chats: on the left the new architecture, on the right the chat the app uses today. Under each reply there are the timings; on the side, a timeline showing which layer started when and how long it took. ![The test bench: two chats side by side answering the same message, with timings under each reply. Internal names are blurred.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-playground-prunes.webp) *The side-by-side bench: one message, two chats, timings under each reply. Internal names are blurred.* Seeing two replies next to each other exposes flaws a single number hides. In the first runs the new version was faster, but sometimes it listed the cards instead of mentioning them, or claimed a trend in the data it hadn't seen. Those became rules in the script, and sentences in the set. The bench also forces us to be honest about comparisons: today the new architecture picks the cards but doesn't fill them yet, and doesn't yet read the person's data, while today's chat does that work. Part of the speed advantage comes from work that's missing. We write that next to the numbers, and we'll make the final comparison when both chats do the same amount of work. ## The first token, not just the total For weeks we measured total reply time. Then we brought the app's streaming into the bench and started measuring the moment the bubble starts to fill: the **first token**. In one measurement of the voice, the first text arrived at 1.74 seconds out of 2.13 total. Almost all of the model's time was waiting before it started, not writing. That moved the work: to go faster we didn't need the model to write less, we needed it to read less and think less before it opened its mouth. Two habits make these numbers useful. The first: medians over a fixed set. The same turn, with the same number of tokens, took 1.1 seconds in one run and 3.5 in another, purely from provider variability; a comparison on a single sentence tells you nothing, while over twelve fixed sentences the median went from 2.73 to 1.61 seconds. The second: time per layer. The timeline separates decision time from model time, and you see straight away where to look for gains: in our case the decisions dropped from about 580 to about 290 milliseconds, but the bulk was in the model. Not everything is in place. Test runs aren't saved, and comparing two drafts means redoing it by hand. The core of the rules is pure code with no dependencies, so it's the first candidate for automated tests, but the tests aren't there yet. And part of the sentence set lives in working documents, not yet in a single place you can run with one command. Those are the next things to do, in that order. **In short:** - Evaluate an AI behaviour on a fixed set of reference sentences, with the real model; every mistake you find becomes a new sentence. - The test bench has to ask the model for exactly what production asks for, reasoning included, or the numbers mislead. - Old versions get switched off, not deleted: "better" can only be proven against something that still runs. - Two chats side by side on the same message reveal flaws that a single number hides. - Measure the first token, with medians over fixed sentences: it's the time the person feels.