# Decide first, speak later: how J2LM made our chat fast

> We took the deciding away from the LLM: JOLs classify in parallel, a script writes the brief, the model only speaks. Sweet Pal now replies in about 1.5 s.

- Author: Simone Tolomelli
- Published: 7 October 2026 (2026-10-07)
- Category: AI
- URL: https://sweetpal.ai/en/blog/j2lm-decide-first-speak-later
- Versione italiana: https://sweetpal.ai/blog/j2lm-prima-decidere-poi-parlare.md
- Publisher: Sweet Pal (https://sweetpal.ai) — AI companion for type 1 diabetes

---
How we took away from the LLM the work it should never have been doing, and brought Sweet Pal's chat replies under a second and a half.

Sweet Pal is a companion for people living with diabetes. You write "I just had a pizza" and it has to do many things at once: understand it's a meal worth logging, prepare the card, recall how that food moved your glucose the last few times, decide whether to suggest something or leave you alone, and answer with a sentence that sounds human. Until a few weeks ago, a single language model did all of it.

It worked. But it was slow, and in this app slowness weighs more than elsewhere. The person writing has just eaten, or has just watched their glucose drop. Waiting ten seconds in front of three blinking dots isn't a performance detail. It feels like not being heard.

## The problem: a model that thinks out loud

Every message went to the LLM with a huge prompt: who Sweet Pal is, the safety rules, the baseline clinical knowledge, the manual for more than thirty tools. The model read it all, decided which tools to call, waited for the results, sometimes called again, and only at the end wrote. Two or three rounds per turn were normal.

The thing is, most of that work wasn't language. It was classification. Is this a meal or a question? Does the bolus card belong here? Is the person struggling? These are questions with closed answers, and we were having them solved by the slowest and most expensive component in the system, in the most expensive way possible: in prose, while reasoning.

## The idea: separate deciding from saying

The turning point was JEV, TypeSafe's "System One" model. JEV doesn't write text. It takes some content and a set of typed questions, and answers all of them at once, in parallel, with values: one option out of many, a level on a scale, a probability that a statement is true. In about a quarter of a second.

On top of JEV we built an **Operator layer**: the **JOLs**, JEV Operator Layers. A JOL is a group of atomic questions plus a set of rules, written in the open, that turn the answers into a decision. JEV says *what* is true; the rules say *what to do*; a confidence threshold says *how far to trust it*. The language model only arrives at the end, when everything has been decided, and does the one thing it does better than anyone: talk.

We called the architecture **J2LM**: JOLs, then LLM.

![Diagram comparing a single LLM working in several rounds with J2LM: message, four parallel JOLs, script, LLM in one round.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-fig1-architecture.svg)

*Figure 1. Before, one component decided, searched and wrote, over several rounds. In J2LM the decision is spread across specialist JOLs that all start at the same instant; the model only comes in for the voice.*

## A wave, not a chain

At first the JOLs ran in a line: a first triage layer handed off to the specialists. Measuring, we noticed that the first layer *always* handed off to all of them, whatever it answered. Waiting for it was half of the JOL time, spent learning something we already knew.

Today the engine recognizes decisions that don't depend on the answers and launches everything in a single wave. There are four specialists: what to show, what to read, what to compute, what to remember or fix. Next to them sits the direction layer, which works out *how* to answer. Median JOL time dropped from about 580 to about 290 milliseconds.

## How you build a JOL

Writing questions for JEV looks easy, and it isn't. A question always seems clear to whoever just wrote it. We learned some rules the hard way.

- **One question, one thing.** A single compound question ("do we need to read the data or the sources?") missed obvious cases. Split into three atomic one-line questions, it got every test sentence right.
- **Every specialist owns its "nothing".** A gate in front that guesses "is a specialist needed?" redoes, badly, work the specialist already does well. Better to ask all of them and let each one say "not mine".
- **Test first, then activate.** No version goes live without passing, against the real model, a set of reference sentences. Old versions are never deleted: they're switched off and kept for comparison.
- **Prune.** The direction layer had twelve registers to describe what the person is doing. We removed four, because the specialists could already draw those distinctions. With fewer options JEV became more confident: minimum confidence in our tests went from 0.36 to 0.55, and every sentence was classified correctly.
- **Safety has two doors.** "I'm at 62, I feel fine, it happens all the time" was read as a story. An atomic question about a risky value, which overrides everything else, turns it into what it is: an emergency. Here erring on the side of caution costs little; erring the other way doesn't.

## The script: the last layer

The first J2LM was already faster than the app, yet the model still took longer than it needed. The JOLs decided, but didn't tell the model *how* to speak. It received the entire persona and a list of cards, and worked out on its own whether this was an emergency, a greeting or someone venting, how long to be, whether to end with a question.

A JOL can't be the last layer, because JEV doesn't write. The last layer is code: the **script**. From the direction layer's answers and the specialists' decisions, it picks which parts of the persona this turn needs, how many sentences to write, what to say about each card and what not to say, and whether to close with a suggestion. The model is left with language alone.

> Before, the model received everything and decided what to ignore. Now it receives only what it has to use.

- **−41%** — median end-to-end reply time on a fixed set of sentences
- **−35%** — prompt tokens for the voice once the script was introduced
- **~25×** — fewer tokens read per turn than the multi-round agent
## The number that matters is the first one

For weeks we measured total time. Then we brought into our test bench the same streaming the app uses, and started measuring what the person actually feels: the moment the bubble starts to fill. We call it the **first token**, and it turned out to be the most instructive number of all.

In one measurement of the J2LM voice, the first text arrived at 1.7 seconds out of 2.1 for the complete reply. Almost all of the model's time is waiting before it starts, not writing. That moved the work: to go faster we didn't need the model to write less, we needed it to read less and think less before opening its mouth.

![Timeline bars for two models, J2LM against the agent, marking first token and the moment the cards appear.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-fig2-first-token.svg)

*Figure 2. A meal being reported, same message, two different models. Teal: JOLs. Purple: the model. Circle: first token. With J2LM the reply and the cards land together; with the agent the text can start early, but the cards only come at the end of the turn.*

![Screenshot of the test bench with two chat panes and two timelines; model and layer names are blurred.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-playground-meal.webp)

*From the test bench. One message goes to both chats at once: J2LM on the left, the classic agent on the right. Under each reply, first token and total time. On the right, the JOLs over time, all launched in the same wave, then the voice; next to them, the agent's rounds. The interface is in Italian; internal names are blurred.*

![Screenshot of the two chats answering “I ate two prunes”, with timings under each reply.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-playground-prunes.webp)

*Same message, second model. J2LM: first token at 1.4 s, reply and three cards at 1.6 s. Agent: first token at 2.3 s, two rounds, one card at 3 s.*

## Independent of reasoning

Today's models can "reason" before answering, and that costs time. In the classic chat it's a price worth paying, because the model has to decide everything. In J2LM the decision has already been made. We measured what happens if the voice reasons anyway: at a medium reasoning level the first text goes from about one second to four to eight. At a high level, in one case, the model thought for 57 seconds to write four sentences the script had already laid out. And it wrote them worse.

![Log-scale bars of first-token time for flat, medium and high reasoning: about 1, 4 to 8, and 57 seconds.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-fig3-reasoning.svg)

*Figure 3. Voice first text in J2LM on short sentences, by reasoning level; bars show the measured range. The thinking is done by the JOLs; the model's own thinking, here, is mostly waiting.*

![Screenshot of a turn where the voice took 57 seconds to start, with the JOL timeline finishing in under half a second.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-playground-reasoning.webp)

*The 57-second turn. The same voice with a high reasoning level: the JOLs decided in about 300 ms, then the model thought for 57 seconds before writing four sentences. (The reasoning-token count shown is an early underestimate; the model only streams a summary of its reasoning.)*

Hence the rule we now call the **two-speed main**. The J2LM voice is flat, with no reasoning, almost always. It becomes deep, with the reasoning level the subscription provides, only when the JOLs ask for it: the person wants to go deeper, the direction layer couldn't read the message, or there's a computation on their data to explain. The model no longer decides how much to think. The architecture does.

There's a valuable side effect. Chat speed no longer depends on how much tomorrow's model likes to reason. Two very different models, one open and one proprietary, both completed a reported meal, reply and cards included, in about a second and a half.

## Costs

A classic chat turn reads tens of thousands of prompt tokens, and when the model takes two rounds it reads them twice. A J2LM turn reads the script and the slice of persona it needs: three to four thousand tokens in a single round, plus a JEV call that costs a fraction of that. Every token not read is time saved before the first token, and one less line on the bill.

![Two horizontal bars comparing prompt tokens per turn: about 70,000 for the agent and 2,800 for J2LM.](https://sweetpal.ai/media/j2lm-decide-first-speak-later/j2lm-fig4-prompt-tokens.svg)

*Figure 4. How much the model has to read before answering. The JEV call isn't in the chart: it's small, and it runs in parallel.*

## What's still missing

Today J2LM picks the cards but doesn't fill them, and the specialists decide what to read but reading the actual data is still to be wired in. The classic chat already does that work, so part of the advantage comes from work J2LM hasn't started yet. The next layer is exactly that: extract the values, read the data, compute, before the voice speaks. We'll measure it on the same test bench, because it's the only way to know whether the advantage holds.

One lesson is already settled. A language model is an extraordinary tool for saying things, and a slow and expensive one for deciding things that have a closed answer. Separating the two gave us back the time the people who use Sweet Pal deserve.

---

*Figures come from our internal test bench, which reproduces the app's chat with the same prompt, the same providers and the same streaming. They're medians over fixed sets of sentences, October 2026.*