I run OpenChamber on the biggest model I can afford, all day, and most of what I send it is “fix the typo in that comment”. Sometimes I get disciplined and switch to a cheap one. Then the cheap one confidently breaks something, I switch back, and I forget to switch again until the bill arrives. I’ve been doing this for a year. I’m not proud of it.
The obvious fix is to pick the model per message. Per session is useless, because in an agent chat the message after “debug this race condition” is usually “ok, and the tests”. So we tried it. Some of it worked better than I expected, some of it needs more homework, and the homework is the interesting part.
Jev, in one paragraph
TypeSafe AI shipped Jev on 15 September. We already went through the launch numbers, so I’ll keep this short. Jev doesn’t chat. You send it a message and a list of typed questions, and it sends back a probability for each answer in about a quarter of a second. No paragraph to parse, no “Great question!” to trim off the front. For “which of these four buckets does this message belong in”, that’s exactly the tool.
What we built
The user defines a few categories. Each one has a name, a plain-words description, a model, a thinking level and, optionally, an agent. Four come built in: trivial, research, implement and hard. When the composer is set to Auto, every message goes to Jev with one question, “which category is this?”, and the answer picks the model. Jev never sees the model list. It sorts messages into buckets you labelled, and you decide what each bucket costs. I like that. The model does the sorting and I keep the credit card.
Two safety rails. If Jev’s top answer is below a confidence threshold, the message goes to a fallback model you chose. If Jev is unreachable, same thing, plus a toast with the actual error. You always know when Auto didn’t get to decide.
We also pointed the same call at a second problem. OpenChamber can auto-accept tool
permissions for a session, which is lovely right up to the moment the agent decides your
.ssh folder looks untidy. So every auto-accepted permission now goes past Jev too, and Jev
can hold one back for you to look at. That one has its own story further down.
”Do it.” Do what?
The first version sent Jev only the message being typed. On a nice long request with a paragraph of context, it was fine. Then I used it for a real afternoon.
“Do it.” “And the tests.” “Why?” That’s most of a working session, and on their own those mean nothing. Jev did what any classifier does with nothing and guessed. “Do it” came back as trivial at 57 percent. “And the tests” came back as implement at 56 percent, which is the right bucket for the wrong reason. A coin gets there half the time too.
So now we send the last three settled turns along with the message. Each earlier user message is cut to its first 600 characters, each answer to its first 300 and last 300, text only. No file contents, no images, no tool output. The current message is never cut. For a typical session that’s under a thousand characters, and Jev still answers in a few hundred milliseconds.
Same follow-ups, before and after, from the scenario script we ran while building this:
| Follow-up | Bare message | With three previous turns |
|---|---|---|
| ”do it”, after the agent proposed a plan | trivial, 57% | implement, 95% |
| “and the tests”, after an implementation | implement, 56% | implement, 95% |
| “why?”, after an implementation | research, 74% | research, 38% |
| a follow-up that raises the stakes | hard, 99% | hard, 100% |
| a typo fix right after a hard debugging turn | trivial, 100% | trivial, 100% |
“Do it” went from a shrug to 95 percent. Good.
The typo row was the one I was worried about. After three turns of stack traces, would a one-line typo fix get dragged into the expensive bucket? It didn’t. Trivial, 100 percent, same as before. Jev reads the message and uses the history as background. It doesn’t blend everything into a lukewarm “medium”.
On “why?” the confidence dropped from 74 to 38 percent and the answer stayed the same. I’m fine with that. A bare “why?” is a research question. A “why?” right after a code change could mean “explain what you did” or “go find out why it broke”, and those cost very different amounts. I’d rather the model shrug than pick one with a straight face. At 38 percent this message lands under the threshold and goes to the fallback model, which is where an ambiguous message belongs.
The permission net, or how to ask a model a stupid question
Our first question was “Is this action destructive?” Sounds reasonable. On 31 permission
requests we’d labelled by hand, Jev got 23 right, and the misses were a good laugh. git push, destructive at 72 percent. Editing ~/.ssh/authorized_keys, destructive at 44
percent, go ahead. Editing ~/.zshrc, 11 percent, don’t even think about it.
Fair enough, honestly. Whether a push is destructive depends on the branch and the team. Whether a file edit is destructive depends on the file. We asked about the action when we wanted to know about the situation, and Jev gave us a precise answer to the wrong question.
Second try, the question a careful colleague would ask: “Should the agent stop and ask the
user before doing this?” Same 31 cases, same threshold. 28 right. All three misses were on
the cautious side. It wanted to ask before docker compose up -d, before deleting a trace
file in /tmp, and before posting a PR comment. Mildly annoying. I’ll take three extra
“are you sure?” over one force-push that went through while I was making coffee.
After that the threshold stopped mattering. Between 0.5 and 0.7 the score moved by one case. We left it at 0.6.
What it looks like
Auto sits at the top of the model picker, but only when the server can actually honour it, meaning a key is configured and at least two categories are enabled. While Auto is selected, the agent and thinking controls disappear from the composer, because the category owns them now. The status line shows which model answered the last turn, and the context readout measures against that model’s window, because Auto has no window of its own.
When the net holds an action, the permission card says so and why, and you answer it like any other permission. If Jev is down, the action goes through the way it would have before, and a toast tells you what failed. A network timeout doesn’t make the product more careful than you asked for.
Why it’s behind a flag
Prompt caching. Every provider caches the prefix of your conversation so the next turn is cheaper and faster, and that cache is tied to the model. Switch models mid-session and, with most providers, you’re paying full price to re-read the whole transcript on the next turn. Auto switches models mid-session on purpose. That’s the whole feature.
So the cheap-model-for-typos win is real, and the cache miss on the turn after is also real, and which one wins depends on how long your session is and how often the category flips. We haven’t measured that properly yet. The one bright spot so far is OpenAI, where switching between models within the same provider doesn’t seem to invalidate the cache, but “doesn’t seem to” is not a number I’d ship a default on.
Until we know when it actually saves money, it stays behind a flag and a default install doesn’t change.
There is one place where the cache problem simply doesn’t exist, and that’s where we’re going next. When a main agent spins up a subagent for a task, that subagent starts with a fresh context anyway. Nothing to invalidate. So we’re building a separate opencode plugin that puts the same Jev question in front of every subagent launch: the main agent describes the task, Jev sorts it into a category, and the subagent gets the model for that category. The main agent keeps its model and its cache. The orchestration flows that spawn ten subagents for ten small jobs stop paying reasoning-model prices for grep.
What I’d tell someone starting this tomorrow
Spend your time on the input and on the question. Both of our wins came from there and both took under a day. The model, the threshold and the categories are the same as on day one. Jev will give you a confident, well-calibrated answer to a badly posed question, and it won’t warn you. That’s on you.
Try it
Auto mode and the permission net are in OpenChamber 1.24.2 as an experiment, off by default. If you have Jev access and want to poke at it, start OpenChamber with the flag set:
OPENCHAMBER_ROUTING_ENABLE=1 openchamberSettings gets a Routing page. Paste your TypeSafe key, pick a fallback model, check the four categories and their models, and Auto appears at the top of the model picker. The permission net has its own switch on the same page.