Jev before Claude Sonnet: how rephrasing a question pushed AUC to 0.95

The Lab and the problem of expensive, empty breakdowns

The author runs a pipeline called the Lab. Everything interesting during the day — reels, posts, links to repositories — gets saved to Telegram. Once a day, a script pulls in the new items, and Claude Sonnet writes a breakdown for each one. Then a second pass, acting as editor, decides whether to publish the breakdown or send it to the archive.

The problem: the saved items mix useful guides, work messages, personal notes, and images with no text at all. When 67 items were labeled, 28 turned out to be junk. Sonnet dutifully wrote a full breakdown for every one of those junk items, only for the editor to throw it out — a pure waste of time and quota. Sonnet runs through a Claude Code subscription with a weekly limit, so a cheap filter was needed to screen out obvious junk before it ever reached the big model.

Jev from TypeSafe: a model for closed-ended questions

The solution was Jev — a model from the startup TypeSafe, released September 15, 2026. It's designed for closed-ended questions: you send text plus a dictionary of questions, and instead of a paragraph of text, you get back values with probabilities. Input costs $0.042 per million tokens; output is free.

Three question types:

• noul: returns the probability of "yes," from 0 to 1
• choice: picks from a submitted list (up to 255 options) and returns the probability of each option plus a confidence score
• score: rates on an ordered scale

All questions about a single text are processed in one pass. According to TypeSafe's own benchmarks, Jev's answers match the ground truth from large models two times out of three (68%).

When the author tried to sign up, TypeSafe wasn't registering new users. But Jev is available on Cloudflare Workers AI under the name typesafe/jev, marked Zero data retention. Quirks: you can only pay from a prepaid AI Gateway balance, and the answer sits one level deeper in the response than TypeSafe's documentation suggests (result.result.answers).

The main lesson: phrasing matters more than the model

Before the Lab, the author had tried Jev on a different task — filtering facts for a knowledge base. The first question was generic: "is this a useful fact?" On the labeled dataset, that produced an AUC of 0.46 — worse than a coin flip. The author then rewrote the question to include an actual reason: "is this about the real world, rather than a document edit or technical detail." Same model, same texts, and AUC jumped to 0.95. On taste-based judgments, it never climbed above 0.72–0.77 — there, Jev only works as a cautious pre-filter.

The author carried that rule into the Lab: don't ask the model about "usefulness" in the abstract — list the actual reasons an item gets archived.

A spec for the

Lab: six categories instead of fuzzy questions

Instead of several yes/no questions, the Lab uses a single choice question, where the options are mutually exclusive and mirror the real rejection reasons from the editor's log:

• guide: a specific tool, service, repository, prompt, skill, setting, or technique is shown or named, and the reader could find and replicate it. This counts even if it's an ad
• idea: a scheme, framework, formula, or step-by-step model that can be broken down into points
• no_text: almost no text or speech, just a link, image, or photo
• owner_work: the owner's own work — a message to a colleague, a post draft, a link to their own product
• personal: personal content unrelated to work with AI — sports, family, health, daily life
• no_method: an opinion, claim, ad, or news item with no tool, technique, or framework

The probability of "keep" is calculated as the sum of the guide and idea probabilities. If it falls below a 0.2 threshold, the item never reaches Sonnet.

Two weeks in production: $0.0028 and one real saving

From September 26 to October 6, the log shows 26 calls. Together they cost $0.0028 — roughly a hundredth of a cent per call, on texts ranging from 90 to 3,100 characters. From Moscow, responses take longer than the stated 70–500 ms (TypeSafe measured from West Coast laptops): median 1.3 seconds, first call 2–9 seconds, subsequent calls usually under 1.5 seconds.

In the first production run on October 5, Jev filtered out one item — a media file with almost no text, with a probability of 0.01. The money saved is trivial, but the quota saved is noticeable: previously, every such item ate up a full Sonnet breakdown and an editor pass.

Where Jev doesn't work

Open-ended tasks that require a plain-language explanation are still a job for an LLM. Taste-based judgments without clear signals remain weak (the author's own AUC of 0.72–0.77 confirms this). Counting, dates, and long noisy text are also weak points — the model reads the question literally, counts poorly, and gets confused by dates. Sensitive data is another concern — texts pass through an American service via Cloudflare, so it's worth deciding in advance what's safe to send.

Jev works well where the same small fork in the road repeats many times, where there are clear signals, and where an error in one direction is cheaper than in the other. For the author, that's screening content before it reaches an expensive model.