Jev is the only sub-second model that will tell you it doesn't know

TypeSafe shipped a model this week that returns calibrated probabilities instead of text, so I ran it against fifteen others on the same 150 judgments. It finished tenth on accuracy. I threw away two headlines before the data stopped contradicting me: the first said Haiku was more accurate, which held on exactly one task, and the second said Jev's confidence was usable and Haiku's was not, which died the moment I computed calibration error properly and got 0.121 against 0.122.

What survived is a column I almost did not measure. Only four of the sixteen models will place an answer in the ambiguous middle of the probability scale. Everything else answers almost everything at near-certainty, and price does not predict which is which – the most expensive model in the set finished last on accuracy and last on calibration, while the cheapest ran the whole benchmark for $0.003 and beat Jev outright. If your code escalates uncertain rows to a human, that rule fires on 34.7% of Jev's rows and 2.7% of Haiku's. The ambiguous rows did not go anywhere; they just went through.

Read the 16-model benchmark →

Reply and tell me what your code does with a row it is unsure about. I read every reply.

– Alex

P.S. – I put the Claude Code hook that came out of this in a public repo: wotai-dev/typesafe-jev-tools. It fires on Write and Edit and asks whether the decision needs a model at all. The conversations behind reads like these happen in the WotAI Skool community. 760+ builders, three live calls a week, free to jump in.