What independent tests found

Typed output is not correct output, and confidence is a group-level property. This page collects what third parties measured, the failure modes they documented, and the checklist that follows from both. If a number here changes your architecture, that is the point.

The findings

Study-specific observations, not a leaderboard. Read each source's protocol before carrying a number into your own policy.

Study	What was measured	Result	What to do about it
KoBBQ calibration audit	Abstention behavior on 300 ambiguous items	Jev chose "unknown" for 95% when the option existed. With it removed, 79% of answers picked the dataset's stereotype.	Always include an unknown or no-match option, and measure forced-answer behavior on your own ambiguous cases.
jev-certify (CLINC150)	A 5% bound on silently misrouted incoming queries	Held on 400 in-scope examples: 84.75% auto-routed at 2.25% loss per incoming query. A separate scope gate missed its 5% target by 3.6x when out-of-scope traffic rose.	Certify on deployment-like traffic, and monitor the traffic mix. The bound does not survive a shifted population.
jev-orderby-bench	Whether a good classifier is a good sort key	Passed six ranking gates on 360 topic rows, then failed four of six on 306 shopping pairs. 53 rows tied at 0.99; batching 40 rows changed a passing gate into a failure.	Measure pairwise order, ties at the cutoff, and your exact request shape on your own relevance labels.
Janus cascade study	Whether a Jev-first cascade beats either model alone	Improved Banking77 accuracy over either model alone, but on Web of Science it matched Jev alone at 47% higher cost.	Label representative cases and price both legs before adopting a cascade. A fallback is only worth it where it fixes real errors.
Action-gate study (111 cases)	Approving or rejecting agent actions	Jev matched 100 case labels, Claude matched 102; each had one unsafe allow. Contract and policy mapping was the largest single error source for both.	Test the answer-to-action mapping as seriously as the model. Keep deterministic policy on consequential tool families.
Workflow evals (vendor-run)	Four structured automation workflows against a frontier-model consensus	67.8% agreement at $0.0004 per case, versus GPT-5.6 Terra at 67.9% at roughly 76x the cost.	Treat the multiplier as directional. Vendor-run means vendor-chosen workflows; replicate on your own.
LangChain agent-trace judge	Five fixed traces, 100 repeats each, against an independent human reviewer	Agreed with the human on 500 of 500 calls for $0.18 total. Claude Sonnet 4.6 on the same 500 cost $28.17.	The cost inversion is the finding: a judge at $0.00035 per call can be run on every step. Perfect agreement on five traces is still five traces.
Jev vs LLMs as rubric judges (paper, 24 Sep)	5,003 submission-criterion pairs across nine panels from seven public benchmarks	Jev's confidence ranked its own errors on most panels. But the LLM fallbacks repeated Jev's wrong answer in 96% of sampled verdict pairs.	A cascade is not automatically a safety net. When the fallback makes the same mistake, you have paid twice for one error.
Jev-as-a-Judge cascade (paper, 22 Sep)	A confidence-gated Jev-to-frontier cascade on 510 held-out preference pairs	Jev handled 53.7% of pairs at a 0.90 threshold. Cascade accuracy 92.5% versus 93.1% for the frontier model alone, a -0.59 point difference with a 95% interval of -1.78 to +0.59.	A real but statistically thin loss. Price both legs and check whether the accepted slice is where the fallback is actually needed.
Good Start Labs rubric study	6,003 rubric checks scored by Jev and five frontier models on identical instructions	Jev matched Claude Fable 5.1's verdict 91.5% of the time, at $160 per million graded answers against roughly $33,000 for the model doing the same job.	Agreement is not accuracy, and this measures agreement. Note the reviewer was a model, not a person.
Deel production report	Jev inside a live HR platform's decision paths	Up to 59x lower cost and 4x faster responses with quality maintained, per the vendor's own account of a customer result.	A vendor quoting a customer. Ask for the label set and the fallback rate before repeating the multiple.

Two failure modes recur across these studies and are worth stating plainly: Jev cannot abstain, so when no option is right it picks the least wrong one, and it never explains itself, so a wrong answer arrives as a tidy probability with no rationale to inspect. Design an explicit needs-review lane, and produce the "why" in your own code.

Cross-model benchmarks worth reading

Public harnesses with published code and per-decision logs. Some latency and hosting costs in these are estimates.

JevBench

Composite score across accuracy, calibration, speed, and cost, with frozen tasks, adapters, scoring code, and result artifacts.

Jevals.com

Independent hosted-model benchmark with public suites and per-decision logs.

jev-sec-bench

Blind prompt-injection and vulnerable-code detection: 96.5% accuracy and ECE 0.0588 across 662 samples.

Agent failure benchmark

Diagnosing what broke an agent across 6,257 traces for $1.28, beating GPT-5.4 on every measured axis.

Rerank bench

nDCG@10 of 0.692 versus Cohere's 0.691 across 14 datasets. A tie is a result.

Phishing bench

2,000 emails judged against Claude Haiku 4.5.

awesome-jev-security

Curated index of papers on the security, robustness, and safety of Jev and System One models. New in late September.

zh-decision-bench

First Chinese-language calibration benchmark for Jev-class decision models, covering a gap in an English-dominated literature.

JEVals

Open-source playground for evaluating System One and Jev-compatible models with deterministic replay.

The measurement checklist

What every serious deployment in this ecosystem does, regardless of domain.

Label your ownAccuracy, abstention rate, and calibration are per-task properties. Public numbers do not transfer without your own labeled set.

Pin the versionAn alias like jev-latest moves. Record the exact model version next to every threshold you tune, and rerun the labeled set before bumping.

Threshold the probability, not the vibeSet cutoffs from measured curves: precision, recall, coverage, and the cost of each error direction.

Log the decisionState, questions, model version, probabilities, chosen action, and the eventual outcome. There is no model-provided rationale, so this log is your audit trail.

Design the review bandThe uncertain middle needs somewhere to go that is not an auto-action. Human queue, escalation model, or a cheaper re-ask.

Watch the mixScope drift breaks certified bounds. Monitor what kinds of traffic actually arrive, not what you assumed.

Documented failure modes

TypeSafe publishes a per-version jaggedness page. The summary, because it saves a week of debugging.

Reads literallyIt answers the question you wrote, not the one you meant. Negations and scope words land at face value.

Not a calculatorCounting, arithmetic, and date comparison are unreliable. Keep them in code.

Context rotAccuracy falls as the state fills with material the question does not need. Filter first, send only what the question needs.

Hostile stateText engineered to steer its own classification can move the answer. Treat user-controlled content as untrusted input.

Cannot abstainA forced binary picks the least-wrong answer. Design the escape hatch yourself.

Option orderReordering the options can move the answer. Test important classifications with reordered choices.

Source: TypeSafe's published model jaggedness documentation, plus the independent studies above. Both are worth reading in full before a production commitment.

Last updated 30 September 2026.