tacit

An independent EU benchmark for System 1 AI

How much can you hand to a fast model?

Tacit measures how many business decisions in EU languages, under EU rules, a System 1 model can take on its own while staying under a fixed error rate. The answer is proven on a hidden test the model has never seen.

Model
Error target

Jev 1.13 takes 585 of 652 decisions on its own, with 20 wrong (3.4%, bound 5.2%). The other 67 go to a person.

Automated by the modelDeferred to a personWrong. The count is exact; where they sit in the grid is illustrative.

Independent. Not affiliated with or endorsed by EU institutions.

Languages

Items are written in each language, not translated from English. v0.1 is Polish only.

Leaderboard at 1 in 20

v0.1, hidden test, n = 652, run 19 October 2026 by the maintainer

Ranked by certified coverage at a 5% error target. There is no composite score. All 6 models hold the target on test.

#ModelCertified coverageErrorsRealised error, 95% interval (target 5%)
1Jev 1.13API, size not disclosed, OpenRouterCoverage89.7%Errors20 of 585Realised error3.42% [2.10–5.23]
2basal-omni-1.0-max12B, Gemma 4, Apple M5 Max, localCoverage86.0%Errors9 of 561Realised error1.60% [0.74–3.02]
3basal-2.0-max11B, Bielik v3, Apple M5 Max, localCoverage85.6%Errors27 of 558Realised error4.84% [3.21–6.96]
4basal-2.0-4.5B4.5B, Bielik v3, Apple M5 Max, localCoverage69.2%Errors10 of 451Realised error2.22% [1.07–4.04]
5basal-omni-1.0-mini4B, Gemma 4 E4B, Apple M5 Max, localCoverage64.9%Errors11 of 423Realised error2.60% [1.31–4.61]
6basal-2.0-mini1.5B, Bielik v3, Apple M5 Max, localCoverage50.6%Errors14 of 330Realised error4.24% [2.34–7.02]

At 1 in 100, no model is certified. Full leaderboard with calibration, accuracy, latency and cost

What an item looks like

A short business text and one structured question. This one is from the public dev split. Try it before you see what the models said.

R39Complaints (reklamacja)Yes or notricky

Customer message to a furniture shop

Chciałbym zamówić jeszcze dwa takie same fotele. Aha, i przy okazji: w tym, który już mam, po miesiącu pękł podłokietnik, więc proszę też o wymianę tej części.

I’d like to order two more of the same armchairs. Oh, and by the way: on the one I already have, the armrest cracked after a month, so please replace that part too.

Czy ta wiadomość jest reklamacją…?

Is this message a complaint, i.e. the customer reports a defect and asks for repair, replacement, a price cut or a refund?

Gold label: yes. Both blind labelers agreed.

Each model’s confidence in “yes”

After each model’s own temperature, fitted on dev. The tick is that model’s own threshold at 1 in 20. At or above it, the model answers on its own; below, the case goes to a person. A model can be very sure in raw terms and still be sent to a person once it is calibrated.

Jev 1.130.974Answers on its own (threshold 0.861, raw 0.900)
basal-omni-1.0-max0.974Answers on its own (threshold 0.842, raw 0.996)
basal-2.0-max0.962Answers on its own (threshold 0.833, raw 0.998)
basal-omni-1.0-mini0.931Goes to a person (threshold 0.963, raw 0.997)
basal-2.0-4.5B0.922Answers on its own (threshold 0.906, raw 0.998)
basal-2.0-mini0.850Goes to a person (threshold 0.909, raw 0.975)

Take part

Put your model on the board

Run the open harness on the dev split, then open a submission thread. The maintainer runs the hidden test once and publishes the row with full run details.

How to submit a model

Review items in your language

v0.2 adds English, German and Norwegian, about 1,000 items each, written in the language. Native speakers who know customer service, compliance, public administration or security are credited by name.

Offer to review

Supported byProsit ASSupporters get no access to the hidden test and no say in results. Independence