Leaderboard
Each row is the share of 652 hidden test decisions a model takes on its own, with a Clopper-Pearson bound on its error. Ranked by coverage at the selected target. There is no composite score.
- Benchmark
- v0.1
- Last updated
- 20 October 2026
- Protocol
- tacit-devfit-1
- Hidden test
- n = 652
- Runs
- by the maintainer, no vendor approval
Error target
Calibration
Tacit fit: temperature and threshold are fitted per question type on the public dev split, then frozen and applied once to the hidden test.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage94.2%Automated614Errors14Realised error, 95% interval2.28% [1.25–3.80]Holds at 5%Accuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
2Jev 1.14API, size not disclosedOpenRouterCertified coverage92.0%Automated600Errors20Realised error, 95% interval3.33% [2.05–5.10]Holds at 5%Accuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
3basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage91.6%Automated597Errors9Realised error, 95% interval1.51% [0.69–2.84]Holds at 5%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.0-ultra.json (Tacit fit on dev)
4pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage90.6%Automated591Errors14Realised error, 95% interval2.37% [1.30–3.94]Holds at 5%Accuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
5Mercury DecideAPI, size not disclosedOpenRouterCertified coverage90.6%Automated591Errors18Realised error, 95% interval3.05% [1.81–4.77]Holds at 5%Accuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
6Jev 1.13API, size not disclosedOpenRouterCertified coverage89.7%Automated585Errors20Realised error, 95% interval3.42% [2.10–5.23]Holds at 5%Accuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
7Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage89.1%Automated581Errors19Realised error, 95% interval3.27% [1.98–5.06]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
8basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage87.1%Automated568Errors11Realised error, 95% interval1.94% [0.97–3.44]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.1-max.json (Tacit fit on dev)
9basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage86.0%Automated561Errors9Realised error, 95% interval1.60% [0.74–3.02]Holds at 5%Accuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-max.json (Tacit fit on dev)
10basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage85.6%Automated558Errors27Realised error, 95% interval4.84% [3.21–6.96]Holds at 5%Accuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-max.json (Tacit fit on dev)
11Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage84.4%Automated550Errors11Realised error, 95% interval2.00% [1.00–3.55]Holds at 5%Accuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
12Respan Span-02API, size not disclosedOpenRouterCertified coverage84.2%Automated549Errors17Realised error, 95% interval3.10% [1.81–4.91]Holds at 5%Accuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
13basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage69.2%Automated451Errors10Realised error, 95% interval2.22% [1.07–4.04]Holds at 5%Accuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-4.5B.json (Tacit fit on dev)
14basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage64.9%Automated423Errors11Realised error, 95% interval2.60% [1.31–4.61]Holds at 5%Accuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-mini.json (Tacit fit on dev)
15basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage50.6%Automated330Errors14Realised error, 95% interval4.24% [2.34–7.02]Holds at 5%Accuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-mini.json (Tacit fit on dev)
16GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage47.5%Automated310Errors9Realised error, 95% interval2.90% [1.34–5.44]Holds at 5%Accuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Vendor file: the calibration file the model author ships, applied unchanged. Only 7 of 16 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 5%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage99.7%Automated650Errors27Realised error, 95% interval4.15% [2.75–5.99]Holds at 5%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage92.9%Automated606Errors17Realised error, 95% interval2.81% [1.64–4.45]Holds at 5%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage91.9%Automated599Errors16Realised error, 95% interval2.67% [1.53–4.30]Holds at 5%Accuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage87.0%Automated567Errors29Realised error, 95% interval5.11% [3.45–7.26]Misses the 5% targetAccuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage69.2%Automated451Errors5Realised error, 95% interval1.11% [0.36–2.57]Holds at 5%Accuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage57.5%Automated375Errors5Realised error, 95% interval1.33% [0.43–3.08]Holds at 5%Accuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage22.4%Automated146Errors2Realised error, 95% interval1.37% [0.17–4.86]Holds at 5%Accuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
8Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
10pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
11Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
12Jev 1.13API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
13Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
14Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
15Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
16GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Tacit fit: temperature and threshold are fitted per question type on the public dev split, then frozen and applied once to the hidden test.
No model is certified at 1% yet. Every decision goes to a person.
A threshold needs at least 50 accepted decisions on dev with a one-sided 95% upper bound on error at or below 1%. With 602 dev items, no model reaches that yet. The table still shows each model's accuracy and calibration.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
2Jev 1.14API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
3basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.0-ultra.json (Tacit fit on dev)
4pplx-decider-v1.2API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
5Mercury DecideAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
6Jev 1.13API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
7Microsoft-Decision-1API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
8basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- basal-omni-1.1-max.json (Tacit fit on dev)
9basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-max.json (Tacit fit on dev)
10basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-max.json (Tacit fit on dev)
11Cloudflare ClefAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
12Respan Span-02API, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
13basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-4.5B.json (Tacit fit on dev)
14basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-omni-1.0-mini.json (Tacit fit on dev)
15basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- basal-2.0-mini.json (Tacit fit on dev)
16GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Vendor file: the calibration file the model author ships, applied unchanged. Only 7 of 16 models ship one. A model can reach a coverage and still miss the target on test; that is reported, not hidden.
#ModelCertified coverageAutomatedErrorsRealised error, 95% interval (target 1%)Accuracy, 95% CIECE raw / calBrier raw / calLatency p50 / p95$ per 1kDetails
1basal-omni-1.0-ultra31B, Gemma 4Apple M5 Max, localCertified coverage81.7%Automated533Errors1Realised error, 95% interval0.19% [0.00–1.04]Holds at 1%Accuracy, 95% CI95.6 [93.7–96.9]ECE raw / cal.024 / .016Brier raw / cal.060 / .056Latency p50 / p95549 / 963 ms$ per 1klocal
- Model id
Remek/basal-omni-1.0-ultra- Revision
e347c65- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
2basal-omni-1.0-max12B, Gemma 4Apple M5 Max, localCertified coverage71.9%Automated469Errors5Realised error, 95% interval1.07% [0.35–2.47]Misses the 1% targetAccuracy, 95% CI94.6 [92.6–96.1]ECE raw / cal.012 / .004Brier raw / cal.079 / .077Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-max- Revision
3278a02- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
3basal-omni-1.1-max12B, Gemma 4Apple M5 Max, localCertified coverage68.7%Automated448Errors3Realised error, 95% interval0.67% [0.14–1.94]Holds at 1%Accuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.015 / .010Brier raw / cal.080 / .079Latency p50 / p95225 / 409 ms$ per 1klocal
- Model id
Remek/basal-omni-1.1-max- Revision
a3af675- Engine
- basal-serve --no-calibration --mode eager --device mps, one option order, batch size 1
- Hardware
- Apple M5 Max, 128 GB unified memory
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- local, batch size 1; local, not priced
- Calibration file
- Vendor file, applied unchanged
4basal-omni-1.0-mini4B, Gemma 4 E4BApple M5 Max, localCertified coverage63.3%Automated413Errors4Realised error, 95% interval0.97% [0.26–2.46]Holds at 1%Accuracy, 95% CI90.2 [87.7–92.2]ECE raw / cal.049 / .021Brier raw / cal.145 / .135Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-omni-1.0-mini- Revision
d087a7b- Engine
- basal 2.2.0 (v2.2.0 = e49665c), eager --device mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
5basal-2.0-max11B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI91.3 [88.8–93.2]ECE raw / cal.050 / .025Brier raw / cal.133 / .122Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-max- Revision
2fffc5c- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
6basal-2.0-4.5B4.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI87.0 [84.2–89.3]ECE raw / cal.085 / .037Brier raw / cal.200 / .171Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-4.5B- Revision
e5ba914- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
7basal-2.0-mini1.5B, Bielik v3Apple M5 Max, localCertified coverage0%Automated0Errors0Realised error, 95% intervalNo threshold found on devNot certifiedAccuracy, 95% CI83.3 [80.2–85.9]ECE raw / cal.080 / .024Brier raw / cal.258 / .238Latency p50 / p95not timed$ per 1klocal
- Model id
Remek/basal-2.0-mini- Revision
cb81b32- Engine
- basal 2.2.0 (v2.2.0 = e49665c), mps, bf16, one option order
- Hardware
- Apple M5 Max, 128 GB unified memory, macOS 27.0.1
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- not timed (see notes); local, not priced
- Calibration file
- Vendor file, applied unchanged
8Liquid d1-ultraAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI96.9 [95.3–98.0]ECE raw / cal.012 / .007Brier raw / cal.051 / .049Latency p50 / p95297 / 580 ms$ per 1k$0.011
- Model id
liquid/d1-ultra- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 580 ms on the provider endpoint; $0.011 per 1,000 decisions
- Calibration file
- d1-ultra.json (Tacit fit on dev)
9Jev 1.14API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.9 [93.0–96.4]ECE raw / cal.022 / .013Brier raw / cal.076 / .075Latency p50 / p95287 / 402 ms$ per 1k$0.024
- Model id
typesafe/jev-1.14- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 287 / 402 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.14.json (Tacit fit on dev)
10pplx-decider-v1.2API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.2 [93.3–96.6]ECE raw / cal.017 / .011Brier raw / cal.069 / .068Latency p50 / p95350 / 448 ms$ per 1k$0.006
- Model id
perplexity/pplx-decider-v1.2- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 350 / 448 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- pplx-decider-v1.2.json (Tacit fit on dev)
11Mercury DecideAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.040 / .027Brier raw / cal.098 / .089Latency p50 / p95327 / 781 ms$ per 1k$0.006
- Model id
inception/mercury-decide- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 327 / 781 ms on the provider endpoint; $0.006 per 1,000 decisions
- Calibration file
- mercury-decide.json (Tacit fit on dev)
12Jev 1.13API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.3 [92.3–95.9]ECE raw / cal.022 / .021Brier raw / cal.083 / .085Latency p50 / p95310 / 403 ms$ per 1k$0.024
- Model id
typesafe/jev-1.13- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 19 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 310 / 403 ms on the provider endpoint; $0.024 per 1,000 decisions
- Calibration file
- jev-1.13.json (Tacit fit on dev)
13Microsoft-Decision-1API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.8 [92.8–96.2]ECE raw / cal.042 / .020Brier raw / cal.091 / .086Latency p50 / p95346 / 455 ms$ per 1k$0.010
- Model id
microsoft/microsoft-decision-1- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 346 / 455 ms on the provider endpoint; $0.010 per 1,000 decisions
- Calibration file
- microsoft-decision-1.json (Tacit fit on dev)
14Cloudflare ClefAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI94.2 [92.1–95.7]ECE raw / cal.016 / .019Brier raw / cal.087 / .086Latency p50 / p95322 / 474 ms$ per 1k$0.018
- Model id
cloudflare/clef- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 322 / 474 ms on the provider endpoint; $0.018 per 1,000 decisions
- Calibration file
- clef.json (Tacit fit on dev)
15Respan Span-02API, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI93.3 [91.1–94.9]ECE raw / cal.035 / .016Brier raw / cal.108 / .102Latency p50 / p95296 / 2262 ms$ per 1k$0.003
- Model id
respan/span-02- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 296 / 2262 ms on the provider endpoint; $0.003 per 1,000 decisions
- Calibration file
- span-02.json (Tacit fit on dev)
16GPT-6 Luna DecisionsAPI, size not disclosedOpenRouterCertified coveragen/aAutomatedn/aErrorsn/aRealised error, 95% intervalNo vendor file shippedAccuracy, 95% CI95.1 [93.2–96.5]ECE raw / cal.012 / .053Brier raw / cal.076 / .101Latency p50 / p95297 / 405 ms$ per 1k$0.036
- Model id
openai/gpt-6-luna-decisions- Revision
not pinned (API)- Engine
- Hardware
- Test run date
- 20 October 2026
- Run by
- The maintainer. No vendor approved these numbers.
- Latency and cost
- 297 / 405 ms on the provider endpoint; $0.036 per 1,000 decisions
- Calibration file
- gpt-6-luna-decisions.json (Tacit fit on dev)
Automated shareDeferred to a personRealised error and 95% intervalError target
Accuracy by question type
Certified at 5%: yes/no and choice. Score items are too few on dev to fit a threshold, so they are reported but not certified.
| Model | Yes/no n=322 | Choice n=310 | Score n=20 |
|---|---|---|---|
| Liquid d1-ultra | 95 | 98.7 | 100 |
| Jev 1.14 | 92.5 | 97.1 | 100 |
| basal-omni-1.0-ultra | 93.2 | 97.7 | 100 |
| pplx-decider-v1.2 | 92.9 | 97.4 | 100 |
| Mercury Decide | 88.8 | 97.7 | 95 |
| Jev 1.13 | 91.6 | 96.8 | 100 |
| Microsoft-Decision-1 | 92.2 | 97.1 | 100 |
| basal-omni-1.1-max | 92.5 | 96.8 | 100 |
| basal-omni-1.0-max | 91.9 | 97.1 | 100 |
| basal-2.0-max | 87.6 | 94.8 | 95 |
| Cloudflare Clef | 91 | 97.1 | 100 |
| Respan Span-02 | 89.8 | 96.5 | 100 |
| basal-2.0-4.5B | 81.4 | 92.3 | 95 |
| basal-omni-1.0-mini | 85.7 | 95.5 | 80 |
| basal-2.0-mini | 79.5 | 87.4 | 80 |
| GPT-6 Luna Decisions | 93.5 | 97.1 | 90 |
Accuracy in %, hidden test.
Accuracy by family
19 families, 27 to 76 test items each. Darker means more accurate. Small families move a lot with one item, so read single cells with care.
FamilyLiquid d1-ultraJev 1.14omni-1.0-ultrapplx-decider-v1.2Mercury DecideJev 1.13Microsoft-Decision-1omni-1.1-maxomni-1.0-max2.0-maxCloudflare ClefRespan Span-022.0-4.5Bomni-1.0-mini2.0-miniGPT-6 Luna Decisions
public administrationadministracja, n=2796.388.992.688.988.988.988.992.692.692.681.596.392.685.277.892.6
bank / AMLbank_aml, n=2792.681.581.588.974.188.988.985.285.281.585.277.881.592.685.288.9
cybersecuritycyberbezpieczenstwo, n=7697.496.196.197.494.794.797.492.192.193.492.196.177.688.269.797.4
escalationeskalacja, n=38100.097.4100.097.4100.097.494.797.497.492.194.792.197.497.494.7100.0
HRhr, n=2792.688.992.692.688.988.988.992.685.281.592.692.677.877.881.588.9
customer irritationirytacja, n=38100.097.494.7100.097.494.7100.094.797.489.5100.092.192.194.784.297.4
IT securityit_security, n=27100.096.396.396.396.396.396.3100.0100.092.696.392.681.588.977.896.3
law firmkancelaria, n=2892.992.992.992.992.992.996.4100.0100.082.192.996.489.389.385.792.9
completenesskompletnosc, n=38100.092.1100.089.586.889.594.797.497.484.297.494.773.784.268.494.7
logisticslogistyka, n=2889.389.389.389.389.389.385.789.385.789.385.792.985.782.182.185.7
moderationmoderacja, n=2892.989.392.992.989.392.992.985.785.7100.092.992.992.992.975.089.3
phishingphishing, n=38100.097.4100.0100.0100.097.4100.0100.0100.097.492.1100.0100.0100.097.4100.0
manufacturingprodukcja, n=2896.492.992.996.489.389.389.389.389.389.392.989.385.782.182.192.9
clinicprzychodnia, n=2892.9100.092.996.489.3100.0100.096.496.489.396.489.378.685.775.0100.0
complaintsreklamacja, n=38100.0100.0100.0100.0100.0100.097.4100.0100.0100.0100.0100.097.4100.097.4100.0
routingrouting, n=54100.0100.0100.098.196.3100.098.1100.0100.098.1100.0100.098.1100.096.3100.0
insuranceubezpieczenia, n=2892.992.992.989.392.989.389.392.989.389.392.982.175.082.171.492.9
universityuczelnia, n=2896.4100.096.496.492.9100.096.492.9100.089.3100.089.389.378.692.992.9
housing communitywspolnota, n=28100.0100.0100.096.4100.092.992.996.496.489.396.489.382.196.489.389.3