NullspanBench — Measured Decision & Extraction Leaderboard
Following the Benchmark Heaven (JevBench) evaluation protocol with strictly measured systems only (measured: true + cryptographic predictions_sha256): comparing 15 empirically measured systems across Intelligence (I), Probability Calibration (C), Speed (S), and Cost (K) on the Headline Sealed Pool (P=164: Stanford SQuAD v2.0 n=60, Multi-Numeric Legal/Commercial Contracts n=96, Adversarial traps n=8) alongside the separately disclosed Open Pool (S=336, shared training/generate_dataset.py templates).
Pool Filter:
Official External Public Benchmarks · NVIDIA L4 GPU · Zero Test Contamination
Typed Decisions · JevBench · SQuAD v2.0
| External Leaderboard | Evaluated Split | Headline Score | Calibration & Schema Validity | Latency & Cost (L4 GPU) | Public Rank / Notes (Snapshot 2026-11-11) |
|---|---|---|---|---|---|
| LocalLLaMA/typed-decisions | Official test (400 cases · 2,000 decisions) |
79.00% Accuracy (noul 84.2%, choice 76.8%, score 76.8%) | KL 0.0760 (#1 <=1B, #2 all) · Brier 0.0420 (#1 <=1B, #2 tied all) · ECE 0.1665 | 9.56 ms/dec (47.8 ms/case) · 0 output tokens | Fitted table: Acc #5/27 <=1B (leader 80.25%), #19/51 all (leader 82.65%) |
| Benchmark Heaven / fstandhartinger/jevbench | Public Suite (231 items via /v1/systemone) |
100.0% Easy (48/48) · 68.06% Original · 36.04% Hard | 100.0% Strict Valid · ECE 0.0755 · v1.3 Geo 46.27 | 38.82 ms p50 · $0.00604 / 1k | Public readout (unlisted on JevBench v1.6.1 snapshot 2026-11-11) |
| Stanford rajpurkar/squad_v2 | Full Official validation (11,873 items; 9,198 zero-overlap) |
87.09% Exact Match · 90.00% F1 | HasAns F1 90.36% · NoAns EM 89.64% | 10.30 ms p50 · $0.0017 / 1k | Zero-overlap (N=9,198): 87.01% EM / 89.88% F1 · N=500: 87.00% / 89.36% |
1. NullspanBench Headline Sealed Pool Capability Score · Mean of Intelligence (I) & Calibration (C)
95% Bootstrap CIs (B=1,000) · Headline Sealed Pool (P=164)
2. Pareto Frontier: Capability Against Cost & Speed
Upper-Right is Best (More Capable & Cheaper / Faster)
Capability vs Cost ($ / 1,000 Decisions · Log Scale)
Encoders generate 0 output tokens; autoregressive API decoders pay per generated JSON token.
Capability vs Speed (p50 Latency ms / Decision · Log Scale)
Bubble size follows the official Composite Score; dashed line traces the Pareto-optimal frontier.
3. Composite Score Leaderboard · Four Axes (0–100) Weighted Harmonic Mean
15 Measured Systems
| # ▲ | System ↕ | Architecture / Runtime ↕ | Composite ↕ | Cap. ↕ | Intel. (I) ↕ | Calib. (C) ↕ | Speed (S) ↕ | Cost (K) ↕ | $/1k dec. ↕ | p50 ms ↕ | Null ECE ↕ | Open (S) ↕ | Sealed (P) ↕ |
|---|
4. Compare Two Systems · Multi-Axis Radar Diagnostics
Four Axes · Request Types · Difficulty Tiers · Subject Domains
The Four Score Axes (0–100)
Competence per Request Type (Open / Sealed)
Competence per Difficulty Tier (Open & Sealed)
Competence by Subject Domain / Use Case
5. Competence Heatmap by Request Type, Difficulty Tier & Subject Domain
Greener = Stronger Within Column · Exact Observation Counts (n=500)
| System | Span (Open) | Span (Sealed) | Noul / p_null (Open) | Noul / p_null (Sealed) | Choice (Bool/Enum) | Math & Invariants | Score (Quantiles) | Tier: Easy | Tier: Standard | Tier: Judge | Tier: Hard | SRE Incident | Cloud Invoice | SQuAD v2.0 QA | Contracts (96) | Adversarial |
|---|