nullspan NullspanBench v3.2 · JevBench Methodology S+P (500 Scored Observations)

NullspanBench — Measured Decision & Extraction Leaderboard

Following the Benchmark Heaven (JevBench) evaluation protocol with strictly measured systems only (measured: true + cryptographic predictions_sha256): comparing 15 empirically measured systems across Intelligence (I), Probability Calibration (C), Speed (S), and Cost (K) on the Headline Sealed Pool (P=164: Stanford SQuAD v2.0 n=60, Multi-Numeric Legal/Commercial Contracts n=96, Adversarial traps n=8) alongside the separately disclosed Open Pool (S=336, shared training/generate_dataset.py templates).

Pool Filter:
Official External Public Benchmarks · NVIDIA L4 GPU · Zero Test Contamination Typed Decisions · JevBench · SQuAD v2.0
External Leaderboard Evaluated Split Headline Score Calibration & Schema Validity Latency & Cost (L4 GPU) Public Rank / Notes (Snapshot 2026-11-11)
LocalLLaMA/typed-decisions Official test (400 cases · 2,000 decisions) 79.00% Accuracy (noul 84.2%, choice 76.8%, score 76.8%) KL 0.0760 (#1 <=1B, #2 all) · Brier 0.0420 (#1 <=1B, #2 tied all) · ECE 0.1665 9.56 ms/dec (47.8 ms/case) · 0 output tokens Fitted table: Acc #5/27 <=1B (leader 80.25%), #19/51 all (leader 82.65%)
Benchmark Heaven / fstandhartinger/jevbench Public Suite (231 items via /v1/systemone) 100.0% Easy (48/48) · 68.06% Original · 36.04% Hard 100.0% Strict Valid · ECE 0.0755 · v1.3 Geo 46.27 38.82 ms p50 · $0.00604 / 1k Public readout (unlisted on JevBench v1.6.1 snapshot 2026-11-11)
Stanford rajpurkar/squad_v2 Full Official validation (11,873 items; 9,198 zero-overlap) 87.09% Exact Match · 90.00% F1 HasAns F1 90.36% · NoAns EM 89.64% 10.30 ms p50 · $0.0017 / 1k Zero-overlap (N=9,198): 87.01% EM / 89.88% F1 · N=500: 87.00% / 89.36%
1. NullspanBench Headline Sealed Pool Capability Score · Mean of Intelligence (I) & Calibration (C) 95% Bootstrap CIs (B=1,000) · Headline Sealed Pool (P=164)
2. Pareto Frontier: Capability Against Cost & Speed Upper-Right is Best (More Capable & Cheaper / Faster)

Capability vs Cost ($ / 1,000 Decisions · Log Scale)

Encoders generate 0 output tokens; autoregressive API decoders pay per generated JSON token.

Capability vs Speed (p50 Latency ms / Decision · Log Scale)

Bubble size follows the official Composite Score; dashed line traces the Pareto-optimal frontier.

3. Composite Score Leaderboard · Four Axes (0–100) Weighted Harmonic Mean 15 Measured Systems
Weights:
Intelligence (I): 25%
Calibration (C): 25%
Speed (S): 25%
Cost (K): 25%
# ▲ System ↕ Architecture / Runtime ↕ Composite ↕ Cap. ↕ Intel. (I) ↕ Calib. (C) ↕ Speed (S) ↕ Cost (K) ↕ $/1k dec. ↕ p50 ms ↕ Null ECE ↕ Open (S) ↕ Sealed (P) ↕
4. Compare Two Systems · Multi-Axis Radar Diagnostics Four Axes · Request Types · Difficulty Tiers · Subject Domains
■ System A:
● System B:

The Four Score Axes (0–100)

Competence per Request Type (Open / Sealed)

Competence per Difficulty Tier (Open & Sealed)

Competence by Subject Domain / Use Case

5. Competence Heatmap by Request Type, Difficulty Tier & Subject Domain Greener = Stronger Within Column · Exact Observation Counts (n=500)
System Span (Open) Span (Sealed) Noul / p_null (Open) Noul / p_null (Sealed) Choice (Bool/Enum) Math & Invariants Score (Quantiles) Tier: Easy Tier: Standard Tier: Judge Tier: Hard SRE Incident Cloud Invoice SQuAD v2.0 QA Contracts (96) Adversarial