project_brier_board_tracker

sports-scout-service/scripts/brier_board.py (built 2026-07-28, NOT yet committed). Forward scoreboard so accuracy work is measured, not argued. Retrospective mining of a frozen dataset manufactures false positives — see project_brier_vs_vegas_closed; forward slates are honest.

What it does: scores market / model-raw / model-calibrated on identical props with Murphy decomposition beside each, tracks forecast-mean MAE, appends per-slate rows to scripts/ops/brier_history.jsonl, and --fit refits the calibration map out-of-time writing it ONLY if held-out Brier improves. Carries the modal-line + game_id-scoping guards so the alt-line and merged-game traps can't return.

Baselines (3,113 props, 2026-06-04..07-22): market Brier 0.24640 (reliab 0.00031, resol 0.00211), model raw 0.25758, model calibrated 0.24955, forecast-mean MAE 3.214. Per-stat MAE: points 5.019, rebounds 2.046, assists 1.662.

SHIPPED: Platt calibration map -> scripts/ops/forecast_calibration.json, +2.20% held-out. BUT slope b=0.0349 compresses every forecast into a 0.453-0.504 band and resolution goes 0.00119 -> 0.00000. The Brier gain is shrinkage toward the base rate, not better forecasting — honest to publish, but it CANNOT rank and must not pick or size bets. The script auto-flags this and stores shrinkage_only in the map metadata so no future refit can pass shrinkage off as skill.

KILLED by the gate (all held-out): per-stat calibration (-1.56% vs pooled -1.08%; the full-season per-stat resolution edge on points/assists was an in-sample artifact), per-stat bias correction (-0.72%, bias not stable across time), rate10 x min5 (-1.49%), L5 (-5.04%).

KEY FINDING — the forecast is a rolling average. Shipped forecast MAE 3.0277 vs plain L10 3.0176 on held-out data; indistinguishable, and the fitted blend weight went to 0.00 (preferred pure L10). Note rate10 x min10 is ALGEBRAICALLY IDENTICAL to L10 — not an independent candidate.

STEER BY SKILL CAPTURE, not raw Brier closeness. (base_rate_brier - ours) / (base_rate_brier - market_brier): 0% = no better than guessing, 100% = matches the price. We are at 2.8%. Raw closeness is a worthless target here — guessing the base rate is only 1.32% behind the market and we are 1.28% behind, so "very close to Vegas on Brier" is satisfied by predicting nothing. A significance target (|z|<2) is equally bad: it gets HARDER as slates accumulate (z=-2.18 on n=1,291 became z=-4.03 on n=3,113), so more data would "fail" a goal that was met on less. Skill capture is sample-size independent and cannot be gamed by predicting nothing. The scoreboard prints it.

METHOD FIX — use PAIRED tests to compare two models. I spent most of a session rejecting candidates against an UNPAIRED floor (2 sd of the error distribution / sqrt n). That is the bar for "better than chance", not for "model A vs model B on the same props": prop-to-prop variance dominates it and cancels entirely in the paired difference, making it ~9x too conservative. Unpaired MAE floor 0.1727 vs paired se 0.0193 on the same data. paired_z / paired_z_mae are now in the script and the scoreboard prints paired z against the market. Re-testing everything paired did NOT rescue any candidate — it sharpened the verdict.

Verdict with correct statistics (held out, n=1221): model calibrated vs market z=-2.23, model raw z=-2.59 — we are genuinely BEHIND, not tied. L10 median z=-2.41 (correctly rejected). L10 mean z=+0.44 and EB z=+0.48 are true ties with the shipped forecast.

THE GAP, in units: the posted line is the market's own median projection and can be scored as one. Market line MAE 3.112 vs our forecast 3.200 vs L10 3.202 (n=3,038); paired diff +0.0880, se 0.0193, z=+4.56. Vegas projects ~0.09 stat-units better per prop (points +3.54%, rebounds +1.22%, assists +1.21%). Closing THAT is what a better Brier requires; there is no route around it.

WIN — per-stat mean/median estimator (MAE, not Brier). The line is a MEDIAN (books set 50% over); our forecast reports a MEAN. Counting stats are right-skewed so mean > median, and MAE rewards medians — part of the gap was an estimator mismatch we created. The card already ships the full distribution, so the median is free from outcome_counts. Estimator picked on train (points->MEAN, rebounds->MEDIAN, assists->MEDIAN), held out: MAE 3.0459 -> 3.0279, paired z=+2.00 (marginal, p.046 — reconfirm forward). Gap to the market line collapses from z=+4.56 to z=+1.64, no longer significant.

Per stat on test: rebounds ours 1.839 vs line 1.913 (we BEAT the line, z=-1.92), assists 1.647 vs 1.639 (parity, z=+0.21), points 4.833 vs 4.646 (z=+2.70, the only remaining significant deficit). The market's entire projection advantage is POINTS. Does NOT affect Brier — Brier runs off the distribution's P(>line), which this leaves untouched.

WIN #2 — de-lucked points (projection v2), from a 9-family ultracode sweep (2026-07-28). exp_pts = 2*(fga10-tpa10)p2 + 3tpa10p3 + fta10ft with efficiencies from ALL pre-date games shrunk toward league by k=100 pseudo-attempts, blended 0.7exp_pts + 0.3fc_mean; rebounds/assists stay on fc_median. Params W=10/k=100/alpha=0.7 selected by TRAIN-ONLY grid (adversarial verifier objected to variant selection on test; re-running the whole choice train-only landed on the SAME model, dissolving the objection). Held out: MAE 3.0459 -> 2.9853 (z=+2.76 vs permm), points 4.833 -> 4.730, and projection parity with the market line overall (z=-0.37; points deficit cut from z=-2.70 to -1.33). Recent shot volume is signal; recent shooting % is luck. Params FROZEN in brier_board.py (V2_W/V2_K/V2_ALPHA) — every forward slate is its true out-of-sample test; do not refit casually. Caveat: the held-out window has been reused all session; treat forward slates as the real confirmation.

Sweep negatives (all train-only fits, all clean): opponent/pace/home/ rest correction (1-SE rule zeroes everything; when it fires it hurts); multi-window blend + EB (collapses onto permm — fc mean/median already summarize the log windows); minutes-conditional (fitted minutes model has ~2% skill that does NOT transfer to prop MAE); L1 feature regression (all slopes zeroed); skewfix amean+bmedian+c*L10 (negative); 6-member stacked ensemble (z=+1.77, short of bar, subsumed by deluck). Brier families: distribution->Platt and direct logistic both collapse to base-rate (resolution 0, fitted slopes on (proj-line)/sd NEGATIVE for points/ rebounds) — re-confirms project_brier_vs_vegas_closed AUC 0.506.

Power limit: even paired, one season bounds what is provable. Floor scales 1/sqrt(n), so the history file accumulating forward is the fix.

How to apply: run before/after any projection or model change; trust the gate over a plausible in-sample number. Assists is the only stat with a positive calibration slope (+0.408) = the only place the model knows something. Related: project_projections_engine_state, project_scout_model_under_bias.