B5 plan: nowcasting, training vintages and model formulation (not launched)¶
Written 6 October 2026 after B4.refineTop2. Nothing in this plan has been run. It proposes one ~5-hour exploration run followed by a refinement run. Every comparison keeps the B4 protocol unless stated: flu admissions + flu ED targets; finalized next-four-week labels; forward fold (train 2022–23, 2023–24, 2024–25; evaluate 2025–26) and retrospective fold (train 2022–23, 2023–24, 2025–26; evaluate 2024–25); evaluation on each Hub's Wednesday reports with finalized fills where nothing was archived; score = location-relative WIS / Hub ensemble, states/DC 80%, US 20%, lower is better.
Where we are¶
- Leaders. Recipe A (Kinsa, sampled head, trained on synthetic reports corrected by real-pair trees) 0.780 combined native; Recipe B (no covariates, neighbors, 23 direct quantiles, trained with artificial errors + reconstruction) 0.796. A + B ensemble 0.741. Which recipe wins flips by season (2024–25: B 0.697 vs A 0.778; 2025–26: A 0.742 vs B 0.874).
- What the reports do (measured 6 October from the archive, archived cells only, log(report / final) with +1 count or +0.0001 floor):
| Signal, newest week (age 0) | 2023–24 | 2024–25 | 2025–26 |
|---|---|---|---|
| Flu admissions: median bias / mean absolute error / share off by >10% | 0.0 / 5% / 15% | −8% / 16% / 45% | −5% / 14% / 39% |
| Flu ED: same | – | 0% / 8% / 28% | −2% / 7% / 15% |
| Archived share of newest flu-admission week | 56% | 69% | 89% |
Admissions are under-reported by ~5–8% in the newest week and revisions are large in 2024–25/2025–26, small in 2023–24 (different reporting regime). Age-1 errors are already 2–3× smaller. ED revisions are about half the admissions size but not negligible, and ED is never corrected today.
- How training vintages are generated today (
dataset/reporting_error.py): log report/final error windows are taken from one donor season only (the latest training season), matched per location to the episode's local level/growth and calendar (8 nearest windows, kernel-weighted bootstrap), scaled byreporting_strength, redrawn each minibatch. Locations draw independently (no shared reporting shock). 2023–24's archive is never used as a donor. - How evaluation inputs are corrected today (
model/revision_tree.py): a gradient-boosted tree predicts one point log correction per location for the newest 2 admission weeks. No uncertainty is carried into the forecast, except the fixed 50/50 "half" mixture of corrected and uncorrected forecasts. - Clearly settled: flu-only inputs; no COVID inputs; weight decay 0 or 1e-4; fourth-root/sqrt transforms; lookback 12; 2-week correction window; ensembles beat single models; post-hoc spread calibration from the inner model does not improve WIS.
Direction 1 — stochastic nowcasting (highest expected payoff)¶
Idea. Replace the point correction with a distribution of plausible recent histories, forecast from each, and mix the forecasts. The half-mixture (two histories, fixed weights) already helps; the earlier context-nowcast work measured a ~1.1% gain from it. A calibrated distribution is the principled version, and it should mostly fix under-coverage at short horizons.
Formulation. For each issuance and location, sample K = 16–32 corrected histories of the newest 2 admission weeks (and optionally ED): corrected = reported × exp(m + e), where m is the tree's predicted log correction and e is drawn from the tree's cross-fitted residuals in the same report age and similar level/growth (the existing kernel matching can be reused). Feed all K histories through the same fitted forecaster (the history_samples path in training.evaluate already supports this) and pool member draws. Training: draw one history per episode per minibatch from the same model so that training inputs match evaluation inputs (A's success suggests this match matters).
Arms. point (today) vs stochastic K=16 at evaluation only (no retraining; can reuse fitted A/B checkpoints) vs stochastic at evaluation and in training. Residual pools: same-age pooled vs kernel-matched.
Assumptions to state. Tree residuals from cross-fitting represent real nowcast errors in a new season; errors are independent across locations unless we add a shared shock (Direction 2c).
Direction 2 — better training vintages¶
a. Multi-season donor bank with recency weights. Use every archived training season as a donor (2023–24, 2024–25, 2025–26 when permitted), weight donors by season recency (e.g. 1, 0.5, 0.25) and by local phase as today. Today's single donor makes the error bank depend on one season's regime; the measurement above shows regimes differ by season. Expected effect: more robust training, less dependence on which fold you are in.
b. Stronger and stochastic error strength. The sweep's best strength was the top of the range (1.0); the code now allows up to 2. Draw strength per episode from a distribution (e.g. log-uniform 0.5–2) rather than a fixed multiplier, so the model sees mild and severe reporting weeks.
c. Shared reporting shock. Add an issuance-wide component: with probability p, all locations in an episode receive errors from the same donor issuance (the synchronous_log method exists but was never selected for the leaders). Real reporting problems are often system-wide (e.g. a late national upload), and the national forecast is a sum of states.
d. Missing-report patterns. Transport archive missingness (reporting_missingness=1) so training sees cells that were absent at the deadline and filled later, as in evaluation (B3 found finalized fills cover much of the older archive).
e. ED vintages. Apply the same machinery to ED (currently admissions only through revision_signals), given ED newest-week errors of 7–8%.
Direction 3 — nowcaster improvements¶
a. Intermediate-revision supervision. Train the tree on age-a → age-b report pairs as well as report → mature (12-week) pairs. More pairs, faster adaptation when the reporting process changes; recent seasons have many pairs that are not yet 12 weeks mature. b. Issuance-wide revision features. Add features summarizing how older weeks were revised at this issuance across all locations (a shared reporting condition), heavily shrunk. c. ED correction. Correct the newest ED week with its own tree (channel 3), evaluated downstream only. d. Learn correction strength per age. Fit strength separately for age 0 and age 1 on training-season downstream loss rather than one global scalar.
All nowcaster changes are judged by downstream forecast WIS, not point nowcast error (B3: better point accuracy did not mean better forecasts).
Direction 4 — formulation¶
a. One objective for native and log scores. A wins native, B wins log. Add an auxiliary training loss computed on log(1 + admissions) (CRPS of transformed samples or pinball on transformed quantiles) with weight λ ∈ {0.25, 0.5}. Hypothesis: one model closer to both. Small code change in prediction_loss.
b. Loss scaled by forecasting difficulty. Today each location's loss is divided by its training Q95 level; the score divides by the Hub ensemble's WIS. Replace the Q95 scale by a training-only baseline-forecast WIS per location (e.g. persistence model fitted on the training seasons). Aligns training weights with what the score rewards.
c. More epidemic seasons as training examples (cross-pathogen pseudo-flu). Feed RSV and COVID admissions/ED episodes from the modern panel as extra training episodes in the flu channels (scaled per location to flu units by fit-only Q95 ratios), with a source flag and loss weight λ. This is different from B4's finding that COVID inputs hurt: here other epidemics are extra examples of growth/peak/decline, not contemporaneous inputs. Reuses the pseudo-panel builder written for historical ILI.
d. Historical ILI pretraining for the leading MLPs (code written 6 October, training.historical_panel; local 2-epoch smoke test of Recipes A and B passed, 612 historical pseudo-panel episodes). Small expected effect (−3% on shared models in B4); included as one cheap arm, not a priority.
e. Training-season ensemble weights. Use the inner validation forecasts that every run now produces (calibration machinery) to fit per-target/per-horizon member weights for the A+B ensemble on training seasons only. Equal weights are the baseline.
f. Seasonal context features. Season-to-date cumulative burden and weeks since season onset as inputs (causal, from the history window plus a 26-week summary). Targets early-season timing and peak/turning point errors, which a 12-week window cannot see.
Direction 5 — untested knobs (cheap, existing fields)¶
One-at-a-time around A and B: latent 8/32; per-location noise (noise=local); US common noise factor (us_error=shared_factor, targets US undercoverage); members 64/256; batch size 4/16; decoder residual2; multiscale signal features; covariate encoder shared (A); spatial distance (both; −2.5% on ED in the sweep); tree penalty 100/300; weight decay 3e-5; learning rate ±2×; patience 40.
Proposed ~5-hour exploration run¶
Budget: 28 concurrent runs (2 H100 × 6 + 4 L40 × 4), ~15 minutes per two-fold run → about 550 runs in 5 hours, i.e. ~270 configurations × 2 seeds.
| Block | Configurations | Code needed | Priority |
|---|---|---|---|
| Direction 5 one-at-a-time around A and B | ~50 | none | cheap baseline |
| 2a–2e vintage generation, each around A and B, plus pairwise combinations | ~40 | 2a multi-season donor bank + recency weights; 2b random strength (small) | high |
| 1 stochastic nowcast, evaluation-only and train+eval, two residual pools | ~16 | stochastic corrector + history-sample evaluation wiring | highest |
| 3a–3d nowcaster variants (evaluated through A and B) | ~16 | intermediate pairs; issuance features; ED tree | medium |
| 4a log auxiliary loss λ 0.25/0.5 | 8 | small | high |
| 4b difficulty-scaled loss | 4 | medium (baseline WIS scale) | medium |
| 4c cross-pathogen pseudo-flu examples, λ 0.1/0.3 | 8 | small (reuses pseudo-panel builder) | medium |
| 4d historical ILI pretraining, 300/1000 steps | 8 | done | low |
| 4f seasonal context features | 4 | small | medium |
| Random 2–4-change combinations of the above around A and B | ~100 | none extra | to find interactions |
| Confirmation: A, B, B-finalized on seeds 44–48 | 3 × 5 seeds | none | required |
Plus the CPU-only items, done first: ensemble stress test (B + deliberately degraded A; adding B-finalized as a member) and training-season ensemble weights (4e) once inner forecasts exist.
Decision rule for the refinement run. Carry forward any arm that improves the combined native or log-admissions score by ≥1% in both seeds and does not worsen the other by >1%; then cross the survivors on A and B with 5 seeds, and evaluate each resulting A+B ensemble, since the ensemble is what we would submit.
Implementation order (if approved): 2a + 2b (vintage bank) → 1 (stochastic nowcast; largest code item) → 4a, 4c, 4f (small) → 3a–3c → 4b. Each lands with a scientific test only where a silent error would matter (residual sampling alignment, donor-season leakage, pseudo-panel scaling uses fit-only data).
Assumptions. Development seasons are reused (both were used for B4 selection), so small gains will shrink on 2026–27. The 2023–24 archive reflects a different reporting regime and is down-weighted, not excluded, in 2a. Cross-fitted tree residuals stand in for real nowcast errors in Direction 1. Historical ILI and cross-pathogen examples transfer epidemic shape only; they are not observed flu admissions.