Emily’s EpiBench configs¶
Yes: these ten configs provide explicit forecast calendars and data-vintage rules for influenza, COVID-19 and RSV. They are epibench create inputs, not scoring configs or complete library scorecards.
Running all ten with epibench create --config-path against pinned local hub snapshots produces 345 reference dates and 345 vintage-history files. These files are not used for B0 training or scoring; their data-quality review is deferred.
B1 now consumes historical Hub inputs through the canonical data reader,
including Git snapshots of target files without as_of.
This recovers the same kind of repository state that Influpaint obtains by
checking out a cutoff commit, while preserving the user's worktree and exposing
the history in the explorer. The generated Emily challenge files remain separate;
they are not silently substituted for canonical observations.
What they define¶
Every file uses vintaging: TRUE, vintaging_method: as_of, and vintaging_offset: -3. The offset is in days: a Saturday reference date gets the latest available revisions through the preceding Wednesday. These histories can be used as forecast inputs; scoring future targets still needs a separate truth policy and scoring config.
| File | Target | First origin | Last origin | Created origins |
|---|---|---|---|---|
covid_inchosp_24-25.yaml |
wk inc covid hosp | 2024-11-23 | 2025-08-30 | 41 |
covid_inchosp_25-26.yaml |
wk inc covid hosp | 2025-09-13 | 2026-08-15 | 49 |
covid_propedvisits_25-26.yaml |
wk inc covid prop ed visits | 2025-09-13 | 2026-08-15 | 49 |
flu_inchosp-25-26.yaml |
wk inc flu hosp | 2025-11-15 | 2026-05-30 | 29 |
flu_inchosp_23-24.yaml |
wk inc flu hosp | 2023-09-30 | 2024-05-25 | 35 |
flu_inchosp_24-25.yaml |
wk inc flu hosp | 2024-11-09 | 2025-05-31 | 30 |
flu_propedvisits_25-26.yaml |
wk inc flu prop ed visits | 2025-12-20 | 2026-05-30 | 24 |
rsv_inchosp_24-25.yaml |
wk inc rsv hosp | 2024-12-07 | 2025-05-31 | 26 |
rsv_inchosp_25-26.yaml |
wk inc rsv hosp | 2025-11-01 | 2026-05-30 | 31 |
rsv_propedvisits_25-26.yaml |
wk inc rsv prop ed visits | 2025-11-01 | 2026-05-30 | 31 |
B0 challenges are unversioned custom configs for epibench score, using
finalized evaluation truth. Emily’s created artifacts are separate.
Consequences for the B0 report¶
B0 scoring uses nine frozen evaluation task sets with the hub's 23 quantiles, through total-WIS ranking or the full EpiBench scoring command. Earlier B0 reports used five quantiles and were withdrawn.
Challenge ground truth is not used, and Emily’s origins do not define the scoring calendar. For example, the RSV 2024–25 config creates historical inputs, but that alone does not supply an official-ensemble comparison on the frozen task set. Switching B0 to these provisional inputs would require constructing and fitting a vintage-aware experiment; B0 fits use finalized data.
These configs do not set forecast horizons, locations, quantiles, baseline/ensemble reference, scoring-truth releases, or scorecard functions. Those choices still need a scoring definition. The five quantiles are shared with the bundled EpiBench challenges, while dates and reference models can differ.
Details to reconcile¶
- The provided COVID ED file is 2025–26, not 2024–26; there is no separate 2024–25 COVID ED config in the folder.
covid_inchosp_25-26.yamlsends output to../covid19/prop-ed. Its target is correctly hospital admissions, so this is an output-folder naming inconsistency. Prepared copies use distinct per-config output directories.- EpiBench derives its date-library key from the local hub directory name. Prepared snapshots retain
FluSight-forecast-hub,covid19-forecast-hub, andrsv-forecast-hubso date validation is enabled.
Local artifacts and reproduction¶
Original files under fromEmily/configs/ are unchanged. Only hub and output paths are adapted in data/epibench/emily/configs/. Full source hashes, commands, pinned commits, statuses and task-list paths are recorded in data/epibench/emily/summary.json; each config has a corresponding log. Path-check outputs are under unvalidated_created/; canonical results are under created/.
scripts/prepare_emily_challenges.py prepares snapshots and invokes all ten configs. EpiBench refuses to overwrite an existing challenge output, so choose a fresh output root in that recipe for a repeat. No hub downloads or modifications to Emily’s originals are required.