Skip to content

EpiBenchmark

Authors: Joseph Lemaitre, Emily Przykucki, Yue Liu, and Justin Lessler

What is EpiBenchmark?

EpiBenchmark is a benchmarking framework and challenge library for infectious disease forecasting. Our challenges define common forecasting tasks associated with fixed versions of ground truth data and evaluation rules to facillitate easy and fair forecasting evaluation. That is, models receive scorecards with a variety of metrics and can be compared under the same conditions. EpiBenchmark provides a reproducible way to compare epidemiologic forecasting methods across diseases, targets, and teams.

Why EpiBenchmark?

Evaluation of epidemiologic forecasting models is difficult, and the field is fragmented. Different groups evaluate forecasts on different targets, different versions of observed data, different geographic units, and different scoring rules. As a result, reported performance is often hard to compare directly across papers.

This problem is amplified by the fact that surveillance data are often revised after initial release (backfilling). A model evaluated against the latest revised data may not be directly comparable to a model evaluated against an earlier data version, even if both were forecasting the same target. For probabilistic forecasts, performance also depends on the exact scoring rule and evaluation procedure used.

The progress in a scientific field is easier to measure when there exists a common protocol for evaluation. EpiBenchmark mirrors similar effort in other fields such as WeatherBench and WeatherBench 2, but is adapted to the specific challenges associated with epidemiologic forecasting (such as backfilled surveillance data) and its conventions (probabilistic evaluation, hubverse format). Note that earlier work also argued for a common evaluation protocol in epidemic forecasting (Srivastava et al. 2021).

EpiBenchMark vs Real-time hubs

Real-time collaborative hubs are and remain the gold standard for operational epidemiologic forecasting. Examples include FluSight, RSV Forecast Hub, COVID-19 Forecast Hub, and Flu MetroCast.

But real-time hub evaluation is tied to ongoing submission cycles, changing data, and operational timelines. That makes comparison slower and models harder to re-run, resulting in less reproducibility across studies. EpiBenchmark is intended to provide a faster benchmarking layer around these hubs, while staying compatible with their forecasting setup.

EpiBenchmark vs Hubverse

EpiBenchmark is a thin layer on top of Hubverse. Hubverse defines the data format and shared infrastructure. EpiBenchmark defines the benchmark tasks, frozen ground truth snapshots, scoring procedures, scorecards, and plots. The goal is to add a benchmarking layer to the hubverse that makes evaluation faster to run, easier to reproduce, and easier to compare across models.

EpiBenchmark in practice

EpiBenchmark exposes three workflows:

  • epibench create: facilitate model runs for any reference date with vintaged ground truth data fetched and organized by the tool
  • epibench score: score model forecasts with a WIS (includes over prediction, under prediction, coverage, etc.)
  • epibench plot: create an array of plots to visualize model performance (epibench plot)

Funding

This project was made possible by the Insight Net cooperative agreement CDC-RFA-FT-23-0069 from the CDC's Center for Forecasting and Outbreak Analytics. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention.

EpiBenchmark is being developed at UNC Chapel Hill through ACCIDDA, the Atlantic Coast Center for Infectious Disease Dynamics and Analytics.

Get started!

Overview

  • Overview – understand the scope and usage of EpiBenchmark

Installation Guide

Installation_Longleaf Guide

Attribution

EpiBenchmark relies on the Hubverse structure as a standard for data. Without the Hubverse and its associated tools, EpiBenchmark would not be possible. The scoring component of EpiBenchmark utilizes scoringutils, a CRAN package that facillitates the evaluation of forecasts and is highly-compatible with the Hubverse structure.

Contact

Have a question, comment, or suggestion? Get in touch with the developers by raising an issue on the EpiBenchmark repository.