publication . Article . 2019

An open challenge to advance probabilistic forecasting for dengue epidemics.

Johansson, Michael A.; Apfeldorf, Karyn M.; Dobson, Scott; Devita, Jason; Buczak, Anna L.; Baugher, Benjamin; Moniz, Linda J.; Bagley, Thomas; Babin, Steven M.; Guven, Erhan; ...
View all 82 authors
Open Access
  • Published: 26 Nov 2019
  • Publisher: eScholarship, University of California
Abstract
A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecasts would be most valuable. A comparison of modeling approaches revealed that average forecast skill was lower for models including biologically meaningful data and mechanisms and that both multimodel and multiteam ensemble forecasts consistently outperformed individual model forecasts. Leveraging these insights, data, and the forecasting framework will be critical to improve forecast skill and the application of forecasts in real time for epidemic preparedness and response. Moreover, key components of this project—integration with public health needs, a common forecasting framework, shared and standardized data, and open participation—can help advance infectious disease forecasting beyond dengue.
Significance Forecasts routinely provide critical information for dangerous weather events but not yet for epidemics. Researchers develop computational models that can be used for infectious disease forecasting, but forecasts have not been broadly compared or tested. We collaboratively compared forecasts from 16 teams for 8 y of dengue epidemics in Peru and Puerto Rico. The comparison highlighted components that forecasts did well (e.g., situational awareness late in the season) and those that need more work (e.g., early season forecasts). It also identified key facets to improve forecasts, including using multiple model ensemble approaches to improve overall forecast skill. Future infectious disease forecasting work can build on these findings and this framework to improve the skill and utility of forecasts.
Persistent Identifiers

doi: 10.1073/pnas.1909865116

pmc: PMC6883829

pmid: 31712420

Sustainable Development Goals (SDG)
3. Good health
Subjects
Medical Subject Headings: educationhealth care economics and organizationssocial sciencespopulation characteristics
ACM Computing Classification System: ComputingMilieux_MISCELLANEOUS
free text keywords: Humans, Dengue, Epidemiologic Methods, Incidence, Models, Statistical, Disease Outbreaks, Puerto Rico, Peru, Epidemics, epidemic, forecast, [SDV.MHEP.ME]Life Sciences [q-bio]/Human health and pathology/Emerging diseases, Biological Sciences, Medical Sciences, forecast, dengue, epidemic, Peru, Puerto Rico, Multidisciplinary, Preparedness, Econometrics, High skill, Public health, medicine.medical_specialty, medicine, Probabilistic logic, Situation awareness, Forecast skill, Probabilistic forecasting, Dengue fever, medicine.disease, Computer science
Related Organizations
View more
Funded by
NIH| Modeling contact investigation and rapid response
Project
Modeling contact investigation and rapid response
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5U01GM087728-04
  • Funding stream: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
, NIH| Inference for interacting pathogens with mechanistic and phenomenological models
Project
Inference for interacting pathogens with mechanistic and phenomenological models
  • Funder: National Institutes of Health (NIH)
  • Project Code: 1R21AI115173-01
  • Funding stream: NATIONAL INSTITUTE OF ALLERGY AND INFECTIOUS DISEASES
, NIH| Methods for Reducing Spatial Uncertainty and Bias in Disease Surveillance
Project
Methods for Reducing Spatial Uncertainty and Bias in Disease Surveillance
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5R01AI102939-05
  • Funding stream: NATIONAL INSTITUTE OF ALLERGY AND INFECTIOUS DISEASES
, NSF| Advancing Theory and Computation in Statistical Learning Problems
Project
Advancing Theory and Computation in Statistical Learning Problems
  • Funder: National Science Foundation (NSF)
  • Project Code: 1309174
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Mathematical Sciences
, NIH| Development and Dissemination of Operational Real-Time Respiratory Virus Forecast
Project
Development and Dissemination of Operational Real-Time Respiratory Virus Forecast
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5U01GM110748-05
  • Funding stream: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
View more
Download fromView all 8 versions
Open Access
eScholarship - University of California
Article . 2019
Providers: eScholarship - University of California
Open Access
LSHTM Research Online
Article . 2019
Providers: CORE (RIOXX-UK Aggregator)
Open Access
Proceedings of the National Academy of Sciences
Article . 2019
Providers: Datacite; Crossref
Open Access
Proceedings of the National Academy of Sciences
Article
Providers: UnpayWall
View more
38 references, page 1 of 3

1 Bhatt S., The global distribution and burden of dengue. Nature 496, 504–507 (2013).23563266 [OpenAIRE] [PubMed]

2 Johansson M. A., Cummings D. A., Glass G. E., Multiyear climate variability an d dengue–El Niño southern oscillation, weather, and dengue incidence in Puerto Rico, Mexico, and Thailand: A longitudinal data analysis. PLoS Med.6, e1000168 (2009).19918363 [OpenAIRE] [PubMed]

3 van Panhuis W. G., Region-wide synchrony and traveling waves of dengue across eight countries in Southeast Asia. Proc. Natl. Acad. Sci. U.S.A.112, 13069–13074 (2015).26438851 [OpenAIRE] [PubMed]

4 Constenla D., Garcia C., Lefcourt N., Assessing the economics of dengue: Results from a systematic review of the literature and expert survey. Pharmacoeconomics 33, 1107–1135 (2015).26048354 [PubMed]

5 Shepard D. S., Undurraga E. A., Halasa Y. A., Stanaway J. D., The global economic burden of dengue: A systematic analysis. Lancet Infect. Dis.16, 935–941 (2016).27091092 [PubMed]

6 Johansson M. A., Reich N. G., Hota A., Brownstein J. S., Santillana M., Evaluating the performance of infectious disease forecasts: A comparison of climate-driven and seasonal dengue forecasts for Mexico. Sci. Rep.6, 33707 (2016).27665707 [OpenAIRE] [PubMed]

7 Johansson M. A., Hombach J., Cummings D. A., Models of the impact of dengue vaccines: A review of current research and potential approaches. Vaccine 29, 5860–5868 (2011).21699949 [OpenAIRE] [PubMed]

8 Reiner R. C.Jr, A systematic review of mathematical models of mosquito-borne pathogen transmission: 1970-2010. J. R. Soc. Interface 10, 20120921 (2013).23407571 [OpenAIRE] [PubMed]

9 Lowe R., Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil. eLife 5, e11285 (2016).26910315 [OpenAIRE] [PubMed]

10 Ferguson N., Anderson R., Gupta S., The effect of antibody-dependent enhancement on the transmission dynamics and persistence of multiple-strain pathogens. Proc. Natl. Acad. Sci. U.S.A.96, 790–794 (1999).9892712 [OpenAIRE] [PubMed]

11 Cummings D. A., Schwartz I. B., Billings L., Shaw L. B., Burke D. S., Dynamic effects of antibody-dependent enhancement on the fitness of viruses. Proc. Natl. Acad. Sci. U.S.A.102, 15259–15264 (2005).16217017 [OpenAIRE] [PubMed]

12 Wearing H. J., Rohani P., Ecological and immunological determinants of dengue epidemics. Proc. Natl. Acad. Sci. U.S.A.103, 11802–11807 (2006).16868086 [OpenAIRE] [PubMed]

13 Adams B., Cross-protective immunity can account for the alternating epidemic pattern of dengue virus serotypes circulating in Bangkok. Proc. Natl. Acad. Sci. U.S.A.103, 14234–14239 (2006).16966609 [OpenAIRE] [PubMed]

14 Lourenço J., Recker M., Natural, persistent oscillations in a spatial multi-strain disease system with application to dengue. PLoS Comput. Biol.9, e1003308 (2013).24204241 [OpenAIRE] [PubMed]

15 Pandemic Prediction and Forecasting Science and Technology Working Group, Towards epidemic prediction: Federal efforts and opportunities in outbreak modeling. https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/NSTC/towards_epidemic_prediction-federal_efforts_and_opportunities.pdf. Accessed 22 October 2019.

38 references, page 1 of 3
1 research outcomes, page 1 of 1
Abstract
A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecasts would be most valuable. A comparison of modeling approaches revealed that average forecast skill was lower for models including biologically meaningful data and mechanisms and that both multimodel and multiteam ensemble forecasts consistently outperformed individual model forecasts. Leveraging these insights, data, and the forecasting framework will be critical to improve forecast skill and the application of forecasts in real time for epidemic preparedness and response. Moreover, key components of this project—integration with public health needs, a common forecasting framework, shared and standardized data, and open participation—can help advance infectious disease forecasting beyond dengue.
Significance Forecasts routinely provide critical information for dangerous weather events but not yet for epidemics. Researchers develop computational models that can be used for infectious disease forecasting, but forecasts have not been broadly compared or tested. We collaboratively compared forecasts from 16 teams for 8 y of dengue epidemics in Peru and Puerto Rico. The comparison highlighted components that forecasts did well (e.g., situational awareness late in the season) and those that need more work (e.g., early season forecasts). It also identified key facets to improve forecasts, including using multiple model ensemble approaches to improve overall forecast skill. Future infectious disease forecasting work can build on these findings and this framework to improve the skill and utility of forecasts.
Persistent Identifiers

doi: 10.1073/pnas.1909865116

pmc: PMC6883829

pmid: 31712420

Sustainable Development Goals (SDG)
3. Good health
Subjects
Medical Subject Headings: educationhealth care economics and organizationssocial sciencespopulation characteristics
ACM Computing Classification System: ComputingMilieux_MISCELLANEOUS
free text keywords: Humans, Dengue, Epidemiologic Methods, Incidence, Models, Statistical, Disease Outbreaks, Puerto Rico, Peru, Epidemics, epidemic, forecast, [SDV.MHEP.ME]Life Sciences [q-bio]/Human health and pathology/Emerging diseases, Biological Sciences, Medical Sciences, forecast, dengue, epidemic, Peru, Puerto Rico, Multidisciplinary, Preparedness, Econometrics, High skill, Public health, medicine.medical_specialty, medicine, Probabilistic logic, Situation awareness, Forecast skill, Probabilistic forecasting, Dengue fever, medicine.disease, Computer science
Related Organizations
View more
Funded by
NIH| Modeling contact investigation and rapid response
Project
Modeling contact investigation and rapid response
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5U01GM087728-04
  • Funding stream: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
, NIH| Inference for interacting pathogens with mechanistic and phenomenological models
Project
Inference for interacting pathogens with mechanistic and phenomenological models
  • Funder: National Institutes of Health (NIH)
  • Project Code: 1R21AI115173-01
  • Funding stream: NATIONAL INSTITUTE OF ALLERGY AND INFECTIOUS DISEASES
, NIH| Methods for Reducing Spatial Uncertainty and Bias in Disease Surveillance
Project
Methods for Reducing Spatial Uncertainty and Bias in Disease Surveillance
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5R01AI102939-05
  • Funding stream: NATIONAL INSTITUTE OF ALLERGY AND INFECTIOUS DISEASES
, NSF| Advancing Theory and Computation in Statistical Learning Problems
Project
Advancing Theory and Computation in Statistical Learning Problems
  • Funder: National Science Foundation (NSF)
  • Project Code: 1309174
  • Funding stream: Directorate for Mathematical & Physical Sciences | Division of Mathematical Sciences
, NIH| Development and Dissemination of Operational Real-Time Respiratory Virus Forecast
Project
Development and Dissemination of Operational Real-Time Respiratory Virus Forecast
  • Funder: National Institutes of Health (NIH)
  • Project Code: 5U01GM110748-05
  • Funding stream: NATIONAL INSTITUTE OF GENERAL MEDICAL SCIENCES
View more
Download fromView all 8 versions
Open Access
eScholarship - University of California
Article . 2019
Providers: eScholarship - University of California
Open Access
LSHTM Research Online
Article . 2019
Providers: CORE (RIOXX-UK Aggregator)
Open Access
Proceedings of the National Academy of Sciences
Article . 2019
Providers: Datacite; Crossref
Open Access
Proceedings of the National Academy of Sciences
Article
Providers: UnpayWall
View more
38 references, page 1 of 3

1 Bhatt S., The global distribution and burden of dengue. Nature 496, 504–507 (2013).23563266 [OpenAIRE] [PubMed]

2 Johansson M. A., Cummings D. A., Glass G. E., Multiyear climate variability an d dengue–El Niño southern oscillation, weather, and dengue incidence in Puerto Rico, Mexico, and Thailand: A longitudinal data analysis. PLoS Med.6, e1000168 (2009).19918363 [OpenAIRE] [PubMed]

3 van Panhuis W. G., Region-wide synchrony and traveling waves of dengue across eight countries in Southeast Asia. Proc. Natl. Acad. Sci. U.S.A.112, 13069–13074 (2015).26438851 [OpenAIRE] [PubMed]

4 Constenla D., Garcia C., Lefcourt N., Assessing the economics of dengue: Results from a systematic review of the literature and expert survey. Pharmacoeconomics 33, 1107–1135 (2015).26048354 [PubMed]

5 Shepard D. S., Undurraga E. A., Halasa Y. A., Stanaway J. D., The global economic burden of dengue: A systematic analysis. Lancet Infect. Dis.16, 935–941 (2016).27091092 [PubMed]

6 Johansson M. A., Reich N. G., Hota A., Brownstein J. S., Santillana M., Evaluating the performance of infectious disease forecasts: A comparison of climate-driven and seasonal dengue forecasts for Mexico. Sci. Rep.6, 33707 (2016).27665707 [OpenAIRE] [PubMed]

7 Johansson M. A., Hombach J., Cummings D. A., Models of the impact of dengue vaccines: A review of current research and potential approaches. Vaccine 29, 5860–5868 (2011).21699949 [OpenAIRE] [PubMed]

8 Reiner R. C.Jr, A systematic review of mathematical models of mosquito-borne pathogen transmission: 1970-2010. J. R. Soc. Interface 10, 20120921 (2013).23407571 [OpenAIRE] [PubMed]

9 Lowe R., Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil. eLife 5, e11285 (2016).26910315 [OpenAIRE] [PubMed]

10 Ferguson N., Anderson R., Gupta S., The effect of antibody-dependent enhancement on the transmission dynamics and persistence of multiple-strain pathogens. Proc. Natl. Acad. Sci. U.S.A.96, 790–794 (1999).9892712 [OpenAIRE] [PubMed]

11 Cummings D. A., Schwartz I. B., Billings L., Shaw L. B., Burke D. S., Dynamic effects of antibody-dependent enhancement on the fitness of viruses. Proc. Natl. Acad. Sci. U.S.A.102, 15259–15264 (2005).16217017 [OpenAIRE] [PubMed]

12 Wearing H. J., Rohani P., Ecological and immunological determinants of dengue epidemics. Proc. Natl. Acad. Sci. U.S.A.103, 11802–11807 (2006).16868086 [OpenAIRE] [PubMed]

13 Adams B., Cross-protective immunity can account for the alternating epidemic pattern of dengue virus serotypes circulating in Bangkok. Proc. Natl. Acad. Sci. U.S.A.103, 14234–14239 (2006).16966609 [OpenAIRE] [PubMed]

14 Lourenço J., Recker M., Natural, persistent oscillations in a spatial multi-strain disease system with application to dengue. PLoS Comput. Biol.9, e1003308 (2013).24204241 [OpenAIRE] [PubMed]

15 Pandemic Prediction and Forecasting Science and Technology Working Group, Towards epidemic prediction: Federal efforts and opportunities in outbreak modeling. https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/NSTC/towards_epidemic_prediction-federal_efforts_and_opportunities.pdf. Accessed 22 October 2019.

38 references, page 1 of 3
1 research outcomes, page 1 of 1
graphPowered by OpenAIRE Research Graph
Any information missing or wrong?Report an Issue