Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Emily Howerton; Lucie Contamin; Luke C. Mullany; Michelle Qin; Nicholas G. Reich; Samantha Bents; Rebecca K. Borchering; Sung mok Jung; Sara L. Loo; Claire P. Smith; John Levander; Jessica Kerr; J. Espino; Willem G. van Panhuis; Harry Hochheiser; Marta Galanti; Teresa Yamana; Sen Pei; Jeffrey Shaman; Kaitlin Rainwater-Lovett; Matt Kinsey; Kate Tallaksen; Shelby Wilson; Lauren Shin; Joseph C. Lemaitre; Joshua Kaminsky; Juan Dent Hulse; Elizabeth C. Lee; Clifton D. McKee; Alison Hill; Dean Karlen; Matteo Chinazzi; Jessica T. Davis; Kunpeng Mu; Xinyue Xiong; Ana Pastore y Piontti; Alessandro Vespignani; Erik T. Rosenstrom; Julie S. Ivy; Maria E. Mayorga; Julie L. Swann; Guido España; Sean Cavany; Sean Moore; Alex Perkins; Thomas Hladish; Alexander Pillai; Kok Ben Toh; Ira Longini; Shi Chen; Rajib Paul; Daniel Janies; Jean Claude Thill; Anass Bouchnita; Kaiming Bi; Michael Lachmann; Spencer J. Fox; Lauren Ancel Meyers; Ajitesh Srivastava; Przemyslaw Porebski; Srini Venkatramanan; Aniruddha Adiga; Bryan Lewis; Brian Klahn; Joseph Outten; Benjamin Hurt; Jiangzhuo Chen; Henning Mortveit; Amanda Wilson; Madhav Marathe; Stefan Hoops; Parantapa Bhattacharya; Dustin Machi; Betsy L. Cadwell; Jessica M. Healy; Rachel B. Slayton; Michael A. Johansson; Matthew Biggerstaff; Shaun Truelove; Michael C. Runge; Katriona Shea; Cécile Viboud; Justin Lessler

doi:10.1038/s41467-023-42680-x

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Emily Howerton, Lucie Contamin, Luke C. Mullany, Michelle Qin, Nicholas G. Reich, Samantha Bents, Rebecca K. Borchering, Sung mok Jung, Sara L. Loo, Claire P. Smith, John Levander, Jessica Kerr, J. Espino, Willem G. van Panhuis, Harry Hochheiser, Marta Galanti, Teresa Yamana, Sen Pei, Jeffrey Shaman, Kaitlin Rainwater-LovettMatt Kinsey, Kate Tallaksen, Shelby Wilson, Lauren Shin, Joseph C. Lemaitre, Joshua Kaminsky, Juan Dent Hulse, Elizabeth C. Lee, Clifton D. McKee, Alison Hill, Dean Karlen, Matteo Chinazzi, Jessica T. Davis, Kunpeng Mu, Xinyue Xiong, Ana Pastore y Piontti, Alessandro Vespignani, Erik T. Rosenstrom, Julie S. Ivy, Maria E. Mayorga, Julie L. Swann, Guido España, Sean Cavany, Sean Moore, Alex Perkins, Thomas Hladish, Alexander Pillai, Kok Ben Toh, Ira Longini, Shi Chen, Rajib Paul, Daniel Janies, Jean Claude Thill, Anass Bouchnita, Kaiming Bi, Michael Lachmann, Spencer J. Fox, Lauren Ancel Meyers, Ajitesh Srivastava, Przemyslaw Porebski, Srini Venkatramanan, Aniruddha Adiga, Bryan Lewis, Brian Klahn, Joseph Outten, Benjamin Hurt, Jiangzhuo Chen, Henning Mortveit, Amanda Wilson, Madhav Marathe, Stefan Hoops, Parantapa Bhattacharya, Dustin Machi, Betsy L. Cadwell, Jessica M. Healy, Rachel B. Slayton, Michael A. Johansson, Matthew Biggerstaff, Shaun Truelove, Michael C. Runge, Katriona Shea, Cécile Viboud, Justin Lessler

Research output: Contribution to journal › Article › peer-review

19 Scopus citations

Abstract

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Original language	English (US)
Article number	7260
Journal	Nature communications
Volume	14
Issue number	1
DOIs	https://doi.org/10.1038/s41467-023-42680-x
State	Published - Dec 2023

All Science Journal Classification (ASJC) codes

General Chemistry
General Biochemistry, Genetics and Molecular Biology
General Physics and Astronomy

Access to Document

10.1038/s41467-023-42680-x

Cite this

Howerton, E., Contamin, L., Mullany, L. C., Qin, M., Reich, N. G., Bents, S., Borchering, R. K., Jung, S. M., Loo, S. L., Smith, C. P., Levander, J., Kerr, J., Espino, J., van Panhuis, W. G., Hochheiser, H., Galanti, M., Yamana, T., Pei, S., Shaman, J., ... Lessler, J. (2023). Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty. Nature communications, 14(1), Article 7260. https://doi.org/10.1038/s41467-023-42680-x

@article{f7e2ebab00424254b39378f10d3e26d1,

title = "Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty",

abstract = "Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.",

author = "Emily Howerton and Lucie Contamin and Mullany, {Luke C.} and Michelle Qin and Reich, {Nicholas G.} and Samantha Bents and Borchering, {Rebecca K.} and Jung, {Sung mok} and Loo, {Sara L.} and Smith, {Claire P.} and John Levander and Jessica Kerr and J. Espino and {van Panhuis}, {Willem G.} and Harry Hochheiser and Marta Galanti and Teresa Yamana and Sen Pei and Jeffrey Shaman and Kaitlin Rainwater-Lovett and Matt Kinsey and Kate Tallaksen and Shelby Wilson and Lauren Shin and Lemaitre, {Joseph C.} and Joshua Kaminsky and Hulse, {Juan Dent} and Lee, {Elizabeth C.} and McKee, {Clifton D.} and Alison Hill and Dean Karlen and Matteo Chinazzi and Davis, {Jessica T.} and Kunpeng Mu and Xinyue Xiong and {Pastore y Piontti}, Ana and Alessandro Vespignani and Rosenstrom, {Erik T.} and Ivy, {Julie S.} and Mayorga, {Maria E.} and Swann, {Julie L.} and Guido Espa{\~n}a and Sean Cavany and Sean Moore and Alex Perkins and Thomas Hladish and Alexander Pillai and {Ben Toh}, Kok and Ira Longini and Shi Chen and Rajib Paul and Daniel Janies and Thill, {Jean Claude} and Anass Bouchnita and Kaiming Bi and Michael Lachmann and Fox, {Spencer J.} and Meyers, {Lauren Ancel} and Ajitesh Srivastava and Przemyslaw Porebski and Srini Venkatramanan and Aniruddha Adiga and Bryan Lewis and Brian Klahn and Joseph Outten and Benjamin Hurt and Jiangzhuo Chen and Henning Mortveit and Amanda Wilson and Madhav Marathe and Stefan Hoops and Parantapa Bhattacharya and Dustin Machi and Cadwell, {Betsy L.} and Healy, {Jessica M.} and Slayton, {Rachel B.} and Johansson, {Michael A.} and Matthew Biggerstaff and Shaun Truelove and Runge, {Michael C.} and Katriona Shea and C{\'e}cile Viboud and Justin Lessler",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s).",

year = "2023",

month = dec,

doi = "10.1038/s41467-023-42680-x",

language = "English (US)",

volume = "14",

journal = "Nature communications",

issn = "2041-1723",

publisher = "Nature Publishing Group",

number = "1",

}

Howerton, E, Contamin, L, Mullany, LC, Qin, M, Reich, NG, Bents, S, Borchering, RK, Jung, SM, Loo, SL, Smith, CP, Levander, J, Kerr, J, Espino, J, van Panhuis, WG, Hochheiser, H, Galanti, M, Yamana, T, Pei, S, Shaman, J, Rainwater-Lovett, K, Kinsey, M, Tallaksen, K, Wilson, S, Shin, L, Lemaitre, JC, Kaminsky, J, Hulse, JD, Lee, EC, McKee, CD, Hill, A, Karlen, D, Chinazzi, M, Davis, JT, Mu, K, Xiong, X, Pastore y Piontti, A, Vespignani, A, Rosenstrom, ET, Ivy, JS, Mayorga, ME, Swann, JL, España, G, Cavany, S, Moore, S, Perkins, A, Hladish, T, Pillai, A, Ben Toh, K, Longini, I, Chen, S, Paul, R, Janies, D, Thill, JC, Bouchnita, A, Bi, K, Lachmann, M, Fox, SJ, Meyers, LA, Srivastava, A, Porebski, P, Venkatramanan, S, Adiga, A, Lewis, B, Klahn, B, Outten, J, Hurt, B, Chen, J, Mortveit, H, Wilson, A, Marathe, M, Hoops, S, Bhattacharya, P, Machi, D, Cadwell, BL, Healy, JM, Slayton, RB, Johansson, MA, Biggerstaff, M, Truelove, S, Runge, MC, Shea, K, Viboud, C & Lessler, J 2023, 'Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty', Nature communications, vol. 14, no. 1, 7260. https://doi.org/10.1038/s41467-023-42680-x

TY - JOUR

T1 - Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

AU - Howerton, Emily

AU - Contamin, Lucie

AU - Mullany, Luke C.

AU - Qin, Michelle

AU - Reich, Nicholas G.

AU - Bents, Samantha

AU - Borchering, Rebecca K.

AU - Jung, Sung mok

AU - Loo, Sara L.

AU - Smith, Claire P.

AU - Levander, John

AU - Kerr, Jessica

AU - Espino, J.

AU - van Panhuis, Willem G.

AU - Hochheiser, Harry

AU - Galanti, Marta

AU - Yamana, Teresa

AU - Pei, Sen

AU - Shaman, Jeffrey

AU - Rainwater-Lovett, Kaitlin

AU - Kinsey, Matt

AU - Tallaksen, Kate

AU - Wilson, Shelby

AU - Shin, Lauren

AU - Lemaitre, Joseph C.

AU - Kaminsky, Joshua

AU - Hulse, Juan Dent

AU - Lee, Elizabeth C.

AU - McKee, Clifton D.

AU - Hill, Alison

AU - Karlen, Dean

AU - Chinazzi, Matteo

AU - Davis, Jessica T.

AU - Mu, Kunpeng

AU - Xiong, Xinyue

AU - Pastore y Piontti, Ana

AU - Vespignani, Alessandro

AU - Rosenstrom, Erik T.

AU - Ivy, Julie S.

AU - Mayorga, Maria E.

AU - Swann, Julie L.

AU - España, Guido

AU - Cavany, Sean

AU - Moore, Sean

AU - Perkins, Alex

AU - Hladish, Thomas

AU - Pillai, Alexander

AU - Ben Toh, Kok

AU - Longini, Ira

AU - Chen, Shi

AU - Paul, Rajib

AU - Janies, Daniel

AU - Thill, Jean Claude

AU - Bouchnita, Anass

AU - Bi, Kaiming

AU - Lachmann, Michael

AU - Fox, Spencer J.

AU - Meyers, Lauren Ancel

AU - Srivastava, Ajitesh

AU - Porebski, Przemyslaw

AU - Venkatramanan, Srini

AU - Adiga, Aniruddha

AU - Lewis, Bryan

AU - Klahn, Brian

AU - Outten, Joseph

AU - Hurt, Benjamin

AU - Chen, Jiangzhuo

AU - Mortveit, Henning

AU - Wilson, Amanda

AU - Marathe, Madhav

AU - Hoops, Stefan

AU - Bhattacharya, Parantapa

AU - Machi, Dustin

AU - Cadwell, Betsy L.

AU - Healy, Jessica M.

AU - Slayton, Rachel B.

AU - Johansson, Michael A.

AU - Biggerstaff, Matthew

AU - Truelove, Shaun

AU - Runge, Michael C.

AU - Shea, Katriona

AU - Viboud, Cécile

AU - Lessler, Justin

PY - 2023/12

Y1 - 2023/12

N2 - Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

AB - Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

UR - http://www.scopus.com/inward/record.url?scp=85177180750&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85177180750&partnerID=8YFLogxK

U2 - 10.1038/s41467-023-42680-x

DO - 10.1038/s41467-023-42680-x

M3 - Article

C2 - 37985664

AN - SCOPUS:85177180750

SN - 2041-1723

VL - 14

JO - Nature communications

JF - Nature communications

IS - 1

M1 - 7260

ER -

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Cite this