TY - JOUR
T1 - When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting
AU - Wade-Malone, La Keisha
AU - Howerton, Emily
AU - Probert, William J.M.
AU - Runge, Michael C.
AU - Viboud, Cécile
AU - Shea, Katriona
N1 - Publisher Copyright:
© 2024 The Authors
PY - 2024/6
Y1 - 2024/6
N2 - Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.
AB - Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.
UR - http://www.scopus.com/inward/record.url?scp=85192172454&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85192172454&partnerID=8YFLogxK
U2 - 10.1016/j.epidem.2024.100767
DO - 10.1016/j.epidem.2024.100767
M3 - Article
C2 - 38714099
AN - SCOPUS:85192172454
SN - 1755-4365
VL - 47
JO - Epidemics
JF - Epidemics
M1 - 100767
ER -