TY - JOUR
T1 - Defining predictive maturity for validated numerical simulations
AU - Hemez, François
AU - Atamturktur, H. Sezer
AU - Unal, Cetin
N1 - Funding Information:
This work is performed under the auspices of the Validation and Uncertainty project of the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program at Los Alamos National Laboratory (LANL). at Los Alamos National Laboratory (LANL). The first two authors are grateful to Cetin Unal, M&S project leader, for his support and technical leadership. LANL is operated by the Los Alamos National Security, LLC for the National Nuclear Security Administration of the US Department of Energy under Contract DE-AC52-06NA25396.
PY - 2010/4
Y1 - 2010/4
N2 - The increasing reliance on computer simulations in decision-making motivates the need to formulate a commonly accepted definition for "predictive maturity." The concept of predictive maturity involves quantitative metrics that could prove useful while allocating resources for physical testing and code development. Such metrics should be able to track progress (or lack thereof) as additional knowledge becomes available and is integrated into the simulations for example, through the addition of new experimental datasets during model calibration, and/or through the implementation of better physics models in the codes. This publication contributes to a discussion of attributes that a metric of predictive maturity should exhibit. It is contended that the assessment of predictive maturity must go beyond the goodness-of-fit of the model to the available test data. We firmly believe that predictive maturity must also consider the "knobs," or ancillary variables, used to calibrate the model and the degree to which physical experiments cover the domain of applicability. The emphasis herein is placed on translating the proposed attributes into mathematical properties, such as the degree of regularity and asymptotic limits of the maturity function. Altogether these mathematical properties define a set of constraints that the predictive maturity function must satisfy. Based on these constraints, we propose a Predictive Maturity Index (PMI). Physical datasets are used to illustrate how the PMI quantifies the maturity of the non-linear, Preston-Tonks-Wallace model of plastic deformation applied to beryllium, a light-weight, high-strength metal. The question "does collecting additional data improve predictive power?" is answered by computing the PMI iteratively as additional experimental datasets become available. The results obtained reflect that coverage of the validation domain is as important to predictive maturity as goodness-of-fit. The example treated also indicates that the stabilization of predictive maturity can be observed, provided that enough physical experiments are available.
AB - The increasing reliance on computer simulations in decision-making motivates the need to formulate a commonly accepted definition for "predictive maturity." The concept of predictive maturity involves quantitative metrics that could prove useful while allocating resources for physical testing and code development. Such metrics should be able to track progress (or lack thereof) as additional knowledge becomes available and is integrated into the simulations for example, through the addition of new experimental datasets during model calibration, and/or through the implementation of better physics models in the codes. This publication contributes to a discussion of attributes that a metric of predictive maturity should exhibit. It is contended that the assessment of predictive maturity must go beyond the goodness-of-fit of the model to the available test data. We firmly believe that predictive maturity must also consider the "knobs," or ancillary variables, used to calibrate the model and the degree to which physical experiments cover the domain of applicability. The emphasis herein is placed on translating the proposed attributes into mathematical properties, such as the degree of regularity and asymptotic limits of the maturity function. Altogether these mathematical properties define a set of constraints that the predictive maturity function must satisfy. Based on these constraints, we propose a Predictive Maturity Index (PMI). Physical datasets are used to illustrate how the PMI quantifies the maturity of the non-linear, Preston-Tonks-Wallace model of plastic deformation applied to beryllium, a light-weight, high-strength metal. The question "does collecting additional data improve predictive power?" is answered by computing the PMI iteratively as additional experimental datasets become available. The results obtained reflect that coverage of the validation domain is as important to predictive maturity as goodness-of-fit. The example treated also indicates that the stabilization of predictive maturity can be observed, provided that enough physical experiments are available.
UR - http://www.scopus.com/inward/record.url?scp=77049125900&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77049125900&partnerID=8YFLogxK
U2 - 10.1016/j.compstruc.2010.01.005
DO - 10.1016/j.compstruc.2010.01.005
M3 - Article
AN - SCOPUS:77049125900
SN - 0045-7949
VL - 88
SP - 497
EP - 505
JO - Computers and Structures
JF - Computers and Structures
IS - 7-8
ER -