Significant evidence has mounted in the past several years on the applied effectiveness of the Sandia-developed Discrete Direct (DD) model calibration and uncertainty propagation approach for calibrating models of stochastically varying systems to data from very few replicate tests (≤6) and propagating the corresponding variability in calibration parameters to model predictions of tail percentiles of response or small tail probabilities such as 10-4. Reliably conservative estimation of these types of statistics (i.e., don’t under-estimate the 99th percentile or the 10-4 probability) is desirable for design, performance, and safety related risk analysis. At the same time, efficient (not overly conservative) estimates are desired.The DD calibration-UQ methodology has been specifically developed for these types of problems. The approach calibrates to each test’s data separately and then propagates just the N calibration parameter sets from N tests, instead of a constructed probability density (PD) or joint probability density (JPD) of variation of the calibration parameter(s). The propagated N parameter sets yield N samples or realizations of response quantities (scalar or functional) that directly correspond to system variability in the N tests (after any differences in test conditions such as initial and boundary conditions are normalized out). The N samples of response are processed with specialized sparse-sample 1D UQ (uncertainty quantification) methods that, with reasonably high statistical confidence, yield conservative and efficient estimates as established by testing on 16 diverse and challenging probability distributions, including multi-modal and highly skewed one-tailed distributions.The DD approach is objectively simpler and usually less expensive than Bayesian and other calibration-UQ approaches. For example, DD treats everything the Bayesian approach does in terms of types and sources of uncertainty in the calibration and uncertainty propagation problem, but does not involve the complications of forming likelihood functions; constructing surrogate models for calibration and separately for propagation in the application-level problem; or proposing prior distributions and updating them using Bayes rule and associated machinery. Furthermore, the DD approach has a more extensive experimental data UQ treatment and more cleanly and less expensively incorporates data from new replicate tests.There are also indications that the DD approach is more trustworthy than Bayesian approaches for providing conservative and efficient estimates of tail percentiles and probabilities, but this is more of an open question than the cost and complexity questions. When sparse replicate data are involved, tail probability predictions from Bayesian approaches will be sensitive to the proposed prior distributions. If the Bayesian formulation involves uncertain hyperparameters of proposed variability distributions for the calibration variable(s), it is difficult to see where highly relevant prior knowledge for uncertainty distributions for the hyperparameters might come from in general cases. If the hyperparameter priors are not sufficiently close to the true but unknown values, or if the proposed variability distribution model forms themselves are not correct, then eventual predictions of tail statistics will be substantially incorrect and may not enable highly reliable and efficient uncertainty estimation like the DD method does.Moreover, for models with more than a single calibration parameter, imagine the futility of trying to construct, from just four tests/data-points in a 3D calibration parameter space for example, a suitable 3D joint probability density for the calibration parameters, including correlations between the calibration variables, as enacted through appropriate prior and posterior distributions and correlations of the hyperparameters. Multi-dimensional JPD construction and propagation problems in other calibration-UQ methods are much simpler and less expensive point propagations (only N of them) and 1D UQ post-processing with the DD approach.The many decisions in Bayesian approaches can lead to high analyst-to-analyst variability. Use a discrepancy term or don’t? What type of parametric and/or non-parametric distribution models/formulations will be used? What surrogate modeling and sampling approaches will be used? For these and other reasons to be discussed, &nbsp;it is important to investigate the relative trustworthiness of the DD and Bayesian calibration-UQ approaches on representative test problems. Associated ideas and progress will be presented.

Is Discrete-Direct Model Calibration and Uncertainty Propagation More Trustworthy for Estimating Tail Percentiles and Probabilities Than Bayesian Approaches?

Paper Proceedings: https://asmedigitalcollection.asme.org/VVS/VVS2021/volume/84782

ASME 2021 Verification and Validation (VVS2021)

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.