Sample-specific error estimates help a scientist decide how much to trust a prediction from a statistical QSAR, QSPR, or ADMET model. In this context, a prediction model is one that outputs the predicted response for a compound, and an error model is one that outputs the expected error in the predicted response for that compound. Sheridan [1] has shown that an effective forest error model can be derived from a forest prediction model using as error model descriptors only the predicted response and the standard deviation of single-tree predictions across all trees in the prediction model forest. (For smaller or less diverse training sets, similarities to closest compounds in the training set can be used instead [2].) Here we describe efforts to generalize the concept to other model types as part of an overall “quality model framework.” One approach is to generate an ensemble of sub-models from which we can obtain a standard deviation to use as an error descriptor. For model types that are not ensembles by nature, we do so using a modified bootstrap aggregation approach. We also discuss attempts to derive as error model descriptors other inexpensive alternatives to the similarities to training compounds. We also report efforts to derive goodness models for prediction models. A goodness model is a classification model that predicts whether a prediction is expected to be “good” or “bad” by some criterion. For a regression prediction model, the criterion is whether the predicted response value is within a specified cutoff of the experimental value. For a classification prediction model, the criterion is simply whether the predicted class is correct. In the presentation, we describe the observed effectiveness of the above approaches for various data sets and model-building algorithms. Among the algorithms investigated are recursive partitioning for tree and forest models, support vector machines (SVM), k-nearest-neighbors (kNN), partial least squares (PLS), genetic function approximation (GFA), and Bayesian classification. [1] R. P. Sheridan, “Using Random Forest to Model the Domain Applicability of Another Random Forest Model,” J. Chem. Inf. Model., 53, 2837 (2013). [2] R. P. Sheridan, “The Relative Importance of Domain Applicability Metrics for Estimating Prediction Errors in QSAR Varies with Training Set Diversity,” J. Chem. Inf. Model., 55, 1098 (2015).

Error models, ensemble models, and goodness models

251st National Meeting (2016)

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.