Recent interest in quantitative structure-activity relationships (QSARs) to assist regulatory decision-making [1] has increased awareness of the need to quantify predictive confidence.  Research to date has focused on analyzing changes in overall predictivity or on the reliability of quantitative predictions for individual structures [e.g., 2,3]; the reliability of predictive classifications for individual structures has not been as thoroughly explored.  We describe a way to estimate the confidence in predicted classifications for particular structures from the degree of concordance among the individual QSAR models in an ensemble.Artificial neural network ensembles (ANNEs) are composed of multiple networks, each of which has the same number of neurons and structure-based descriptor inputs. Performance is ultimately assessed against a completely external test set chosen at the beginning of the analysis and set aside during training of the individual networks.Combining results from individual networks into a single prediction is well-documented [2].  Averaging and some form of voting are two of the more popular ways to combine outputs from ensemble members, and we have implemented both.We have found that fitting a beta binomial to the distribution of predicted errors from the training pool provides a useful estimate of confidence when making classification predictions for �new� external structures, and that the technique can be extended to ensembles in which the individual network outputs are averaged directly, rather than first being converted to votes.   [1] NL Kruhlak et al., Clin Pharm Therapeutics 2011 , 91, 529-534.[2] B Beck, A Breindl and T Clark, J Chem Inf Comput Sci 2000 , 40, 1046-1051.[3] RD Clark, J Cheminfo 2009 , 1, 11.

Estimating classification confidence for ensemble models

246th National Meeting (2013)

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.