Profile-QSAR (pQSAR), a massive multi-task machine learning platform that enables transfer learning among thousands of assays, has been applied to over 100 Novartis drug discovery projects. pQSAR overall correlation between prediction and experiment is comparable to 4-concentation IC50s on novel compounds for over 8,000 assays. Until now, however, pQSAR did not estimate confidence intervals for individual predictions. In this study, conformal regression was adapted to profile-QSAR to deliver prediction intervals guaranteed correct at a given confidence level as far as the assumption of exchangeability with the calibration set holds. Finding the right nonconformity measure and calibration set is the key to conformal regression. Conformal analysis is complicated for pQSAR, because it uses a non-random test set of novel compounds which complicates exchangeability, it does not give an ensemble of predictions to estimate non-conformity, and distance to the training set is complicated by the contributions from helper models. Our efforts have been focusing on three questions: (1) which ensemble algorithms that could provide standard deviation as a nonconformity measure generate predictions as accurate as our de facto standard of partial least squared regression (PLSR) at the second step of profile-QSAR; (2) are there residual models that work well as an alternative nonconformity measure; and (3) what is the best strategy in trading-off prediction accuracy, prediction intervals, and computational cost for such a big data problem. We conclude that splitting the “realistic” test set allows for exchangeability, ensemble support vector machines are comparable with PLSR in accuracy and produced fairly good conformal prediction intervals, but at high computational cost, pQSAR residual models that include the profile-QSAR predictions themselves as a descriptor gave good conformal prediction intervals at low computational cost, and weight optimization of the nonconformity score improved the prediction intervals and reduced prediction interval outliers.

Conformal regression for Profile-QSAR: Adding confidence intervals to point predictions

ACS Fall 2020 Virtual Meeting

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.