Cheminformaticians often have to assemble bioactivity data from different studies. However, data aggregation from bioassays using different experimental setups over different chemical series can lead to problems for ligand-based modelling. In this work, we study the impact of such heterogeneity from a predictive modeling point of view. Using four different multiple-source datasets (inhibition of BCRP, P-glycoprotein and hERG, mutagenicity), we compare the estimated performance of models evaluated by repeated random splits against the performance of models evaluated using a 'leave-sources-out' evaluation method (LSO). Leave-sources-out is performed by randomly selecting subsets of data sources (studies) to use as external evaluation sets. This setting actually mirrors real-life use case of models, where independent laboratories use a specific model to predict bioactivities of a set of compounds of interest. We observe that for all datasets and across a variety of descriptors and machine learning methods, random splitting predicts significantly higher performance than LSO at equal train/test split sizes. The tendency is less dramatic but still important for larger datasets. Interestingly, applicability domain filtering does not significantly change the outcome. We argue that this phenomenon is due to a combination of different chemical space, SAR bias in the random split and different assay methodologies employed by different laboratories. This raises the question whether simple random split or cross-validation provide realistic estimates on the performance of models built on datasets assembled from multiple sources. This work received financial support from the University of Vienna, doctoral programme Biopromotion, and from the Innovative Medicines Initiative Joint Undertaking under grant agreements n�115002 (eTOX), resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/2007-2013) and EFPIA companies' kind contribution.

Statistical learning from multiple-source datasets: Mind the gap!

248th National Meeting (2014)

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.