To develop machine learning models to detect the presence of drug-adverse event(D-AE) pairs embedded within hospital discharge summaries.

Our study was conducted using 1000 deidentified and annotated discharge summaries from 2011 which contained 1880 DAE pairs. The data was split 90%-10% for training and validation. Two methods of data transformations were explored prior to modelling. Various machine learning models were used.

Recall, precision and F1 statistics were computed for each machine learning model to identify the best model, on validation using a set of discharge summaries that had not been exposed to the model during training. 

In general, models built using the term frequency-inverse document frequency (TF-IDF) transformations of the discharge summaries fared poorer than models built using a word embedding algorithm (Word2vec). The best model built using TF-IDF had a recall  of 77%, precision of 76% and a F1 score of 77%, whereas the best model built using Word2vec had a recall  of 80%, precision of 88% and an F1 score of 84%. 

Among the Word2vec based models, the random forest model outperformed all other machine learning models (sensitivity: 75%, PPV: 83%, F1: 79%); K-nearest neighbour (sensitivity: 78%, PPV: 78%, F1: 78%), Stochastic Gradient Descent (sensitivity: 66%, PPV: 82%, F1: 74%), Naïve Bayes (sensitivity: 69%, PPV: 82%, F1: 75%).

This study demonstrates the value of applying machine learning on discharge summaries to detect the presence of DAE pairs embedded within the narrative, to potentially augment post-market drug safety surveillance efforts.

The best model detects 75% of all true DAE pairs present within the set of discharge summaries used on validation while having a precision of 83%, illustrating that only a few false positive predictions are made in detecting DAE pairs. 

Pharmacovigilance has to date heavily relied on spontaneous adverse event reports to detect safety signals, but underreporting has been a perennial limitation of spontaneous reporting. Automated methods of detecting potential drug-adverse event pairs are needed to enhance the detection of adverse drug reaction episodes to accelerate safety signal detection. To this end, our study has shown that there is potential to accurately identify discharge summaries that contain true DAE pairs using machine learning approaches.

Predicting the presence of drug-adverse event pairs in discharge summaries