Objective	

To develop machine learning models to detect the presence of drug-adverse event(D-AE) pairs embedded within hospital discharge summaries.


Background: This study was conducted using 1000 deidentified and annotated discharge summaries from 2011 which contained 1800 DAE pairs. The data was split 90%-10% for training and validation. Two methods of data transformations were explored prior to modelling. Various machine learning models were used.


Results	

Sensitivity, positive predictive value (PPV) and F1 statistics were computed for each machine learning model to identify the best model, on validation using a set of discharge summaries that had not been exposed to the model during training. In general, models built using the term frequency-inverse document frequency (TF-IDF) transformations of the discharge summaries fared poorer than models built using a word embedding algorithm (Word2vec). The best model built using TF-IDF had a sensitivity of 77%, PPV of 76% and a F1 score of 77%, whereas the best model built using Word2vec had a sensitivity of 80%, PPV of 88% and an F1 score of 84%. Among the Word2vec based models, the extreme boosted tree algorithm (XGBoost) with 4 layers outperformed all other machine learning models (sensitivity: 80%, PPV: 88%, F1: 84%); K-nearest neighbour (sensitivity: 69%, PPV: 82%, F1: 79%), Stochastic Gradient Descent (sensitivity: 69%, PPV: 80%, F1: 75%), Random Forest (sensitivity: 71%, PPV: 86%, F1: 78%).


Conclusion	

This study demonstrates the value of applying machine learning on discharge summaries to detect the presence of DAE pairs embedded within the narrative, to potentially augment post-market drug safety surveillance efforts. The best model detects 80% of all true DAE pairs present within the set of discharge summaries used on validation while having a PPV of 88%, illustrating that only a few false positive predictions are made in detecting DAE pairs. Pharmacovigilance has to date heavily relied on spontaneous adverse event reports to detect safety signals, but underreporting has been a perennial limitation of spontaneous reporting. Automated methods of detecting potential drug-adverse event pairs are needed to enhance the detection of adverse drug reaction episodes to accelerate safety signal detection. To this end, our study has shown that there is potential to accurately identify discharge summaries that contain true DAE pairs using machine learning approaches.

Predicting the presence of drug-adverse event pairs in discharge summaries

DIA Europe 2020 

A global, member-driven non-profit organization mobilizing life sciences and healthcare. DIA fosters innovation to improve health and well-being worldwide. Professionals from 80 countries continue to engage with DIA through our unparalleled membership network, educational offerings, and professional development opportunities.