PMID- 32410156
OWN - NLM
STAT- MEDLINE
DCOM- 20210805
LR  - 20210805
IS  - 1179-1942 (Electronic)
IS  - 0114-5916 (Print)
IS  - 0114-5916 (Linking)
VI  - 43
IP  - 8
DP  - 2020 Aug
TI  - Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results 
      from the Web-RADR Project.
PG  - 797-808
LID - 10.1007/s40264-020-00942-3 [doi]
AB  - INTRODUCTION: A large number of studies on systems to detect and sometimes 
      normalize adverse events (AEs) in social media have been published, but evidence 
      of their practical utility is scarce. This raises the question of the 
      transferability of such systems to new settings. OBJECTIVES: The aims of this 
      study were to develop an AE recognition system, prospectively evaluate its 
      performance on an external benchmark dataset and identify potential factors 
      influencing the transferability of AE recognition systems. METHODS: A pipeline 
      based on dictionary lookups and logistic regression classifiers was developed 
      using a proprietary dataset of 196,533 Tweets manually annotated for AE relations 
      and prospectively evaluated the system on the publicly available WEB-RADR 
      reference dataset, exploring different aspects affecting transferability. 
      RESULTS: Our system achieved 0.53 precision, 0.52 recall and 0.52 F1-score on the 
      development test set; however, when applied to the WEB-RADR reference dataset, 
      system performance dropped to 0.38 precision, 0.20 recall and 0.26 F1-score. 
      Similarly, a previously published method aiming at automatically detecting 
      adverse event posts reported 0.5 precision, 0.92 recall and 0.65 F1-score on thus 
      another dataset, while performance on the WEB-RADR reference dataset was reduced 
      to 0.37 precision, 0.63 recall and 0.46 F1-score. We identified four potential 
      factors leading to poor transferability: overfitting, selection bias, label bias 
      and prevalence. CONCLUSION: We warn the community about a potentially large 
      discrepancy between the expected performance of automated AE recognition systems 
      based on published results and the actual observed performance on independent 
      data. This study highlights the difficulty of implementing an all-purpose system 
      for automatic adverse event recognition in Twitter, which could explain the lack 
      of such systems in practical pharmacovigilance settings. Our recommendation is to 
      use benchmark independent datasets, such as the WEB-RADR reference, to 
      investigate the transferability of the adverse event recognition systems and 
      ultimately enforce rigorous comparisons across studies on the task.
FAU - Gattepaille, Lucie M
AU  - Gattepaille LM
AD  - Uppsala Monitoring Centre, Box 1051, 75140, Uppsala, Sweden. 
      lucie.gattepaille@who-umc.org.
FAU - Hedfors Vidlin, Sara
AU  - Hedfors Vidlin S
AD  - Uppsala Monitoring Centre, Box 1051, 75140, Uppsala, Sweden.
FAU - Bergvall, Tomas
AU  - Bergvall T
AD  - Uppsala Monitoring Centre, Box 1051, 75140, Uppsala, Sweden.
FAU - Pierce, Carrie E
AU  - Pierce CE
AD  - Uppsala Monitoring Centre, Box 1051, 75140, Uppsala, Sweden.
FAU - Ellenius, Johan
AU  - Ellenius J
AD  - Uppsala Monitoring Centre, Box 1051, 75140, Uppsala, Sweden.
LA  - eng
GR  - 115632/Innovative Medicines Initiative/International
PT  - Evaluation Study
PT  - Journal Article
PT  - Research Support, Non-U.S. Gov't
PL  - New Zealand
TA  - Drug Saf
JT  - Drug safety
JID - 9002928
SB  - IM
MH  - Adverse Drug Reaction Reporting Systems/*standards
MH  - Databases, Factual
MH  - Drug-Related Side Effects and Adverse Reactions/classification/*epidemiology
MH  - Humans
MH  - Logistic Models
MH  - Pharmacovigilance
MH  - Prevalence
MH  - Prospective Studies
MH  - Reproducibility of Results
MH  - Selection Bias
MH  - *Social Media
PMC - PMC7395913
COIS- Carrie E. Pierce has no conflict of interest directly relevant to the content of 
      this study. Lucie M. Gattepaille, Sara Hedfors Vidlin, Tomas Bergvall and Johan 
      Ellenius are employed by the Uppsala Monitoring Centre, a non-profit foundation 
      that commercializes WHODrug Global, the drug dictionary used in this study.
EDAT- 2020/05/16 06:00
MHDA- 2021/08/06 06:00
PMCR- 2020/05/14
CRDT- 2020/05/16 06:00
PHST- 2020/05/16 06:00 [pubmed]
PHST- 2021/08/06 06:00 [medline]
PHST- 2020/05/16 06:00 [entrez]
PHST- 2020/05/14 00:00 [pmc-release]
AID - 10.1007/s40264-020-00942-3 [pii]
AID - 942 [pii]
AID - 10.1007/s40264-020-00942-3 [doi]
PST - ppublish
SO  - Drug Saf. 2020 Aug;43(8):797-808. doi: 10.1007/s40264-020-00942-3.