Search
 
loader loading subjects...
cover
First Steps to Evaluate an NLP Tool’s Medication Extraction Accuracy from Discharge Letters
Deniz Caliskan, Jakob Zierk, Detlef Kraska, Stefan Schulz, Philipp Daumke, Hans-Ulrich Prokosch, Lorenz A. Kapsner
224 - 230
10.3233/SHTI210073
Research Article
Studies in Health Technology and Informatics
Volume 278: German Medical Data Sciences: Bringing Data to Life
Abstract

Introduction:

The aim of this study is to evaluate the use of a natural language processing (NLP) software to extract medication statements from unstructured medical discharge letters.

Methods:

Ten randomly selected discharge letters were extracted from the data warehouse of the University Hospital Erlangen (UHE) and manually annotated to create a gold standard. The AHD NLP tool, provided by MIRACUM’s industry partner was used to annotate these discharge letters. Annotations by the NLP tool where then compared to the gold standard on two levels: phrase precision (whether or not the whole medication statement has been identified correctly) and token precision (whether or not the medication name has been identified correctly within correctly discovered medication phrases).

Results:

The NLP tool detected medication related phrases with an overall F-measure of 0.852. The medication name has been identified correctly with an overall F-measure of 0.936.

Discussion:

This proof-of-concept study is a first step towards an automated scalable evaluation system for MIRACUM’s industry partner’s NLP tool by using a gold standard. Medication phrases and names have been correctly identified in most cases by the NLP system. Future effort needs to be put into extending and validating the gold standard.

Download
Creative Commons License