Search
 
loader loading subjects...
cover
Impact of Professional Background on Inter-Annotator Variability and Accuracy During Annotation of Clinical Notes
Kristina Weishäupl, Mario Schuster, Christina Mayrl, Diana Humberger, Antonia Hoffstädter, Theresa Fischer, Bianca Punz-Reidlinger, Fabian Wiesmüller, Aaron Lauschensky, Dieter Hayn, Karl Kreiner, Bettina Fetz, Luca Brunelli, Gerhard Poelzl, Bernhard Pfeifer, Gerald Slamanig, Sabrina Barbara Neururer, Günter Schreier
248 - 253
10.3233/SHTI230048
Research Article
Studies in Health Technology and Informatics
Volume 301: dHealth 2023
Abstract

Background:

The aging population’s need for treatment of chronic diseases is exhibiting a marked increase in urgency, with heart failure being one of the most severe diseases in this regard. To improve outpatient care of these patients and reduce hospitalization rates, the telemedical disease management program HerzMobil was developed in the past.

Objective:

This work aims to analyze the inter-annotator variability among two professional groups (healthcare and engineering) involved in this program’s annotation process of free-text clinical notes using categories.

Methods:

A dataset of 1,300 text snippets was annotated by 13 annotators with different backgrounds. Inter-annotator variability and accuracy were evaluated using the F1-score and analyzed for differences between categories, annotators, and their professional backgrounds.

Results:

The results show a significant difference between note categories concerning inter-annotator variability (p<0.0001) and accuracy (p<0.0001). However, there was no statistically significant difference between the two annotator groups, neither concerning inter-annotator variability (p=0.15) nor accuracy (p=0.84).

Conclusion:

Professional background had no significant impact on the annotation of free-text HerzMobil notes.

Download
Creative Commons License