Medical Named Entity Recognition (NER) models still rely on labor-intensive and costly annotated training data, while the use of Large Language Models (LLMs) can reduce annotation effort but often introduces quality issues. This paper evaluates NERFlow, a workflow-based subsystem that supports medical experts in creating training data for machine-learning based NER using configurable pipelines and both local and cloud-based LLMs. Integrated into the KM-EP environment, NERFlow offers automated pre-annotation, editing, comparison, and export features. Building on earlier expert walkthroughs, this study presents the first user-centered evaluation using a guided walkthrough and a structured survey based on Kasunic’s framework. Eighteen participants with diverse NER experience assessed usability, usefulness, and interface quality. Results show consistently positive ratings (means >4.9/7); KM-EP sign-in and workflow guidance received the highest scores. Users valued the annotation wizard, flexible model/workflow selection, and comparison tools, while noting limitations for very large annotations and taxonomy handling. Overall, NERFlow is perceived as practical and effective for real-world annotation tasks, with clear directions for further improvement.
Freund et al. (Thu,) studied this question.