What does this research mean for the field?

MediChat, an open-source chatbot, provides reliable and user-friendly medication information based on official patient information leaflets with 80% accuracy. Novelty: ClaimNovelty.METHODOLOGICAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

This research aims to evaluate the effectiveness of an open-source chatbot, MediChat, in providing medication information based on patient information leaflets.

March 12, 2026Future Internet0 citationsOpen Access

Implementation and Evaluation of an Open-Source Chatbot for Patient Information Leaflets

LHLisa HeilerFH JOANNEUM University of Applied Sciences KKKatharina KirchsteigerFH JOANNEUM University of Applied Sciences SHSten HankeFH JOANNEUM University of Applied Sciences

Key Points

This research aims to evaluate the effectiveness of an open-source chatbot, MediChat, in providing medication information based on patient information leaflets.
Developed MediChat using a retrieval-augmented generation architecture.
PILs were accessed via API and processed into a vector database.
Evaluated quantitatively with yes/no questions and qualitatively through simulated dialogues.
Achieved 80% accuracy on 200 yes/no questions across ten medication samples.
Overall precision was 0.977, recall 0.686, and an F1-score of 0.806.
Task completion exceeded 85% with positive ratings for relevance and quantity.

Abstract

Accessing and understanding medication information can be challenging for many people, especially when patient information leaflets (PILs) are long, complex, and printed in small font. This study presents MediChat, an open-source, locally executable chatbot designed to provide reliable, easy-to-read answers to medication-related questions based exclusively on official PILs. MediChat follows a retrieval-augmented generation (RAG) architecture: PILs from the Austrian Medicinal Product Index are received via API, converted to text, split into overlapping chunks, embedded, and stored in a Chroma vector database. From there the top-k relevant chunks are retrieved, and Llama 3.1 generates German responses based on this evidence. The system was evaluated using a hybrid framework. Quantitatively, 200 yes/no questions across ten drugs were answered with 80% accuracy, overall precision 0.977, recall 0.686, F1-score 0.806, and a mean response time of 727 ms. Qualitatively, two personas were used in eight simulated dialogues. Response times were around 1.1–1.3 s, and task completion exceeded 85% with high ratings for relevance and quantity. These results indicate that an open-source RAG chatbot can deliver leaflet-grounded, user-friendly medication information and provide a reproducible template for future healthcare chatbot evaluations.

AIに質問

Bookmark

View Full Paper