Using Large Language Models to Identify Patient–Oncologist Communication Domains: A Feasibility Study

Background: The American Society of Clinical Oncology (ASCO) convened a multidisciplinary panel in 2017, resulting in patient–oncologist communication guidelines. Ideally, these conversations should be documented in the medical records. However, chart review for communication topics is inefficient. Large language models (LLMs) present a computational method for identification of communication domains in clinical notes, subsequently providing feedback for clinicians. Objectives: The purpose of this study was to develop an approach using LLMs to identify communication domains in unstructured free text notes, validating against gold-standard chart review. Setting/Subjects: The study population included 134 clinical notes from 30 patients with advanced cancer seen in June 2024 at one of seven Dana-Farber Cancer Institute clinics (Boston, MA). We used a HIPAA-secure artificial intelligence tool based on GPT-4o to develop an LLM prompt for identification of communication domains. Measurements: We used standard performance metrics to compare the LLM prompt to chart review for all six communication domains. A hallucination index was calculated to assess false information that may be produced by LLMs when applied to large data sets. Results: Across communication domains, compared to chart review, the note-level LLM analysis achieved sensitivity ranging from 0.43 to 1.0, specificity ranging from 0.32 to 0.99, and accuracy ranging from 0.51 to 0.99. The average hallucination index for all domains was low. LLM abstraction required approximately 7 seconds per note, compared to 5–7 minutes with chart review. Conclusion: LLMs have the potential to identify ASCO communication domains. Future directions include applying the method for quality improvement efforts, such as generating feedback for oncologists on topics that may require follow-up.