876 Background: Manual chart review has long been the standard for data collection in clinical research; however, this approach is time-intensive, costly, and prone to human error. Large Language Models (LLMs) are neural networks that are trained on large amounts of data, enabling them to perform natural language processing tasks. Our group has previously developed a framework for branching logic prompt design to leverage LLMs to handle complex reasoning queries and adapt to varied inputs. We sought to evaluate the feasibility and accuracy of this LLM framework using Llama-3b for automated, local extraction of pathologic variables from pathology reports in bladder cancer. Methods: Patients undergoing radical cystectomy from 2001 to 2025 were included in retrospective analysis. Prompts were designed to evaluate nine variables from surgical pathology notes, to include pT stage, pN stage, number of lymph nodes examined/positive, margin status, variant histology, and lymphovascular invasion (LVI). A manually extracted database was used as a reference. Patients with missing manual data were excluded on a variable-by-variable basis. Comparison between LLM and manual extracted data was assessed via % agreement and Cohen’s Kappa statistic. Results: 1898 radical cystectomy patients were included for analysis with matched manual data. The LLM generated 16046 datapoints across 9 variables with an overall agreement of 85.8% compared to manually abstracted data. There were no missing LLM generated datapoints across variables. Agreement and statistical comparisons by variable are demonstrated in Table 1, with (LVI) demonstrating the highest statistical agreement (n = 1734, kappa = 0.78, agreement 91.5%). Urethral margin status demonstrated the lowest statistical agreement with kappa = 0.136. Conclusions: We demonstrated the ability of branch logic prompting with an open-source LLM to accurately review, interpret, and extract pathology data for patients undergoing radical cystectomy. Future iterative improvements to improve variable agreement for tumor stage and margin status are nearing completion. Further refinement of prompt design and manual arbitration is needed to improve LLM accuracy for the eventual utilization of LLMs as the primary means for clinical data extraction. Pathology agreement of LLM and manual data extraction. Variable n Agreement (%) κ Lymphovascular Invasion 1734 91.5 0.7795 # Positive Nodes 1723 92.3 0.7633 # Nodes 1729 74.6 0.7377 pN Stage 1883 86.6 0.6922 Margin Status 1741 93.28 0.6716 pT stage 1883 66.8 0.6317 Variant Histology 1872 79.9 0.4141 Ureteral Margin Status 1741 90.9 0.3588 Urethral Margin Status 1740 98.6 0.136 Cohen's Kappa statistic denoted by κ.
Knorr et al. (Sun,) studied this question.