What question did this study set out to answer?

This research assesses the quality of patient education materials for lung cancer from large language models compared to established sources.

May 29, 2026

Utilizing large language models for lung cancer patient education compared to publicly available information.

Key Points

This research assesses the quality of patient education materials for lung cancer from large language models compared to established sources.
Evaluated 5 common lung cancer questions using LLMs ChatGPT and Gemini along with ACS and NCI resources.
Seven providers scored the content on readability and quality, using various assessment scales.
Readability was measured with the Flesh-Kincaid grade level.
LLMs achieved higher information quality scores with fewer errors compared to publicly available websites.
Readability scores ranged from 6.9-8.2, indicating similar levels of comprehensibility.
Providers identified LLM content correctly more than 50% of the time; however, there was some confusion with human-generated responses.

Abstract

9019 Background: The advent of the internet, social media, and more recently of large language model (LLM) platforms has led patients to seek information about cancer directly through digital sources. Complete, readable, and accurate information could reduce physician time clarifying information and increase patient self-activation, especially in low-resource environments. We evaluated the quality of publicly available information provided for the five most common lung cancer questions asked on the internet with information provided by the American Cancer Society (ACS), the National Cancer Institute (NCI), ChatGPT, and Gemini. We hypothesized that LLM platforms could produce readable and information quality equivalent to that found on nationally recognized websites. Methods: ChatGPT (OpenAI, logged-out version, accessed 12/14/25) was queried to determine the five most common questions patients with lung cancer ask. These question prompts were used to generate responses in ChatGPT (OpenAI, logged-out version, accessed 1/13/26) and Gemini (Google, version 3 Flash, 1/13/26). Additional passages were extracted from NCI and ACS patient education web pages, and all passages were de-identified and reformatted with links and references removed. The deidentified passages were evaluated by seven providers representing medical oncology (n=4), radiation oncology (n=1), surgical oncology (n=2), and onco-primary care (n=1) using Information Quality Grade, Global Quality Scale, Error Classification, Comprehensibility, and Confabulation. Readability was determined via Flesh-Kincaid grade level. Results: Readability was similar between passages, ranging from grade levels 6.9-8.2. LLM’s were rated higher on information quality with fewer total errors. Providers were able to discern LLM content in > 50% of the cases but considered human-generated content as LLM in about one third of cases. The most frequent error reported in LLMs was too little information (n=19) and in websites too much information (n=29). Conclusions: LLMs can provide more succinct high-quality information for patients with lung cancer compared to current publicly available websites. The information provided by LLMs is accessible to the public, with potential positive implications for low-resourced populations. Further research is urgently needed to understand the potential of LLMs to improve lung cancer outcomes, such as patient self-activation and adherence. Source Readability(Grade level)* Info Quality*(1 high - 4 low) Global Quality Scale*(5 high - 1 low) % Comprehensibility % Confabulation Total # of errors % Considered LLM ACS 7.8 1.7 3.7 75.0 7.5 28 30.6 NCI 8.2 1.9 3.1 75.0 7.5 43 35.3 ChatGPT 6.9 1.5 4.3 100.0 2.6 20 63.2 Gemini 8.1 1.5 4.4 52.3 2.6 19 64.1 *Averages across all 5 questions.

Mark Helpful

Bookmark

Relay