Abstract Background Mayo endoscopic scoring in ulcerative colitis (UC) clinical trials follows a 2 + 1 reading workflow, involving a local (LR) and a blinded central reader (CR). Due to differences in experience among readers, around 40% of cases require adjudication by a third gastroenterologist, increasing workload and potentially delaying evaluation of patient eligibility and treatment response 1. AI has demonstrated potential in reducing scoring variability, but scepticism remains for AI use without human oversight 2. We introduce a novel 3 + 1 workflow combining dual human reads with an automatic AI reader (aiR) and we demonstrate how this paradigm can significantly reduce the adjudication rate, and consequently time required for central reading, without compromising endpoint quality while preserving human oversight. Methods We evaluated a 3 + 1 workflow that incorporates AI as a first-line adjudicator to resolve initial human disagreements (Fig. 1). aiR provides an automatic independent assessment that resolves discordance between LR and CR whenever it generates a score matching either primary read result. Full human adjudication is only triggered for challenging cases with a 2-points difference or when LR, CR and AI disagree. This approach ensures that a human score is always reported while decreasing the number of required adjudications. We evaluated the performance using a retrospective, anonymized dataset of 787 endoscopies from 414 UC patients. Agreement between the standard 2 + 1 and the proposed 3 + 1 workflows was assessed using the Quadratic Weighted Kappa (QWK). We analysed the accuracy (ACC) of the AI-assisted workflow in identifying eligibility, endoscopic improvement and remission. Results The 3 + 1 paradigm demonstrated almost perfect agreement with the traditional 2 + 1 workflow (QWK=0.94 95% CI: 0.93–0.95; ACC=86.5%) while substantially reducing the need for human adjudication from 47.6% to 7.9% (Fig. 2). AI agreed in 52.6% of cases with LR and 59.5% with CR, both higher than the inter-rater agreement (52.4%). Very high accuracy was observed in assessing patient eligibility at baseline (ACC=96.6%, N = 414) and in evaluating treatment response at follow-up (N = 373), both for endoscopic improvement (ACC=97.3%) and endoscopic remission (ACC=95.4%). Overall, aiR demonstrated lower variability than individual readers (LR vs CR: QWK=0.70, aiR vs CR: QWK=0.77, aiR vs 2 + 1: QWK=0.79). Conclusion The AI-enabled 3 + 1 workflow demonstrated almost perfect correlation in UC endoscopic scoring to traditional 2 + 1 human central reading while significantly reducing the need for human adjudication. Prospective integration of the proposed paradigm can improve trial operational performance, decrease reader burden and expedite reporting of results in UC studies. References: 1 Food and Drug Administration. Ulcerative Colitis: Developing Drugs for Treatment Internet. 2022 Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/ulcerative-colitis-developing-drugs-treatment 2 Takabayashi K, Kobayashi T, Matsuoka K, et al. Artificial intelligence quantifying endoscopic severity of ulcerative colitis in gradation scale. Dig Endosc. 2024;36(5):582-590. Conflict of interest: Mr. Juhasz, Adam: Clario: Full-time employee. Delmonte, Alessandro: Clario: Full-time employee. Szalma, Janos: Clario: Full-time employee. Vasanji, Amit: Clario: Full-time employee. Gaither, Kenneth: Clario: Full-time employee. Fuerst, Thomas: Clario: Full-time employee. Schaerer, Joel: Clario: Full-time employee. Vieira, Marcela: Clario: Full-time employee.
Juhasz et al. (Thu,) studied this question.