We aimed to statefully validate Retinopathy of Prematurity (ROP) grading by evaluating intergrader variability over age-matched simulated exams. In a prospective retinal photographic grading cohort study, both eyes of twenty patients with five images per eye per visit were graded by twelve expert graders across six weekly visits. For each eye visit, ROP experts graded the Zone, Stage, and Plus, from which the Telemedicine ROP Severity Score (tROP-SS) was calculated. Graders retained knowledge of prior assessments for a given eye and followed International Classification of ROP rules. Intergrader coefficient of variation (CoV) for Zone, Stage, Plus, and tROP-SS remained below 20% across six weeks of age-matched visits. Zone showed the most changes in CoV (p < 0.001) with a minimum at week three of 6.0% (95% CI: 3.0-8.9%). Stage (9.2–13.1%) did not significantly vary over time. CoV of Plus gradually increased (p = 0.002) from week 1 (1.2%, 95% CI: 0.3-2.0%) to week 6 (3.3%, 95% CI: 1.7–4.9%). Agreement for tROP-SS varied over time (p < 0.001) and reached a local minimum at week three (10.5%, 95% CI: 7.3–13.8%). High intergrader agreement in a stateful data set validates the use of tROP-SS as a scoring system to capture week-to-week ROP progression.
Pratt et al. (Tue,) studied this question.