Harmonizing Case Validation for Vaccine Safety in Real-World Data: The VAC4EU Experience

Amirreza Dehghan Tarazjani,1 Daniel Weibel,1,2 Taylor Aurelius,3,4 Laura C Zwiers,5,6 Jesse M Van den Berg,7 Lina Pérez-Breva,8 Antonio Gimeno-Miguel,9,10 Luca Stona,11 Martín Solórzano,1,12 Anteneh Assefa Desalegn,13 Beatriz Poblador-Plou,9,10 Thom S Lysen,7 Jannik Wheler,14 Mahmoud Zidan,13 Juan José Carreras,8,15 Felipe Villalobos,12 Vera Ehrenstein,14 Kathryn Morton,3,4 Cristina Rebordosa,16 Fariba Ahmadizar,1,17 Joan Fortuny,16 Alejandro Arana,16 Miriam Sturkenboom1,2 1Department of Data Science and Biostatistics, Julius Global Health, University Medical Center Utrecht, Utrecht, The Netherlands; 2Vaccine Monitoring Collaboration for Europe (VAC4EU), Brussels, Belgium; 3Drug Safety Research Unit (DSRU), Southampton, UK; 4University of Portsmouth, Portsmouth, UK; 5Julius Global Health, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands; 6Julius Clinical, Zeist, The Netherlands; 7PHARMO Institute for Drug Outcomes Research, Utrecht, The Netherlands; 8Vaccine Research Department, Foundation for the Promotion of Health and Biomedical Research in the Valencian Region (FISABIO – Public Health), Valencia, 46020, Spain; 9Research Network on Chronicity, Primary Care, and Health Promotion (RICAPPS), Institute of Health Carlos III (ISCIII), Madrid, 28029, Spain; 10EpiChron Research Group, Aragon Health Sciences Institute (IACS), Aragon Health Research Institute (IIS Aragón), Miguel Servet University Hospital, Zaragoza, 50009, Spain; 11Fondazione Penta ETS, Padua, Italy; 12Fundació Institut Universitari per a la recerca a l’Atenció Primà ria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain; 13Pharmacoepidemiology and Drug Safety Research Group, Department of Pharmacy, Faculty of Mathematics and Natural Sciences, University of Oslo, Oslo, Norway; 14Department of Clinical Epidemiology, Center for Population Medicine, Department of Clinical Medicine, Aarhus University and Aarhus University Hospital, Aarhus, Denmark; 15Biomedical Research Consortium of Epidemiology and Public Health (CIBER-ESP), Instituto de Salud Carlos III, Madrid, Spain; 16Department of Epidemiology and Risk Management, RTI Health Solutions, Barcelona, Spain; 17General Practice Department, Amsterdam University Medical Center, Amsterdam, The NetherlandsCorrespondence: Miriam Sturkenboom, Department of Data Science and Biostatistics, Julius Global Health, University Medical Center Utrecht, Utrecht, The Netherlands, Email M.C.J.Sturkenboom@umcutrecht.nlPurpose: Applying standardized Brighton Collaboration (BC) case definitions retrospectively to heterogeneous real-world data (RWD) is challenging due to inconsistent clinical detail and data structures across settings. To address this, the Vaccine Monitoring Collaboration for Europe (VAC4EU) developed a structured validation pipeline that operationalizes BC case definitions for vaccine safety outcomes in RWD in a harmonized, scalable, and reusable manner.Methods: VAC4EU developed a systematic, stepwise approach to validate vaccine safety outcomes. BC case definitions were utilized when available, adapted for RWD as needed, and newly developed for outcomes without an existing BC definition. The approach involves: 1) Critical review and adaptation of BC definition by clinical and RWD experts; 2) Creation of dummy cases based on published reports; 3) Creation of REDCap electronic data collection forms (eDCF) incorporating decision logic to assigned levels of certainty (LOC); 4) Iterative testing of decision logic; and 5) comprehensive training of abstractors with real-time feedback. A dedicated task force assigned reference LOCs for dummy cases. Inter-rater reliability was measured using Fleiss’ kappa (κ) by comparing abstractor LOCs to the reference standard.Results: The pipeline was applied to 16 COVID-19 vaccine safety outcomes, of which 13 had existing BC definition. In total, 78 dummy case descriptions were developed across the outcomes for training of abstractors, and 15 REDCap eDCFs were created. Myocarditis and pericarditis shared an eDCF. Across 33 abstractors, 747 dummy case abstractions were completed. Agreement analysis showed 93 discrepancies (12.4%) and moderate overall concordance (κ = 0.55, across all outcomes), with the lowest for thrombosis with thrombocytopenia syndrome (κ = – 0.05).Conclusion: The VAC4EU validation pipeline provides a standardized framework for training and validating vaccine safety outcome in RWD. By adapting BC case definitions and dedicated training of abstractors, we will reduce variability in outcome validation in post-authorization safety studies.Keywords: validation, vaccine safety, real-world data, Brighton collaboration