Data collection in randomized trials is expensive and labor intensive. With the rise in ongoing pragmatic trials, the use of electronic medical records (EMR) as a source of data has increased. Although potentially faster and cheaper, EMR use can lead to errors. Therefore, to ensure accurate data collection and to avoid systematic errors we performed a study comparing automated data extraction (ADE) with manual data extraction (MDE). We performed a retrospective cohort study to compare the accuracy of ADE using Structured Query Language with MDE by blinded physicians from our EMR. We tested the interrater agreement and intraclass correlation coefficient of clinical baseline data and outcomes of a random sample of 30 patients admitted to the ICU, on mechanical ventilation, requiring opioids for analgosedation for an upcoming pragmatic clinical trial. Key data compared included, but not limited to, patient's demographics, laboratory and vital signs, daily morphine milligram equivalent (MME), days alive and free of mechanical ventilation, days alive and free of hospitalization, days alive and free of ICU, days alive and free of vasopressors, and death. Among 238 patients screened over 1-month period, 72 fulfilled inclusion criteria and 30 were randomly selected to be included in the evaluation. We blindly collected 1320 baseline data, 2160 categorical outcomes and 705 continuous outcomes for a total of 4185 data points. The intraclass correlation coefficient and the Cohen's Kappa were perfect or almost perfect for all data, including outcomes such as daily MME, days alive and free of mechanical ventilation, days alive and free of ICU and days alive and free of hospital with p < 0.001. Among all rechecked data, the ADE was correct in 53 (77.9%) of cases, while MDE in 15 (22.1%). The inaccurate data collected by ADE accounted for 0.36% of the total data-points. The performance of ADE had almost perfect agreement for all outcomes and when rechecking for disagreements, it was more accurate than MDE.
Padrão et al. (Fri,) studied this question.