Evaluation of the Consistency of a Speech Verification System With Human Raters in Early Literacy Screening Assessments

This study investigates the use of a speech verification system (SVS) technology, a form of automatic speech recognition (ASR), in the assessment of children's reading skills. Despite the growing integration of ASR systems in educational assessment, significant challenges persist, particularly due to the acoustic, pronunciation and dialectal variability inherent in children's speech. Our research evaluated the consistency between human rater (HR) and speech verification system (SVS) scores produced from SoapBox Labs across three linguistic tasks—phoneme blending, expressive vocabulary and word reading. Results reveal variability in agreement rates, with SVS showing lower consistency with human raters in phonologically complex tasks like phoneme blending, compared to expressive vocabulary and word reading tasks. Additionally, we address potential racial differences in SVS performance, highlighting the importance of diverse speech sample collection to ensure equitable assessments, as well as inter-item differences within a task. The study concludes with recommendations for consideration of using SVS in educational assessments, advocating for ongoing research and algorithmic advancements to better support educational assessment practices.