Designing an accessible and interactive shopping assistant using deep learning for visually impaired individuals in Pakistan

Shopping is an essential activity of everyday routine. It is an easy task for people without impairments; however, this remains a significant challenge for people with visual impairments. Currently, visually impaired people must rely on family members or store personnel to assist them while shopping. It is challenging for them to navigate stores independently. In this study, the development and technical evaluation of an assistive shopping system for visually impaired individuals in Pakistan is presented using computer vision object identification and optical character recognition (OCR) to provide instantaneous product identification and audio feedback. A custom dataset was created using videos captured in various real-world retail environments to improve the accuracy of the system in recognizing critical product information, such as price, name, and expiration date. The performance of four state-of-the-art models, Faster R-CNN, YOLOv7, YOLOv9, and YOLOv10, was evaluated in this study, with YOLOv10 achieving the highest mean average precision (mAP) of 98%, making it the most satisfactory model for this application. A preliminary usability test also verified the system’s practical effectiveness. The trained model processes video frames by generating bounding boxes around identified objects, which are then passed to a pre-trained OCR model for information extraction. The extracted data are subsequently relayed to the user via auditory feedback. This assistive technology aims to enhance shopping independence for visually impaired individuals, enhancing accessibility in indoor retail settings.