تقييم روبوتات الدردشة المعتمدة على الذكاء الاصطناعي التوليدي لبيانات التقييم واسعة النطاق: مقارنة بين نموذج اللغة الكبير كقاضٍ وتصنيفات البشر

Key Points

• The research aims to assess the effectiveness of a Generative AI chatbot in evaluating large-scale educational data against human ratings.
• Developed a customized generative AI chatbot using retrieval-augmented generation (RAG) framework
• Compared LLM-as-a-judge evaluations with human expert ratings based on correctness, completeness, and communication
• Evaluated chatbot responses using a three-dimensional framework and computed interrater reliability using quadratic weighted kappa.
• LLM-as-a-judge demonstrated comparable reliability to human ratings across evaluation dimensions
• No significant differences in inter-human versus human-to-LLM agreement, except in communication quality
• LLM-based evaluation offers a scalable and cost-effective alternative to human assessments.

Abstract