The Impact of Likert Scale Design on Judgment Reliability in Korean and English LLM-as-a-Judge

Key Points

• The research aims to analyze how the design of Likert scales affects judgment reliability in LLM evaluations across Korean and English.
• Utilized the NLG-Eval dataset for experiments.
• Analyzed the impact of ascending and descending score designs on reliability.
• Compared the consistency of numeric and descriptive scales with word-based scales.
• Investigated reliability patterns in both Korean and English contexts.
• Ascending scale design demonstrated higher judgment reliability than descending design.
• Numeric scales combined with descriptions showed greater consistency than word-based scales.
• Overall judgment reliability in Korean was lower than in English, yet similar patterns of reliability variation were observed across both languages.

Abstract