Training Dialogue Systems Using Entire Dialogue Context Evaluation

To enhance user satisfaction in dialogue with a system, it is essential not only to ensure that individual responsesare natural but also to improve the entire dialogue impression, including consistency, personality, and empathy.However, methods for optimizing dialogue systems to such entire dialogue impressions remain unclear. Inrecent studies on large language model (LLM)-based dialogue systems, Reinforcement Learning from AI Feedback(RLAIF) has emerged as a promising approach for improving the consistency and quality of entire dialogue impressions.When applying RLAIF with language models, LLM-based reward models guide the adaptation of the dialoguemodel. However, even with the capabilities of today ’s high-performing language models, it remains extremely challengingto derive accurate reward signals from zero-shot or few-shot prompts. To address this issue, we first constructmultiple reward models that assess entire dialogue impressions based on 12 evaluation metrics. These reward modelsare built using both prompt-based approaches and supervised fine-tuning (SFT), and their respective capabilities areempirically compared. The most effective reward model is then used to fine-tune the dialogue model to improvethe entire dialogue impression. Both automatic and human evaluations demonstrate that leveraging a reward modeltrained to assess entire dialogue impressions leads not only to improvements in evaluation metrics for entire dialogueimpressions but also to enhanced naturalness of the responses.