An LLM-Based Tool for Supporting User Experience Evaluation: A Multi-Perspective Approach Using Usability Heuristics, Accessibility Guidelines, and Personas

Evaluating user experience of websites is a resource-intensive process that typically requires substantial time, costs, and the participation of numerous user experience experts and end users. In particular, accessibility (as an essential aspect of user experience) is often insufficiently evaluated, as the diverse needs of individuals with diverse impairments are not adequately considered. To support user experience evaluations, we present an initial version of an LLM-based tool that automates both expert- and user-based evaluation. On the expert-based side, the tool conducts a heuristic evaluation based on Jakob Nielsen’s ten usability heuristics and performs an accessibility evaluation in accordance with the Web Content Accessibility Guidelines (WCAG). On the user-based side, it simulates diverse users, particularly those with varying needs and impairments. Our evaluation shows that the tool provides valuable insights into website user experience, particularly in detecting accessibility issues. However, the results also indicate that it cannot fully replace human evaluators and should therefore be regarded as an assistance system rather than a complete substitute. In particular, it offers a low-cost means of supporting iterative evaluations across multiple versions of prototypes and software systems.