• Framework integrating LLM roles and usefulness in user research. • Critical examination of LLM risks in user research. • Practical examples for LLM use in mobility service development. User research is an integral part of mobility service development. However, user research is resource-intensive, including the effort required to recruit participants, and facilitate data collection procedures. Consequently, large language models (LLM) have been applied in the domain, building on the notion of LLMs being able to provide human-like outputs and thus simulate human users. Despite the increasing interest, guidelines for LLM implementation in user research processes and for evaluating the usefulness of LLM incorporation remain scarce, hindering researchers and practitioners from leveraging them effectively. Therefore, with the aim of providing a structured understanding of LLM integration, we delineate four main roles LLMs can play in user research (i.e., replace, complement, improve, research), along with three facets (i.e., realism, novelty, effort) for evaluating their usefulness. Combining the roles and usefulness facets, we introduce a framework for how LLMs could be used and what are the conditions for realizing their benefits. Additionally, we describe a hypothetical user research path, applying the approach to the development of new mobility services. The framework presents how and why LLMs could be used in various user research stages, providing a structured lens that aids researchers and practitioners to critically consider when and how to incorporate LLMs, and to evaluate LLM output usefulness, while considering the risks for doing so from the perspective of successful service design. By adopting a pragmatic and critical approach, the paper contributes to research on mobility, as well as more broadly on the use of synthetic participants in design praxis.
Wallius et al. (Thu,) studied this question.