Questionnaire administration in HCI is frequently under-specifed, despite evidence that procedural choices shape how participants map experience onto subjective scales. We define questionnaire administration as a multi-level construct and synthesize nine administration methods. Previous work noticed that participants have difficulty answering questionnaire in an "absolute" way, we discuss the evidence of relative nature of judgement from other fields, and also highlight how reference points can induce anchoring and related response biases. We then report a 2 × 2 mixed-design online study (N = 96) using Raw NASA-TLX with N-back tasks (0-back versus 2-back tasks) on Prolific, comparing ordinary administration with a show previous answer administration method in a within-subjects experiment. Showing previous answer method reduced reported difficulty of rating on a scale and produced a pattern consistent with mild anchoring-based compression of workload differences, while leaving overall workload largely unchanged; self-reports indicate widespread comparative and consistency-seeking rating strategies. We discuss the transparency in reporting administration procedures in HCI papers. Finally, we offer practical guidelines for treating questionnaire administration as an explicit design choice in HCI studies.
Ang Li (Wed,) studied this question.