When setting standards for high-stakes assessments, it is necessary to be clear whether the primary criterion should be importance or difficulty. While standard setting practices frequently rely on the difficulty to determine passing or failing thresholds, this can undermine the intended inferences and uses of assessments. Standard setting based on difficulty conflates importance with subjective perceptions of difficulty, misguiding decisions about what candidates need to demonstrate for safe and effective practice. We explore several reasons why difficulty does not align with the aims of assessment and how prioritising importance can better reflect the intended uses and inferences of a test. Our analysis incorporates: (1) the relevance of validity to setting standards, (2) potential misunderstandings of norm- and criterion referencing, (3) critical differentiation between importance and difficulty, (4) reasons for prioritising validity, (5) the variability of candidates’ difficulty perceptions, (6) challenges of assessor judgments, (7) the need for clear definitions of competence, and (8) the appropriate use of ‘would’ vs. ‘should’ in establishing standards. Ultimately, a valid standard reflects the ability to demonstrate performance of what is important, not difficult, in the workplace.
Burr et al. (Sat,) studied this question.