Rethinking psychological measurement: Validity potential versus realised validity

We propose a concept of validity with a novel feature that we argue can facilitate improved measurement validation practices in the psychological sciences. Following Borsboom and colleagues, our concept of validity is measurement-specific and causal. This contrasts with current guidelines linking validity to the acceptability of both measurement and non-measurement-based interpretations of test scores. Benefits of a measurement-specific concept of validity are that it can make the requirements for valid measurement clearer and make validity claims easier to interpret, which we illustrate by comparing the use of test scores for measurement versus prediction. Our concept of validity also maintains that a causal relationship of sufficient strength from the attribute being measured to the measurement outcomes is necessary and sufficient for valid measurement. This places causal explanations at the centre of the validation process. While causal complexity will make the evaluation of psychological measurements as causal inferences extremely challenging, we describe how the interventionist theory of causation and related work on causal inference can serve as a starting point for addressing this challenge. The novel feature of our concept of validity is that it makes a distinction between the validity potential of measurement procedures in abstracto (e.g., tests) and the realised validity of concrete measurement outcomes (e.g., specific test scores). We describe key benefits of this novel distinction, including its potential to encourage the theoretical refinement of concepts, guide the selection of appropriate measurement procedures for use in research, and increase sample-specific validity evidence reporting. • Many current psychological validation practices are deficient. • Valid measurement requires a causal relationship from what is measured to test scores. • Realised validity applies to test scores generated in specific instances. • Validity potential applies to abstract measurement procedures. • Distinguishing realised validity from validity potential can support validation practices.