The quality of data in content analysis, in surveys with openended questions, in the observation of unstructured social events, and so on, critically depends on the reliability with which primary observations are assigned to categories, scaled, or measured. To help assure valid interpretations, agreement between two independent observers is measured. When agreement is due merely to chance, data may have little to do with the phenomena studied. In order for such data to be empirically meaningful, a high degree of inter-observer agreement must be demonstrated. This paper suggests that several heretofore unrelated bivariate agreement coefficients may in fact be regarded as belonging to one family. It presents a paradigm through which their formal resemblances become transparent and it proposes efficient formulas for their computation.
Klaus Krippendorff (1970) studied this question.