Key points are not available for this paper at this time.
J. Richard Landis, Gary G. Koch, An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers, Biometrics, Vol. 33, No. 2 (Jun., 1977), pp. 363-374
Landis et al. (1977) studied this question.