Modified Sequentially Rejective Multiple Test Procedures

Abstract Suppose that n hypotheses H 1, H 2, …, H n with associated test statistics T 1, T 2, …, T n are to be tested by a procedure with experimentwise significance level (the probability of rejecting one or more true hypotheses) smaller than or equal to some specified value α. A commonly used procedure satisfying this condition is the Bonferroni (B) procedure, which consists of rejecting H i , for any i, iff the associated test statistic T i is significant at the level α′ = α/n. Holm (1979) introduced a modified Bonferroni procedure with greater power than the B procedure. Under Holm's sequentially rejective Bonferroni (SRB) procedure, if any hypothesis is rejected at the level α′ = α/n, the denominator of α′ for the next test is n − 1, and the criterion continues to be modified in a stagewise manner, with the denominator of α′ reduced by 1 each time a hypothesis is rejected, so that tests can be conducted at successively higher significance levels. Holm proved that the experimentwise significance level of the SRB procedure is ≤ α, as is that of the original B procedure. Often, the hypotheses being tested are logically interrelated so that not all combinations of true and false hypotheses are possible. As a simple example of such a situation suppose, given samples from three distributions, we want to test the three hypotheses of pairwise equality: μ i = μ′ i (i < i′ = 1, 2, 3), where μ i is the mean of distribution i. It is easily seen from the relations among the hypotheses that if any one of them is false, at least one other must be false. Thus there cannot be one false and two true hypotheses among these three. If we are testing all hypotheses of pairwise equality with more than three distributions, there are many such constraints. As another example, consider the hypotheses of independence of rows and columns of all 2×2 subtables of a K x L contingency table. It is shown that if one such hypothesis is false, then at least (K − 1) (L − 1) must be false. When there are logical implications among the hypotheses and alternatives, as in the preceding examples, Holm's SRB procedure can be improved to obtain a further increase in power. This article considers methods for achieving such improvement. One way of modifying the SRB method is as follows: Given that j − 1 hypotheses have been rejected, the denominator of α′, instead of being set at n − j + 1 for the next test as in the SRB procedure, can be set at t j , where t j equals the maximum number of hypotheses that could be true, given that at least j − 1 hypotheses are false. Obviously, t j is never greater than n − j + 1, and for some values of j it may be strictly smaller, as for j = 2 in the first example. Then this modified sequentially rejective Bonferroni (MSRB) procedure will never be less powerful (and typically will be more powerful) than the SRB procedure while (as is proved in the article) maintaining an experimentwise significance level ≤ α. The MSRB procedure is readily applicable to a wide variety of standard and nonstandard problems. A number of examples are given, and extensions and generalizations are discussed. It is pointed out that the methods may be adapted in some circumstances to the use of non-Bonferroni multiple test procedures.