Predictive Power Analysis of Multiple Test Procedures Under Arbitrary Dependence

Many statistical problems can be addressed by applying a multiple testing procedure (MTP) that controls either the Family-Wise Error Rate (FWER) or False Discovery Rate (FDR) under unknown arbitrarily interdependent p-values, without explicitly modeling these inter-correlations. They include the FWER-controlling Bonferroni MTP and Holm MTP; the FDR-controlling Benjamini-Yekutieli MTP; and the DP-MTP, based on a Dirichlet process (DP) prior distribution supporting the entire space of MTPs that control either the FWER or the FDR. For such an MTP, this study introduces a new and congenial method for Bayesian predictive power analysis, for power calculation and sample size determination for any given planned future (e.g., replication or interim) study. This novel MTP predictive power analysis method is based on a joint prior distribution defining a scale matrix mixture of asymmetric multivariate normal mean-variance mixture distributions, factorized as a general prior distribution for effect sizes (e.g., obtained from expert judgment or results of prior studies), and a uniform prior distribution for correlation matrices representing arbitrary dependencies between p-values of test statistics of given multiple hypothesis tests under their alternative hypotheses. The new MTP power analysis method also results in p-value weights which can be used to minimize the relative impacts of and assess for significance-chasing biases (e.g., publication bias, p-hacking, etc.) in multiple testing, without needing to assume that p-values (effect sizes) are independent. Previous MTP power analysis methods are conditional in that they assume fixed effect sizes and a fixed correlation matrix for test statistics (p-values) which may be difficult to specify; while, incongenially, not fully accounting for their uncertainty, especially for MTPs that control either the FWER or FDR for arbitrarily correlated p-values. The new simulation-based MTP predictive power analysis method is illustrated through the analysis of p-values obtained by a famous study of lead exposure and re-analyzed by the previous MTP literature, using R package bnpMTP (v1.0).