On Stepwise Control of the Generalized Familywise Error Rate

Abstract

A classical approach for dealing with the multiple testing problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of at least one false rejection. In many applications, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of $k$ or more false rejections, which is called the $k$-FWER. In this article, a unified approach is presented for deriving the $k$-FWER controlling procedures. We first generalize the well-known closure principle in the context of the FWER to the case of controlling the $k$-FWER. Then, we discuss how to derive the $k$-FWER controlling stepwise (stepdown or stepup) procedures based on marginal $p$-values using this principle. We show that, under certain conditions, generalized closed testing procedures can be reduced to stepwise procedures, and any stepwise procedure is equivalent to a generalized closed testing procedure. Finally, we generalize the well-known Hommel procedure in two directions, and show that any generalized Hommel procedure is equivalent to a generalized closed testing procedure with the same critical values.