Effect-Size Estimation and Confounding-Aware Permutations for PERMANOVA

Abstract

Motivation: PERMANOVA is widely used to test associations between covariates and high-dimensional data with arbitrary distributions, such as ecology, metabolomics, or microbial community profiling. In these contexts, its effect size measure (R2) is generally used to indicate how much the high-dimensional measurement covaries with respect to a separate, univariate measure. However, as we show, this conventional effect size is inflated under the null, resulting in values that do not accurately reflect the conventional expectation of "variance explained." Furthermore, standard permutation schemes yield inflated false positives in the presence of non-independent samples, such as repeated measure designs. Results: We introduce confounding-aware permutation schemes for both invariant and variant covariates. We additionally develop a causally-grounded, distance-based effect-size estimator derived from the Hájek average treatment effect estimator, providing interpretable effect sizes in the units of the chosen dissimilarity metric. Simulations using microbiome profiles demonstrate that these approaches control false positives while maintaining power, outperforming standard PERMANOVA implementations. Availability and Implementation: An R implementation is freely available in the permaBrioche module of the bioBakery ecosystem https://huttenhower.sph.harvard.edu/permabrioche. Contact: chuttenh@hsph.harvard.edu