Bias-aware meta-analysis: beyond fixed-effect and random-effects meta-analyses

Abstract

Meta-analysis aims to synthesize multiple studies to obtain a summary estimate of a population parameter of interest. However, existing meta-analysis methods require strong assumptions that render them not robust to bias and heterogeneity of the studies. In particular, fixed-effect meta-analysis assumes a common parameter across the studies, whereas random-effects meta-analysis models heterogeneity as independent and identically distributed random effects, which still implies a common parameter marginally over the random effects. Due to their limitations, we propose a bias-aware meta-analysis framework that allows for heterogeneous studies with fixed, unknown biases. With no bias, our bias-aware meta-analysis reduces to fixed-effect meta-analysis; with a given bound on the bias, we can still use the fixed-effect meta-analysis point estimator and standard error, but we must inflate the critical value for confidence intervals from the standard normal quantile to a folded normal quantile. We recommend reporting the maximum bias for significance, the maximum value of the bias such that the confidence intervals exclude the null value. We propose a formal model checking procedure and recommend reporting the minimum bias for compatibility, the minimum value of the bias such that the bias-aware meta-analysis model is compatible with the observed data. We discuss the practical implementation of our method and, in particular, the approaches to calibrating the maximum bias for significance. We also compare our method with traditional fixed-effect and random-effects meta-analyses.