A geometric view on Pearson's correlation coefficient and a generalization of it to non-linear dependencies

Abstract

Measuring strength or degree of statistical dependence between two random variables is a common problem in many domains. Pearson's correlation coefficient $\rho$ is an accurate measure of linear dependence. We show that $\rho$ is a normalized, Euclidean type distance between joint probability distribution of the two random variables and that when their independence is assumed while keeping their marginal distributions. And the normalizing constant is the geometric mean of two maximal distances, each between the joint probability distribution when the full linear dependence is assumed while preserving respective marginal distribution and that when the independence is assumed. Usage of it is restricted to linear dependence because it is based on Euclidean type distances that are generally not metrics and considered full dependence is linear. Therefore, we argue that if a suitable distance metric is used while considering all possible maximal dependences then it can measure any non-linear dependence. But then, one must define all the full dependences. Hellinger distance that is a metric can be used as the distance measure between probability distributions and obtain a generalization of $\rho$ for the discrete case.