Ensemble equivalence and asymptotic equipartition property in information theory

Abstract

There is a consensus in science that information theory and statistical physics have a close relationship but the literary proofs of the equivalence between most of the conceptions in the two disciplines are still missing. In this work, according to the statistical ensembles' description of the information sequences that are generated by the i.i.d. single variable and multivariate information source, the relationship between the ensemble equivalence and asymptotic equipartition property is established. We find that the description of information sequences in classical information theory is a special case of the canonical ensemble description of information sequences. They are both the maximum entropy approximation of the real signal generation. Vice versa, the conjugate microcanonical ensemble description of the information sequences under hard constraints satisfies the condition in real signal generation exactly. Thus, the microcanoincal ensemble description is closer to the real signal generation than the conjugate canonical ensemble, but the ensemble equivalence between the microcanonical and canonical ensemble in the thermodynamic limit guarantees the effectiveness of classical information theory, i.e., the asymptotic equipartition property in information theory is an isotope of ensemble equivalence from statistical physics.