Weight space structure and analysis using a finite replica number in the Ising perceptron

Abstract

The weight space of the Ising perceptron in which a set of random patterns is stored is examined using the generating function of the partition function $\phi(n)=(1/N)\log [Z^n]$ as the dimension of the weight vector $N$ tends to infinity, where $Z$ is the partition function and $[ ... ]$ represents the configurational average. We utilize $\phi(n)$ for two purposes, depending on the value of the ratio $\alpha=M/N$, where $M$ is the number of random patterns. For $\alpha < \alpha_{\rm s}=0.833 ...$, we employ $\phi(n)$, in conjunction with Parisi's one-step replica symmetry breaking scheme in the limit of $n \to 0$, to evaluate the complexity that characterizes the number of disjoint clusters of weights that are compatible with a given set of random patterns, which indicates that, in typical cases, the weight space is equally dominated by a single large cluster of exponentially many weights and exponentially many small clusters of a single weight. For $\alpha > \alpha_{\rm s}$, on the other hand, $\phi(n)$ is used to assess the rate function of a small probability that a given set of random patterns is atypically separable by the Ising perceptrons. We show that the analyticity of the rate function changes at $\alpha = \alpha_{\rm GD}=1.245 ... $, which implies that the dominant configuration of the atypically separable patterns exhibits a phase transition at this critical ratio. Extensive numerical experiments are conducted to support the theoretical predictions.