UNREALIZABLE LEARNING IN BINARY FEEDFORWARD NEURAL NETWORKS

Abstract

Statistical mechanics is used to study unrealizable generalization in two large feed-forward neural networks with binary weights and output, a perceptron and a tree committee machine. The student is trained by a teacher being larger, i.e. having more units than the student. It is shown that this is the same as using training data corrupted by Gaussian noise. Each machine is considered in the high temperature limit and in the replica symmetric approximation as well as for one step of replica symmetry breaking. For the perceptron a phase transition is found for low noise. However the transition is not to optimal learning. If the noise is increased the transition disappears. In both cases $ε_{g}$ will approach optimal performance with a $(\lnα/α)^k$ decay for large $α$. For the tree committee machine noise in the input layer is studied, as well as noise in the hidden layer. If there is no noise in the input layer there is, in the case of one step of repl! ica symmetry breaking, a phase tra nsition to optimal learning at some finite $α$ for all levels of noise in the hidden layer. When noise is added to the input layer the generalization behavior is similar to that of the perceptron. For one step of replica symmetry breaking, in the realizable limit, the values of the spinodal points found in this paper disagree with previously reported estimates \cite{seung1},\cite{schwarze1}. Here the value $α_{sp} = 2.79$ is found for the tree committee machine and $α_{sp} = 1.67$ for the perceptron.