arXiv subjects

Fabrizio Boncoraglio

Publications and source records attributed to Fabrizio Boncoraglio.

5 recordsLinked to original sources

arXiv2026

The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction $1 - \varepsilon$ of training labels is generated by a structured teacher rule, while a fraction $\varepsilon$ consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regularization and the choice of kernel or nonlinearity control the allocation of capacity between rule learning and memorization. Experiments on a CIFAR10-based RAF task show that the qualitative trade-off persists for fixed kernels, whereas representations learned by deeper neural networks substantially mitigate it, preserving factual recall while improving generalization. The RAF model provides a theoretical foundation for understanding how modern neural networks can infer structure while storing rare or non-compressible information.

stat.ML↗

arXiv2025

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws.

stat.ML↗

arXiv2025

Inference in Spreading Processes with Neural-Network Priors

Stochastic processes on graphs are a powerful tool for modelling complex dynamical systems such as epidemics. A recent line of work focused on the inference problem where one aims to estimate the state of every node at every time, starting from partial observation of a subset of nodes at a subset of times. In these works, the initial state of the process was assumed to be random i.i.d. over nodes. Such an assumption may not be realistic in practice, where one may have access to a set of covariate variables for every node that influence the initial state of the system. In this work, we will assume that the initial state of a node is an unknown function of such covariate variables. Given that functions can be represented by neural networks, we will study a model where the initial state is given by a simple neural network -- notably the single-layer perceptron acting on the known node-wise covariate variables. Within a Bayesian framework, we study how such neural-network prior information enhances the recovery of initial states and spreading trajectories. We derive a hybrid belief propagation and approximate message passing (BP-AMP) algorithm that handles both the spreading dynamics and the information included in the node covariates, and we assess its performance against the estimators that either use only the spreading information or use only the information from the covariate variables. We show that in some regimes, the model can exhibit first-order phase transitions when using a Rademacher distribution for the neural-network weights. These transitions create a statistical-to-computational gap where even the BP-AMP algorithm, despite the theoretical possibility of perfect recovery, fails to achieve it.

stat.ML↗

arXiv2025

Bayes optimal learning of attention-indexed models

We introduce the attention-indexed model (AIM), a theoretical framework for analyzing learning in deep attention layers. Inspired by multi-index models, AIM captures how token-level outputs emerge from layered bilinear interactions over high-dimensional embeddings. Unlike prior tractable attention models, AIM allows full-width key and query matrices, aligning more closely with practical transformers. Using tools from statistical mechanics and random matrix theory, we derive closed-form predictions for Bayes-optimal generalization error and identify sharp phase transitions as a function of sample complexity, model width, and sequence length. We propose a matching approximate message passing algorithm and show that gradient descent can reach optimal performance. AIM offers a solvable playground for understanding learning in self-attention layers, that are key components of modern architectures.

cs.LG↗

arXiv2023

Complex contagion in social systems with distrust

Social systems are characterized by the presence of group interactions and by the existence of both trust and distrust relations. Although there is a wide literature on signed social networks, where positive signs associated to the links indicate trust, friendship, agreement, while negative signs represent distrust, antagonism, and disagreement, very little is known about the effect that signed interactions can have on the spreading of social behaviors when higher-order interactions are taken into account. In this paper we focus on processes of complex contagion, such as the adoption of social norms, where exposure to multiple sources is needed for the contagion to occur. Complex contagion has been recently modeled by higher-order networks, such as simplicial complexes, which allow transmission to happen not only through the links connecting pair of nodes, but also in group interactions, namely over simplices of dimension larger or equal than two. Here, we introduce a model of complex contagion on signed simplicial complexes, and we investigate the role played by trust and distrust on the dynamics of a social contagion process. The presence of higher-order signed structures in our model naturally induces new infection and recovery mechanisms. Through numerical simulations and analytical results in the mean-field approximation, we show how distrust determines the way the system moves from a state where no individuals adopt the social behavior, to a state where a finite fraction of the population actively spreads it. Interestingly, the fraction of spreading individuals displays a non-monotonic dependence on the average number of connections between individuals. We then investigate how social balance affects social contagion, finding that balanced triads either promote or impede contagion based on the relative abundance of fully trusted relations.

physics.soc-ph↗