arXiv subjects

Francesco Cagnetta

Publications and source records attributed to Francesco Cagnetta.

At least 19 recordsLinked to original sources

arXiv2026

Deriving Neural Scaling Laws from the statistics of natural language

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

cs.LG↗

arXiv2026

Deep networks learn to parse uniform-depth context-free languages from local statistics

Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning. Studies of the internal representations of Large Language Models (LLMs) support their ability to parse text when predicting the next word, while representing semantic notions independently of surface form. Yet, which data statistics make these feats possible, and how much data is required, remain largely unknown. Probabilistic context-free grammars (PCFGs) provide a tractable testbed for studying these questions. However, prior work has focused either on the post-hoc characterization of the parsing-like algorithms used by trained networks; or on the learnability of PCFGs with fixed syntax, where parsing is unnecessary. Here, we (i) introduce a tunable class of PCFGs in which both the degree of ambiguity and the correlation structure across scales can be controlled; (ii) provide a learning mechanism -- an inference algorithm inspired by the structure of deep convolutional networks -- that links learnability and sample complexity to specific language statistics; and (iii) validate our predictions empirically across deep convolutional and transformer-based architectures. Overall, we propose a unifying framework where correlations at different scales lift local ambiguities, enabling the emergence of hierarchical representations of the data.

stat.ML↗

arXiv2025

Learning curves theory for hierarchically compositional data with power-law distributed features

Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour.

stat.ML↗

arXiv2025

Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures

How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the Random Hierarchy Model (RHM) -- an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.

cs.LG↗

arXiv2025

How Compositional Generalization and Creativity Improve as Diffusion Models are Trained

Natural data is often organized as a hierarchical composition of features. How many samples do generative models need in order to learn the composition rules, so as to produce a combinatorially large number of novel data? What signal in the data is exploited to learn those rules? We investigate these questions in the context of diffusion models both theoretically and empirically. Theoretically, we consider a simple probabilistic context-free grammar - a tree-like graphical model used to represent the hierarchical and compositional structure of data such as language and images. We demonstrate that diffusion models learn the grammar's composition rules with the sample complexity required for clustering features with statistically similar context, a process similar to the word2vec algorithm. However, this clustering emerges hierarchically: higher-level features associated with longer contexts require more data to be identified. This mechanism leads to a sample complexity that scales polynomially with the said context size. As a result, diffusion models trained on an intermediate dataset size generate data coherent up to a certain scale, but lacking global coherence. We test these predictions across different domains and find remarkable agreement: both generated texts and images achieve progressively larger coherence lengths as the training time or dataset size grows. We discuss connections between the hierarchical clustering mechanism we introduce here and the renormalization group in physics.

stat.ML↗

arXiv2024

Towards a theory of how the structure of language is acquired by deep neural networks

How much data is required to learn the structure of a language via next-token prediction? We study this question for synthetic datasets generated via a Probabilistic Context-Free Grammar (PCFG) -- a tree-like generative model that captures many of the hierarchical structures found in natural languages. We determine token-token correlations analytically in our model and show that they can be used to build a representation of the grammar's hidden variables, the longer the range the deeper the variable. In addition, a finite training set limits the resolution of correlations to an effective range, whose size grows with that of the training set. As a result, a Language Model trained with increasingly many examples can build a deeper representation of the grammar's structure, thus reaching good performance despite the high dimensionality of the problem. We conjecture that the relationship between training set size and effective range of correlations holds beyond our synthetic datasets. In particular, our conjecture predicts how the scaling law for the test loss behaviour with training set size depends on the length of the context window, which we confirm empirically in Shakespeare's plays and Wikipedia articles.

cs.CL↗

arXiv2023

Kernels, Data & Physics

Lecture notes from the course given by Professor Julia Kempe at the summer school "Statistical physics of Machine Learning" in Les Houches. The notes discuss the so-called NTK approach to problems in machine learning, which consists of gaining an understanding of generally unsolvable problems by finding a tractable kernel formulation. The notes are mainly focused on practical applications such as data distillation and adversarial robustness, examples of inductive bias are also discussed.

cs.LG↗

arXiv2023

How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model

Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of sub-features chosen among several equivalent ones and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations, and provide an estimate of the number of data required to learn a hierarchical task.

cs.LG↗

arXiv2022

How deep convolutional neural networks lose spatial information with training

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the net. This loss of sensitivity correlates with performance, and surprisingly correlates with a gain of sensitivity to white noise acquired during training. These facts are unexplained, and as we demonstrate still hold when white noise is added to the images of the training set. Here, we (i) show empirically for various architectures that stability to image diffeomorphisms is achieved by both spatial and channel pooling, (ii) introduce a model scale-detection task which reproduces our empirical observations on spatial pooling and (iii) compute analitically how the sensitivity to diffeomorphisms and noise scales with depth due to spatial pooling. The scalings are found to depend on the presence of strides in the net architecture. We find that the increased sensitivity to noise is due to the perturbing noise piling up during pooling, after being rectified by ReLU units.

cs.LG↗

arXiv2022

What Can Be Learnt With Wide Convolutional Neural Networks?

Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harness the local and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how such structure affects performance, e.g., the rate of decay of the generalisation error with the number of training samples. In this paper, we study infinitely-wide deep CNNs in the kernel regime. First, we show that the spectrum of the corresponding kernel inherits the hierarchical structure of the network, and we characterise its asymptotics. Then, we use this result together with generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function. In particular, we find that if the target function depends on low-dimensional subsets of adjacent input variables, then the decay of the error is controlled by the effective dimensionality of these subsets. Conversely, if the target function depends on the full set of input variables, then the error decay is controlled by the input dimension. We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN with randomly-initialised parameters. Interestingly, we find that, despite their hierarchical structure, the functions generated by infinitely-wide deep CNNs are too rich to be efficiently learnable in high dimension.

stat.ML↗

arXiv2022

Learning sparse features can lead to overfitting in neural networks

It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data. Yet, understanding when and how this feature learning improves performance remains a challenge: for example, it is beneficial for modern architectures trained to classify images, whereas it is detrimental for fully-connected networks trained for the same task on the same data. Here we propose an explanation for this puzzle, by showing that feature learning can perform worse than lazy training (via random feature kernel or the NTK) as the former can lead to a sparser neural representation. Although sparsity is known to be essential for learning anisotropic data, it is detrimental when the target function is constant or smooth along certain directions of input space. We illustrate this phenomenon in two settings: (i) regression of Gaussian random functions on the d-dimensional unit sphere and (ii) classification of benchmark datasets of images. For (i), we compute the scaling of the generalization error with number of training points, and show that methods that do not learn features generalize better, even when the dimension of the input space is large. For (ii), we show empirically that learning features can indeed lead to sparse and thereby less smooth representations of the image predictors. This fact is plausibly responsible for deteriorating the performance, which is known to be correlated with smoothness along diffeomorphisms.

stat.ML↗

arXiv2021

Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a challenge. We study this problem within a teacher-student framework for kernel regression, using `convolutional' kernels inspired by the neural tangent kernel of simple convolutional architectures of given filter size. Using heuristic methods from physics, we find in the ridgeless case that locality is key in determining the learning curve exponent $\beta$ (that relates the test error $\epsilon_t\sim P^{-\beta}$ to the size of the training set $P$), whereas translational invariance is not. In particular, if the filter size of the teacher $t$ is smaller than that of the student $s$, $\beta$ is a function of $s$ only and does not depend on the input dimension. We confirm our predictions on $\beta$ empirically. We conclude by proving, using a natural universality assumption, that performing kernel regression with a ridge that decreases with the size of the training set leads to similar learning curve exponents to those we obtain in the ridgeless case.

stat.ML↗

arXiv2021

A renormalization group study of the dynamics of active membranes: universality classes and scaling laws

Motivated by experimental observations of patterning at the leading edge of motile eukaryotic cells, we introduce a general model for the dynamics of nearly-flat fluid membranes driven from within by an ensemble of activators. We include, in particular, a kinematic coupling between activator density and membrane slope which generically arises whenever the membrane has a non-vanishing normal speed. We unveil the phase diagram of the model by means of a perturbative field-theoretical renormalization group analysis. Due to the aforementioned kinematic coupling the natural dynamical scaling is acoustic, that is the dynamical critical exponent is 1. However, as soon as the the normal velocity of the membrane is tuned to zero, the system crosses over to diffusive dynamic scaling in mean field. Distinct critical points can be reached depending on how the limit of vanishing velocity is realised: in each of them corrections to scaling due to nonlinear coupling terms must be taken into accounts. The detailed analysis of these critical points reveals novel scaling regimes wich can be accessed with perturbative methods, together with signs of strong coupling behaviour, which establishes a promising ground for further non-perturbative calculations. Our results unify several previous studies on the dynamics of active membrane, while also identifying nontrivial scaling regimes which cannot be captured by passive theories of fluctuating interfaces and are relevant for the physics of living membranes.

cond-mat.stat-mech↗

arXiv2021

Universal properties of active membranes

We put forward a general field theory for membranes with embedded activators and analyse their critical properties using renormalization group techniques. Depending on the membrane-activator coupling, we find a crossover between acoustic and diffusive scaling regimes, with mean-field dynamical critical exponents z = 1 and 2 respectively. We argue that the acoustic scaling, which is exact in all spatial dimensions, is a suitable candidate for the universal description of the spatiotemporal patterns observed at the leading edge of motile cells. Furthermore, one-loop corrections to the diffusive mean-field exponents reveal universal behaviour distinct from the Kardar-Parisi-Zhang scaling of passive interfaces and signs of strong-coupling behaviour.

cond-mat.stat-mech↗

arXiv2021

Work Fluctuations in the Active Ornstein- Uhlenbeck Particle model

We study the large deviations of the power injected by the active force for an Active Ornstein-Uhlenbeck Particle (AOUP), free or in a confining potential. For the free-particle case, we compute the rate function analytically in d-dimensions from a saddle-point expansion, and numerically in two dimensions by it a) direct sampling of the active work in numerical solutions of the AOUP equations and b) Legendre-Fenchel transform of the scaled cumulant generating function obtained via a cloning algorithm. The rate function presents asymptotically linear branches on both sides and it is independent of the system's dimensionality, apart from a multiplicative factor. For the confining potential case, we focus on two-dimensional systems and obtain the rate function numerically using both methods a) and b). We find a different scenario for harmonic and anharmonic potentials: in the former case, the phenomenology of fluctuations is analogous to that of a free particle, but the rate function might be non-analytic; in the latter case the rate functions are analytic, but fluctuations are realised by entirely different means, which rely strongly on the particle-potential interaction. Finally, we check the validity of a fluctuation relation for the active work distribution. In the free-particle case, the relation is satisfied with a slope proportional to the bath temperature. The same slope is found for the harmonic potential, regardless of activity, and for an anharmonic potential with low activity. In the anharmonic case with high activity, instead, we find a different slope which is equal to an effective temperature obtained from the fluctuation-dissipation theorem.

cond-mat.stat-mech↗

arXiv2019

Kinetic roughening in active interfaces

The essential features of many interfaces driven out of equilibrium are described by the same equation---the Kardar-Parisi-Zhang (KPZ) equation. How do living interfaces, such as the cell membrane, fit into this picture? In an endeavour to answer such a question, we proposed in [F. Cagnetta, M. R. Evans, D. Marenduzzo, PRL 120, 258001 (2018)] an idealised model for the membrane of a moving cell. Here we discuss how the addition of simple ingredients inspired by the dynamics of the membrane of moving cells affects common kinetic roughening theories such as the KPZ and Edwards-Wilkinson equations.

cond-mat.stat-mech↗

arXiv2019

Efficiency of one-dimensional active transport conditioned on motility

By conditioning a stochastic process on the value of an observable, one obtains a new stochastic process with different properties. We apply this idea in the context of active matter, and condition interacting self-propelled particles on their individual motility. Using the effective process formalism from dynamical large deviations theory, we derive the interactions that actuate the imposed mobility against jamming interactions in two toy models---the totally asymmetric exclusion process and run-and-tumble particles, \emil{in the case of two or three particles}. We provide a framework which takes into account the energy-consumption required for self-propulsion, and address the question of how energy-efficient the emergent interactions are. Upon conditioning, run-and-tumble particles develop an alignment interaction and achieve a higher gain in efficiency than TASEP particles. A point of diminishing returns in efficiency is reached beyond a certain level of conditioning. With recourse to a general formula for the change in energy efficiency upon conditioning, we conclude that the most significant gains occur when there are large fluctuations in mobility to exploit. From a detailed comparison of the emergent effective interaction in a two- versus a three-body scenario, we discover evidence of a screening effect which suggests that conditioning can produce topological rather than metric interactions.

cond-mat.stat-mech↗

arXiv2019

Inviscid limit of the active interface equations

We present a detailed solution of the active interface equations in the inviscid limit. The active interface equations were previously introduced as a toy model of membrane-protein systems: they describe a stochastic interface where growth is stimulated by inclusions which themselves move on the interface. In the inviscid limit, the equations reduce to a pair of coupled conservation laws. After discussing how the inviscid limit is obtained, we turn to the corresponding Riemann problem: the solution of the set of conservation laws with discontinuous initial condition. In particular, by considering two physically meaningful initial conditions, a giant trough and a giant peak in the interface, we elucidate the generation of shock waves and rarefaction fans in the system. Then, by combining several Riemann problems, we construct an oscillating solution of the active interface with periodic boundaries conditions. The existence of this oscillating state reflects the reciprocal coupling between the two conserved quantities in our system.

cond-mat.stat-mech↗