arXiv Science⌕ Search

arXiv · 2611.13146

Everything in its right place: attention is a radio head

Abstract

Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Lluís Arola-Fernández, Aissam Khadraoui, Alex Arenas. 2026-11-09. Everything in its right place: attention is a radio head. https://arxiv.science/abs/2611.13146

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Transient Signal Propagation in Heterogeneous Networks under Various Inputs

Transient signals in neural, biological and technological networks rarely follow a single route. Their strength and timing depend on where the input enters, on the intrinsic dynamics of each node visited, and on the feedforward and recurrent pathways that carry it. Previous theory has clarified important limits of signal propagation, but often for a single input type, averaged network structure, or asymptotic regimes. It has remained unclear how finite heterogeneous networks transform sustained, impulsive, finite-duration and stochastic inputs into node-resolved response timescales. Here, we develop a theory for network dynamics near stable operating states that answers this question. The framework identifies a canonical set of gains and timescales shared across deterministic and stochastic forcing and expresses them through matrix resolvents, spectra and directed walks. In positive, sufficiently damped networks, the walk representation gives a simple physical picture: response strength accumulates multiplicatively along a single path, whereas response time accumulates additively, through local intrinsic time constants. Competing pathways, feedback and inhibition further reshape propagation by amplifying, suppressing, reversing or delaying signals. We derive perturbative criteria that specify when these signatures, and the resulting node rankings, persist beyond the positive regime. Applied to a Wilson--Cowan model of the primate cortex, the theory links structural connectivity, excitatory--inhibitory balance and local biophysical timescales to hierarchical response timescales. Our results provide a mechanistic framework for input-dependent timescales in finite heterogeneous networks.

physics.soc-ph↗

Exploration stabilizes learning dynamics and promotes cooperation in stochastic games

Understanding and stabilizing collective learning dynamics is a central challenge in stochastic games. Although exploration is widely used in learning algorithms, its effects on the coupled dynamics of agents and their environment remain insufficiently understood. In this work, we derive analytical equations for entropy-regularized actor-critic learning in stochastic games by separating the timescales of value estimation and policy adaptation. Without exploration, the system preserves a weighted volume in the policy phase space, ruling out asymptotically stable fully mixed equilibria. We provide a unified characterization of exploration's effects on objective curvature, preference dynamics in log-ratio coordinates, and the forgetting of past action preferences. Exploration reduces local volume expansion and can stabilize learning dynamics. In a two-state Matching Pennies game, it suppresses chaotic learning and promotes convergence to a mixed equilibrium. In a Prisoner's Dilemma with environmental feedback, intermediate exploration eliminates the defective attractor and expands the basin of attraction of the cooperative equilibrium. Our findings reveal how exploration shapes collective behavior, informing the design of cooperative and resilient multi-agent systems.

physics.soc-ph↗

Reconstructing the Probability Measure of a Multi-group Curie-Weiss Model with Interacting Groups

We study the problem of reconstructing the probability measure of a multi-group version of the Curie-Weiss or mean-field model of ferromagnetism from a sample of the voting behaviour of the population. While originally used to study phase transitions in statistical mechanics, the Curie-Weiss model has been applied to study phenomena where many agents interact with each other, in particular in case of a heterogeneous population with identifiable subpopulations. The degree of social cohesion within social groups manifests in the way the members of the group influence each others' decisions as well as how they behave under outside influence from voters belonging to another group. While the maximum likelihood estimator of the coupling parameters has desirable statistical properties in theory, computational challenges make applications to larger populations impractical. Therefore, we analyse an estimator based on asymptotic approximations to the behaviour of the Curie-Weiss model valid for large populations. Due to the wide application of models such as Curie-Weiss, the estimator is potentially useful in disciplines such as political science, sociology, automated voting, and preference aggregation.

physics.soc-ph↗