arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.13518

Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers

Claudio Altafini
View original source ↗Read source document ↗Export metadata ↓

Abstract

In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.

Explore related subjects

math.DScs.LGcs.SYeess.SY

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Claudio Altafini. 2026-10-09. Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers. https://arxiv.org/abs/2610.13518

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Decomposition results for multiplicative actions and applications

Motivated by partition regularity problems of homogeneous quadratic equations, we prove multiple recurrence and convergence results for multiplicative measure preserving actions with iterates given by rational sequences involving polynomials that factor into products of linear forms in two variables. We focus mainly on actions that are finitely generated, and the key tool in our analysis is a decomposition result for any bounded measurable function into a sum of two components, one that mimics concentration properties of pretentious multiplicative functions and another that mimics vanishing properties of aperiodic multiplicative functions. Crucial to part of our arguments are some new seminorms that are defined by a mixture of addition and multiplication of the iterates of the action, and we prove an inverse theorem that explicitly characterizes the factor that these seminorms naturally determine.

math.DS↗
arXiv2026

Shadowing in CR-Dynamical Systems

A CR-dynamical system is a pair $(X, G)$, where $X$ is a compact metric space and $G$ is a closed relation on $X$. We introduce four $(i,j)$-shadowing properties, obtained by independently varying the quantifiers in the definitions of pseudo-orbit and shadowing. These properties coincide with the classical shadowing property for single-valued systems, while $(2, 2)$-shadowing agrees with the established notion of shadowing for set-valued dynamical systems. We investigate how the classical characterisations of shadowing for the identity map, periodic maps and isometries extend to closed relations. In particular, we obtain rigidity results for $(2, 1)$-shadowing and positive results for $(1, 2)$-shadowing on totally disconnected spaces. Finally, whenever the infinite Mahavier product is non-empty, we prove that a closed relation has the $(2, 2)$-shadowing property if, and only if, its associated Mahavier dynamical system has the shadowing property. This removes the surjectivity hypothesis from a recent corresponding result for set-valued dynamical systems.

math.DS↗
arXiv2026

Upper bound for the number of limit cycles for symmetric piecewise linear differential equations with three zones

The study of the periodic dynamics of the family of continuous observable and non-controllable three-dimensional symmetric piecewise linear systems with three zones considered here can be reduced to the study of periodic solutions of the piecewise differential equation $x'=ax+(b-a)\mathop{\rm sat}(x)+μ\sin t$, where $\operatorname{sat}$ stands for the normalized saturation function. This paper proves that the number of limit cycles (that is, isolated periodic solutions) of the equation is uniformly bounded independently of $a,b,μ$. Moreover, when $ab>0$, the equation has exactly one limit cycle for every $μ>0$. When $ab<0$, it has exactly three limit cycles for sufficiently small $μ>0$ and exactly one for sufficiently large $μ$. When $ab<0$, using Melnikov theory for a deformation of the equation, we prove that, for each fixed $μ>0$ away from two critical values and for sufficiently small nonzero values of the perturbation parameter, there are exactly three, five or one limit cycles, depending on $μ$. This strengthens the conjecture that the equation has at most five limit cycles.

math.DS↗