arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2608.30417

No Equivariant Architecture Covers All Equivariant Attention

Tīkun Ông
View original source ↗Read source document ↗Export metadata ↓

Abstract

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For $G=D_4$ acting on $C$ copies of the regular representation as the token feature space, we show that there are $Ω(C^{64})$ components for eight attention heads.

Explore related subjects

cs.LGmath.AGmath.RT

Keep this discovery

BibTeXRIS

Tīkun Ông. 2026-08-31. No Equivariant Architecture Covers All Equivariant Attention. https://arxiv.org/abs/2608.30417

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.LG subjectmath.AG subjectmath.RT subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

The Alexander-Hirschowitz theorem for neurovarieties

We study the dimension and identifiability of neurovarieties associated to polynomial neural networks. We give an independent geometric proof that the linear bounds $d_i\geq 2n_i-1$ on the activation degrees imply non defectiveness for any number of outputs, a dimension statement previously obtained from finite identifiability. The proof is based on a direct analysis of the differential of the parameterization. We also investigate secant and Grassmann-secant obstructions outside this range and prove global identifiability for multi-output architectures under the same degree bounds.

math.AG↗
arXiv2026

Quiver Semistability and Structured Kalman Decompositions for Networked Linear Dynamical Systems

We introduce new notions of controllability and observability for networked linear time-invariant (LTI) systems based on $σ$-semistability of quiver representations. Utilizing King's criterion for $σ$-semistability, we define a network generalization of the Kalman decomposition for networked LTI systems, which systematically decomposes the local and interconnection dynamics while respecting the underlying network structure. Furthermore, we present efficient algorithms for deciding the proposed controllability and observability of a given networked LTI system and for finding the Kalman-type decomposition. We also show efficient algorithms for deciding the $σ$-semistability of representations of acyclic quivers with self-loops if the weight $σ$ has the same sign for all vertices with self-loops. Such quiver representations and weights arise from networked LTI systems.

math.OC↗
arXiv2026

Learning Fast Monomial Orders for Gröbner Basis Computations

The efficiency of Gröbner basis computation, the standard engine for solving systems of polynomial equations, depends on the choice of monomial ordering. Despite a near-continuum of possible monomial orders, most implementations rely on static heuristics such as GrevLex, guided primarily by expert intuition. We address this gap by casting the selection of monomial orderings as a reinforcement learning problem over the space of admissible orderings. Our approach leverages domain-informed reward signals that accurately reflect the computational cost of Gröbner basis computations and admits efficient Monte Carlo estimation. Experiments on benchmark problems from systems biology and computer vision show that the resulting learned policies consistently outperform standard heuristics, yielding substantial reductions in computational cost. Moreover, we find that these policies resist distillation into simple interpretable models, providing empirical evidence that deep reinforcement learning allows the agents to exploit non-linear geometric structure beyond the scope of traditional heuristics.

cs.SC↗