arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.25998

Internalizing Chain-of-thought into Looped Transformers

Kehang Zhou·Pinyan Lu
View original source ↗Read source document ↗Export metadata ↓

Abstract

Chain-of-thought (CoT) enables constant-bit-size transformers to perform general-purpose computation through generated tokens (Li and Wang, 2025a), whereas looped transformers carry out the same computation by recurrence in hidden states (latent thought). We construct a family of constant-bit-size looped transformers that interpolates between CoT and latent thought. For a prescribed loop count $L=L(n)$ and a fixed integer $k\ge1$ satisfying $s(n)^{1/k}=Ω(L)$, our transformer simulates a time-$t(n)$, space-$s(n)$ Turing machine using a context window of $O(s(n)/L)$ and $O(t(n)s(n)^{1/k}/L)$ generated tokens, reducing both resources by a factor of $L$ relative to the corresponding unlooped queue-machine simulation. The construction also holds with softmax attention at constant bit-size, under the finite-precision kernel specified in this paper.

Explore related subjects

cs.CCcs.DScs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kehang Zhou, Pinyan Lu. 2026-10-19. Internalizing Chain-of-thought into Looped Transformers. https://arxiv.org/abs/2610.25998

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

The Computational Complexity of Counting Linear Regions in ReLU Neural Networks

An established measure of the expressive power of a given ReLU neural network is the number of linear regions into which it partitions the input space. There exist many different, non-equivalent definitions of what a linear region actually is. We systematically assess which papers use which definitions and discuss how they relate to each other. We then analyze the computational complexity of counting the number of such regions for the various definitions. Generally, this turns out to be an intractable problem. We prove NP- and #P-hardness results already for networks with one hidden layer and strong hardness of approximation results for two or more hidden layers. Finally, on the algorithmic side, we demonstrate that counting linear regions can at least be achieved in polynomial space for some common definitions.

cs.CC↗
arXiv2026

Intractability of Hilbert's Nullstellensatz implies algebraic hardness of permanent

We study the logical relation of the P-NP separation conjecture in the Blum-Shub-Smale-model over the complex numbers with the P-NP separation conjecture in Valiant's algebraic model. This amounts to comparing Hilbert's Nullstellensatz Problem, that is, deciding feasibility of a given system of polynomial equations over the complex numbers, with the problem of evaluating the permanent of a given complex matrix. We compare the respective uniform models of computations and prove that $P_C\ne NP_C$ in the Blum-Shub-Smale-model over $C$ implies the separation $VP^0(u)\ne VNP^0(u)$ of the uniform versions of Valiant's constant-free complexity classes over $C$. For the nonuniform models we show the analogous implication: the separation $P^0_C(nu)\ne NP^0_C(nu)$ of the nonuniform, constant-free Blum-Shub-Smale classes over $C$ implies the separation $VP^0\ne VNP^0$ of Valiant's constant-free complexity classes over $C$. In the reverse direction, we conjecture that $VNP_C\not\subseteq\overline{VP}_C$ implies that $P_C(nu)\ne NP_C(nu)$.

cs.CC↗
arXiv2026

Tropical Circuits with Scalar Multiplication Gates

We study tropical circuits with scalar multiplication gates, that is, algebraic circuits whose gates implement $\max$, $+$, or multiplication with a positive constant. For such circuits, we prove exponential size lower bounds for computing maximum weight directed spanning trees and maximum weight bipartite perfect matchings. As a corollary, we obtain an exponential size separation between monotone and non-monotone maxout neural networks, which generalize the popularly used ReLU neural networks. One conclusion from this is that neural network models with enforced convexity constraints, such as input-convex neural networks (ICNNs), sometimes need to be exponentially larger than their unrestricted counterparts in order to express the same functions.

cs.CC↗