arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.28587

Two $\operatorname{ReLU}$ Layers is All You Need

Florestan Brunck·Christoph Hertrich·Kilian Ruess·Jack Stade·Amir Yehudayoff
View original source ↗Read source document ↗Export metadata ↓

Abstract

We prove that every $\operatorname{ReLU}$ network can be exactly simulated by a $\operatorname{ReLU}$ network with two hidden layers. We present two proofs: one that is based on fillings in homological algebra and a direct combinatorial proof.

Explore related subjects

cs.CCcs.CGcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Florestan Brunck, Christoph Hertrich, Kilian Ruess, Jack Stade, Amir Yehudayoff. 2026-10-20. Two $\operatorname{ReLU}$ Layers is All You Need. https://arxiv.org/abs/2610.28587

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Separating Circuits: Switching Lemmas and Random Restrictions

Non-uniform circuit complexity classifies Boolean functions by the size and depth of the gate networks required to compute them. Because $\mathsf{P} \subseteq \mathsf{P/poly}$, proving super-polynomial circuit lower bounds for $\mathsf{NP}$ problems offers a combinatorial route toward separating $\mathsf{P}$ from $\mathsf{NP}$, formalized by the Karp-Lipton theorem ($\mathsf{NP} \subseteq \mathsf{P/poly} \implies \mathsf{PH} = Σ_2^p$). While general circuit lower bounds for explicit functions in $\mathsf{NP}$ remain linear, exponential lower bounds hold for bounded-depth circuits via random restrictions and polynomial approximations. We give a self-contained treatment of both paradigms and their consequences. First, we establish Shannon's counting lower bound, Lupanov's $(1+o(1))2^n/n$ synthesis upper bound, and the Karp-Lipton theorem via SAT self-reducibility. Second, we prove Håstad's Switching Lemma via Razborov's canonical decision-tree encoding, showing that a $k$-CNF or $k$-DNF under a $p$-random restriction collapses to decision-tree depth less than $s$ except with probability at most $(5pk)^s$, and derive the optimal $2^{Ω(n^{1/(d-1)})}$ lower bound (and matching $O(n 2^{n^{1/(d-1)}})$ upper bound) for Parity in depth-$d$ $\mathsf{AC}^0$. Third, we present the Razborov-Smolensky method over $\mathbb{F}_p$, proving that $\mathrm{MOD}_q \notin \mathsf{AC}^0[p]$ for distinct primes $p \neq q$ and establishing a $2^{Ω(n^{1/(2d)})}$ lower bound for Parity against $\mathsf{AC}^0[3]$. Finally, we examine structured restrictions for $k$-Clique and average-case depth hierarchies, four consequences of the Switching Lemma -- oracle separations of $\mathsf{PH}$ from $\mathsf{PSPACE}$, Linial-Mansour-Nisan Fourier concentration, Nisan-Wigderson pseudorandom generators, and $\mathsf{AC}^0$-SAT algorithms -- and the Razborov-Rudich Natural Proofs barrier.

cs.CC↗
arXiv2026

Classifying Approximation Algorithms: Understanding the APX Complexity Class

Many NP-hard optimization problems that are equivalent under Karp reductions behave very differently when we ask for polynomial-time approximations. This paper studies the complexity class APX, a subclass of NPO consisting of optimization problems that admit polynomial-time algorithms with constant-factor approximation guarantees. We first review the formal structure of NPO, define L-reductions and PTAS-reductions, and prove that L-reductions preserve polynomial-time approximation schemes (PTAS). Using the Probabilistically Checkable Proofs (PCP) theorem as a starting point, we present self-contained L-reductions establishing the APX-completeness of Max-3SAT, Max-2SAT, Max-NAE-3SAT, and Max-Cut, and we situate APX within the broader hierarchy FPTAS, PTAS, APX, logAPX, and polyAPX. We then examine structural consequences of the polynomial hierarchy for self-improvability and APX-intermediate problems such as Min-Bin-Packing, and review how APX-hardness arises in Bayesian optimal mechanism design for three-bidder single-item auctions. Turning to algorithm design, we analyze three core techniques across the APX and logAPX boundary: linear programming relaxation with Raghavan-Thompson randomized rounding for multicommodity congestion minimization, Goemans-Williamson semidefinite programming rounding for Max-Cut, and greedy, dual-fitting, and randomized rounding methods for Min-Set-Cover. We close by discussing bounded-query approximation oracles and recent constant-factor and sub-3/2 advances on the asymmetric and metric traveling salesman problems.

cs.CC↗
arXiv2026

Fine-Grained $\mathsf{AC}^0$ Lower Bounds for $k$-OV, $k$-XOR, and $k$-SUM via Colored Subgraph Isomorphism

We study the circuit cost of selecting $k$ objects satisfying orthogonality, zero XOR, or modular zero SUM. Our main lower bounds in nonuniform $\mathsf{AC}^0$ use a common framework of depth-zero projections from colored subgraph isomorphism. The projections use dimension, row count, or bit width $O(k\log n)$ and preserve circuit size, depth, and gate orientation. OV inputs satisfy pairwise distinctness within each array. Our direct XOR bounds require odd $k$, while SUM admits both parities. Stronger lower bounds for suitable source patterns transfer through the same framework to all three targets. For every fixed depth $d$ and sufficiently large fixed $k$, we prove unconditional lower bounds $n^{Ω(k)}$ for OV and $(n/k)^{Ω(k)}$ for XOR and SUM, with an absolute constant in the exponent independent of $d$ and $k$; only the threshold on $n$ may depend on $(d,k)$. For $k\to\infty$ with $k=n^{o(1)}$, combining our projections with Beame's small-clique lower bound yields $n^{Ω_d(\min\{\sqrt{k},\log n\})}$ at every fixed depth $d$. We strengthen the exponent to $Ω(k)$ at depth two in both orientations and at depth three with a top disjunction. The depth-three result allows unrestricted bottom fan-in and literal polarity; its proof bounds the probability that a planted copy forms a minterm of an arbitrary fixed CNF over a random background.

cs.CC↗