Zheng Zhang | arXiv Science

arXiv2026

Representation Redundancy and Structural Complexity in Finite-Field Inversion

The representation chosen for a mathematical operation can affect both its algebraic form and its empirical learning difficulty. We study this phenomenon for inversion over $\mathbb F_{2^n}$, with field elements expressed in varying ordered $\mathbb F_2$-bases. We prove that two ordered bases induce the same coordinate inversion map if and only if they belong to the same Galois orbit. Since every orbit has size $n$, the correspondence between ordered bases and distinct inversion maps is exactly $n$-to-one. We then analyze three Boolean formulations of inversion. The reference formulation has algebraic degree $n-1$ and joint ANF leap $1$, the mixed representation formulation has degree $2(n-1)$ and joint ANF leap $2$, and the complete raw formulation has degree at most $3(n-1)$ and joint ANF leap at least $n$. Exhaustive computations agree with the theoretical results and bounds in the cases considered. Controlled experiments with multilayer perceptrons show the same ordering in learning difficulty, while Galois orbit redundancy provides only a limited generalization benefit under the tested conditions. These results show that exact redundancy among representations can coexist with changes in Boolean structure and learning behavior when the representation is exposed as part of the input.

cs.LG↗

arXiv2026

GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning

Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per mini-batch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. We prove that GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO. Across RoBERTa-large, Llama3-8B, and OPT-13B over multiple tasks, GRZO improves average accuracy on Llama3-8B by $+3.0$ over MeZO at $23\%$ lower peak GPU memory; as a drop-in replacement for the MeZO core, it lifts sparse, low-rank, and quantized ZO variants by $+6.0$ on average.

cs.LG↗