arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2207.04932

Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks

Xia Xiong·Yong-Cong Chen·Chunxiao Shi·Ping Ao
View original source ↗Read source document ↗Export metadata ↓

Abstract

Stochastic gradient descent (SGD), a widely used algorithm in deep-learning neural networks has attracted continuing studies for the theoretical principles behind its success. A recent work reports an anomaly (inverse) relation between the variance of neural weights and the landscape flatness of the loss function driven under SGD [Feng & Tu, PNAS 118, 0027 (2021)]. To investigate this seemingly violation of statistical physics principle, the properties of SGD near fixed points are analysed via a dynamic decomposition method. Our approach recovers the true "energy" function under which the universal Boltzmann distribution holds. It differs from the cost function in general and resolves the paradox raised by the the anomaly. The study bridges the gap between the classical statistical mechanics and the emerging discipline of artificial intelligence, with potential for better algorithms to the latter.

Explore related subjects

nlin.AOcs.LG

Keep this discovery

BibTeXRIS

Xia Xiong, Yong-Cong Chen, Chunxiao Shi, Ping Ao. 2022-07-11. Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks. https://doi.org/10.1088/0256-307x%2F40%2F8%2F080202

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Reconstructing the information processing capacity of physical systems from noisy observations

Driven dynamical systems can compute when their transient states encode complex transformations of past inputs. The information processing capacity (IPC) framework allows for a detailed accounting of these computational properties, however its interpretation in noisy systems has remained incomplete. In this work, we clarify how noise affects the IPC and how one can reconstruct the noiseless IPC. First, we show how to distinguish the dynamics of an unperturbed system from the noise-free component of the stochastic dynamics: The IPC measured for responses averaged over noise realizations is in general not the same as the IPC of the unperturbed system. We explicitly demonstrate that noise can redistribute computational capacity and sometimes even enhance performance on particular tasks, rather than merely degrading a fixed computation. We then introduce covariance reconstruction by orthogonal projection (CROP), which reconstructs the covariance and IPC of the noise-free component directly from noisy observations, without requiring a detailed model of either the system or the noise. At fixed total measurement budget, numerical tests on a classical nonlinear reservoir show that CROP estimates the noise-free IPC more accurately than the standard practice of ensemble averaging over repeated trials. We find the same advantage in a quantum reservoir subject to unavoidable measurement noise. Our results provide a general route to recovering the computational structure of noisy physical systems from finite observations.

nlin.AO↗
arXiv2026

Double explosive transitions in adaptive multilayer networks with higher-order interactions

Can asymmetry between two interacting networks fundamentally change how they synchronize? We identify double explosive transitions in the forward direction, backward direction, or a combination thereof, with single or double hysteresis loops in an adaptive bilayer multiplex network of Kuramoto oscillators with pairwise and three-body interactions and asymmetric phase lags. Using the Ott-Antonsen reduction, we derive a low-dimensional system and perform a stability analysis. The reduced model accurately captures the full microscopic dynamics and enables analytical expressions for the bifurcation points. Systematic mapping across multiple parameter planes reveals eight distinct synchronization regimes. The relative ordering of saddle-node and pitchfork bifurcation points---controlled by phase-lag asymmetry, cross-layer adaptation, and the higher-order interaction strength---creates two distinct coherent branches (weak and strong), giving rise to double explosive transitions. Crucially, phase-lag asymmetry acts as a robust control knob: while symmetric phase lags suppress explosive transitions, layer-specific differences promote multistability and double explosive transitions. The higher-order interaction strength $K_2$ and adaptation strengths $q$, $p$, and $h$ further modulate these transitions in a complex, parameter-dependent manner. Excellent agreement between analytical predictions and numerical simulations confirms the robustness of our reduced description.

nlin.AO↗
arXiv2026

Spatio-temporal structures in frog chorus with two species examined by laboratory experiments and mathematical modeling

Synchronization can be observed in various systems in physics and biology. The choruses of male frogs are known as an example of biological synchronization in which the well-organized temporal structure, i.e., anti-phase synchronization between neighbors, is realized. Given that male frogs produce sounds to advertise their territories to competitors, the dynamics of phases and spatial coordinates should be mutually coupled in the frog choruses. In this study, we examined the spatio-temporal dynamics in the choruses consisting of male Japanese tree frogs and other acoustic animals. First, we carried out playback experiments using actual frogs and observed that male Japanese tree frog synchronized in anti-phase with the stimuli of a similar frequency but did not synchronize with the stimuli of a much different frequency. Second, we modeled the choruses with two species as a system of coupled mobile oscillators and numerically evaluated how the spatio-temporal structure depends on the similarity of call frequencies. Numerical simulations of the model showed that (1) the two-cluster antisynchronization is established in the same species when the distributions of call frequencies are much different between two species and (2) the two-cluster antisynchronization is disturbed when the distributions of call frequencies are similar. These results highlight the occurrence of various spatio-temporal patterns in the proposed model, indicating the importance of repulsive effects with different weights on the variation of the spatio-temporal patterns.

nlin.AO↗