The Standard Deviation in VICReg Buys a Safety Margin

Hawar Dizayee

Home/Archive/arXiv

arXiv · 2610.32630

The Standard Deviation in VICReg Buys a Safety Margin

Hawar Dizayee

View original source ↗Read source document ↗Export metadata ↓

Abstract

VICReg prevents collapse with a hinge on the standard deviation of each embedding dimension, and its authors argue from the gradient that a hinge on the variance would fail. To our knowledge, this claim has not been tested experimentally. We test it in a controlled setting (ResNet-18, CIFAR-10 and CIFAR-100), changing only the hinge and sweeping its weight $μ$ over five orders of magnitude. At the default weight ($μ= 1$), the two hinges reach the same linear-probe accuracy on CIFAR-10 (about 89.5%). Both collapse below a threshold weight, but the standard-deviation hinge's threshold is 35-130$\times$ lower. CIFAR-100 shows a consistent gap. In training, the variance hinge's gradient fades as the feature variance shrinks, while the standard-deviation hinge's does not until that variance is very small. The standard-deviation hinge therefore buys a safety margin: it keeps training at weights far below the point where the variance hinge collapses.

Explore related subjects

cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Hawar Dizayee. 2026-10-23. The Standard Deviation in VICReg Buys a Safety Margin. https://arxiv.org/abs/2610.32630

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Maximum Likelihood on the Joint (Data, Condition) Distribution for Solving Ill-Posed Problems with Conditional Flow Models

In the absence of conditioning information, normalizing flow models are trained using maximum likelihood. Previous extensions to conditional settings have abandoned the simple maximum-likelihood approach, as obtaining the full joint distribution of data and conditions for training purposes (or equivalently, the true conditional distribution for each point in the training dataset) is impossible. However, under special conditions that nevertheless apply to a wide class of useful problems, it is possible to obtain a simple, computationally cheap proxy with the same global minimum as the conditional likelihood. Conditional flow models trained using this proxy do not require explicit knowledge of the conditional distribution, auxiliary networks or other specific architecture, or additional loss terms beyond maximum likelihood. They provide a natural and powerful way to solve a class of ill-posed inverse problems. I demonstrate these properties on easily visualized toy problems, then use the method to successfully generate class-conditional images and to reconstruct semantically degraded images via super-resolution.

cs.LG↗

arXiv2026

FC-PINO: High Precision Physics-Informed Neural Operators via Fourier Continuation

The physics-informed neural operator (PINO) is a machine learning paradigm that has demonstrated promising results for learning solutions to partial differential equations (PDEs). It leverages the Fourier Neural Operator to learn solution operators in function spaces and leverages physics losses during training to penalize deviations from known physics laws. Spectral differentiation provides an efficient way to compute derivatives for the physics losses, but it inherently assumes periodicity. When applied to non-periodic functions, this assumption can lead to significant errors, including Gibbs phenomena near domain boundaries which degrade the accuracy of both function representations and derivative computations. To overcome this limitation, we introduce the FC-PINO (Fourier-Continuation-based Physics-Informed Neural Operator) architecture which extends the accuracy and efficiency of PINO and spectral differentiation to non-periodic and non-smooth PDEs. In FC-PINO, we propose integrating Fourier continuation into the PINO framework, and test two different continuation approaches: FC-Legendre and FC-Gram. By transforming non-periodic signals into periodic functions on extended domains in a well-conditioned manner, Fourier continuation enables fast and accurate derivative computations. This approach avoids the discretization sensitivity of finite differences and the memory overhead of automatic differentiation. We demonstrate that standard PINO fails (without padding) or struggles (even with padding) to solve non-periodic and non-smooth PDEs with high precision, across challenging benchmarks. In contrast, the proposed FC-PINO provides accurate, robust, and scalable solutions, substantially outperforming PINO alternatives, and demonstrating that Fourier continuation is critical for extending PINO to a wider range of PDE problems when high-precision solutions are needed.

cs.LG↗

arXiv2026

Learning from Hypervectors: A Survey on Hyperdimensional Encoding

Hyperdimensional computing (HDC) is an emerging computing paradigm that imitates the brain's structure to offer a powerful and efficient processing and learning model. In HDC, the data are encoded with long vectors, called hypervectors, which typically range from 1,000 to 10,000 in length. The literature provides several encoding methods to generate orthogonal or correlated hypervectors, based on the application. Existing surveys focus on overall HDC system aspects like inputs, primary computations, and outputs. However, this study takes a more specific approach. It zeroes in on the HDC system input and the generation of hypervectors, directly influencing the hypervector encoding. This survey brings together various hypervector generation and encoding methods, exploring the limitations, challenges, and potential benefits they entail.

cs.LG↗