Pixel-Based Similarities as an Alternative to Neural Data for Improving Convolutional Neural Network Adversarial Robustness

Abstract

Convolutional Neural Networks (CNNs) excel in many visual tasks but remain susceptible to adversarial attacks-imperceptible perturbations that degrade performance. Prior research reveals that brain-inspired regularizers, derived from neural recordings, can bolster CNN robustness; however, reliance on specialized data limits practical adoption. We revisit a regularizer proposed by Li et al. (2019) that aligns CNN representations with neural representational similarity structures and introduce a data-driven variant. Instead of a neural recording-based similarity, our method computes a pixel-based similarity directly from images. This substitution retains the original biologically motivated loss formulation, preserving its robustness benefits while removing the need for neural measurements or task-specific augmentations. Notably, this data-driven variant provides the same robustness improvements observed with neural data. Our approach is lightweight and integrates easily into standard pipelines. Although we do not surpass cutting-edge specialized defenses, we show that neural representational insights can be leveraged without direct recordings. This underscores the promise of robust yet simple methods rooted in brain-inspired principles, even without specialized data, and raises the possibility that further integrating these insights could push performance closer to human levels without resorting to complex, specialized pipelines.