arXiv Science⌕ Search

arXiv subjects

Israel Mason-Williams

Publications and source records attributed to Israel Mason-Williams.

6 recordsLinked to original sources

Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations

Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model -- they are called robust -- while others cannot -- they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work

cs.LG↗

A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

We release a battery of 28 propensities for AI research, with expert-written rubrics, synthetic datasets, and annotated demand intervals, built with the method of Romero-Alvarado et al. (2026). We selected a wide collection of generic propensities that may influence the behavior of AI systems and humans. They are inspired by psychometric, psychological, and behavioral research, so that experimental results can be understood in the context of familiar constructs previously used also for human behavior research. The rubrics are useful for annotating existing benchmarks whose items may be relevant or not for one or more propensities. For easier profiling, however, the synthetic datasets are defined in such a way that they should be sensitive to one propensity only, allowing for a quick prototyping of propensity profiles using the battery of datasets. We also outline directions for future propensity research.

cs.AI↗

A Function-Centric Perspective on Flat and Sharp Minima

Flat minima are strongly associated with improved generalisation in deep neural networks. However, this connection has proven nuanced in recent studies, with both theoretical counterexamples and empirical exceptions emerging in the literature. In this paper, we revisit the role of sharpness in model performance and examine sharpness as a function-dependent property rather than as a direct indicator of poor generalisation. We conduct extensive empirical studies ranging from single-objective optimisation, synthetic non-linear binary classification tasks, to modern image classification tasks. In single-objective optimisation, we show that flatness and sharpness are relative to the function being learned: equally optimal solutions can exhibit markedly different local geometry. In synthetic non-linear binary classification tasks, we show that increasing decision-boundary tightness can increase sharpness even when models generalise perfectly, suggesting that sharpness is not reducible to memorisation alone. Finally, in large-scale experiments, we find that sharper minima often emerge when models are regularised (e.g., via weight decay, data augmentation, or SAM), and can coincide with better generalisation, calibration, robustness, and functional consistency. Our findings are consistent with a function-centric interpretation of minima geometry: flatness should not be treated as a universal proxy for generalisation, and sharper minima can reflect desirable inductive biases associated with improved predictive behaviour, calling for a function-centric reappraisal of minima geometry.

cs.LG↗

Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generation

Sound design workflows frequently oscillate between time-consuming library searches and the complexity of procedural synthesis, with practitioners typically relying on disconnected tools to address each challenge separately. This paper introduces Quality Audio Prototyping (QuAP), a working prototype that unifies content-based audio retrieval and procedural sound generation within a single interface, reducing the procedural distance between a narrative concept and its sonic realisation. QuAP integrates a similarity-based retrieval engine with real-time procedural audio models, complemented by a rule-based assistant that provides perceptually informed parameter guidance, offering definitions and recommendations derived from empirical optimisation rather than requiring prior synthesis knowledge. Preliminary evaluation confirms the viability of this approach: subjective assessment demonstrated statistically significant quality improvements in five of six embedded synthesis models, and an encoder ablation study established the preferred retrieval architecture on a sound effect dataset. A user evaluation with 16 practitioners confirmed the tool's workflow utility, with all participants agreeing that the parameter assistant preserved creative agency while lowering the barrier to procedural interaction.

cs.SD↗

A Functional Perspective on Knowledge Distillation in Neural Networks

Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge distillation and the resulting knowledge transfer from a functional perspective, decoupling compression from architectural reduction to provide an improved understanding of knowledge distillation. We employ a control-driven experimental protocol with hypothesis testing and random control distillation to isolate and understand knowledge transfer mechanisms across data modalities. To test the breadth and limits of our analyses, we study self-distillation, standard distillation, feature-map matching variants, distillation scaling laws across model sizes, and the impact of temperature on knowledge transfer. We find statistically supported knowledge transfer in some modalities and architectures; however, the extent of this transfer is less pronounced than anticipated, even under conditions that maximise knowledge sharing. Notably, in cases of significant functional transfer, we identify a consistent and severe asymmetric transfer of negative knowledge to the student, raising safety concerns for knowledge distillation. Across 22 experimental setups, 9 architectures, and 7 datasets, our results suggest that knowledge distillation functions less as a robust compression-by-transfer mechanism and more as a data-dependent regulariser whose transfer component is biased towards negative asymmetric transfer.

cs.LG↗

Reproducibility: The New Frontier in AI Governance

AI policymakers are responsible for delivering effective governance mechanisms that can provide safe, aligned and trustworthy AI development. However, the information environment offered to policymakers is characterised by an unnecessarily low Signal-To-Noise Ratio, favouring regulatory capture and creating deep uncertainty and divides on which risks should be prioritised from a governance perspective. We posit that the current publication speeds in AI combined with the lack of strong scientific standards, via weak reproducibility protocols, effectively erodes the power of policymakers to enact meaningful policy and governance protocols. Our paper outlines how AI research could adopt stricter reproducibility guidelines to assist governance endeavours and improve consensus on the AI risk landscape. We evaluate the forthcoming reproducibility crisis within AI research through the lens of crises in other scientific domains; providing a commentary on how adopting preregistration, increased statistical power and negative result publication reproducibility protocols can enable effective AI governance. While we maintain that AI governance must be reactive due to AI's significant societal implications we argue that policymakers and governments must consider reproducibility protocols as a core tool in the governance arsenal and demand higher standards for AI research. Code to replicate data and figures: https://github.com/IFMW01/reproducibility-the-new-frontier-in-ai-governance

cs.AI↗