arXiv subjects

William Dorrell

Publications and source records attributed to William Dorrell.

10 recordsLinked to original sources

arXiv2026

How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations

Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts, providing a basis for understanding and control. However, what exactly SAEs extract, and, correspondingly, the scientific conclusions we can draw from them, are not obvious. Empirically, the proof is in the pudding: SAEs learn interpretable features. Theoretically, we lack a clear account of what properties a 'concept' must satisfy for an SAE to extract it. There has been extensive identifiability work studying the conditions under which sparse coding recovers ground-truth features; however, these approaches tends to focus on simple data-generating models (e.g. sparse independent features) which poorly approximate the internet-swallowing language-model representations on which SAEs are trained. Here, avoiding data-generating models, we ask simply what properties any dictionary learning optimum must satisfy. Concretely, we extend local optimality analyses (Gribonval & Schnass, 2010) to the nonnegative joint-optimisation problem that vanilla SAEs approximate, and derive constraints relating optimal SAE features to their distributions. We use these constraints to explain a range of observed SAE behaviours - hierarchical splitting & absorption, the structure of residuals, and dense antipodal features - each reflecting how L1+nonnegativity interact with data to structure optimal dictionaries. Finally, we construct a novel large-dictionary convex problem and explore the wide atom-per-datapoint limit. In sum, we hope to tease model assumptions from unexpected observations, letting us learn more from SAEs' successes and provide principles for designing their successors.

q-bio.NC↗

arXiv2026

If Grid Cells are the Answer, What is the Question? A Review of Normative Grid Cell Theory

For 20 years the beautiful structure in the grid cell code has presented an attractive puzzle: what computation do these representations subserve, and why does it manifest so curiously in neurons. The first question quickly attracted an answer: grid cells subserve path-integration, the ability to keep track of one's position as you move about the world. Subsequent work has only solidified this link: bottom-up mechanistic models that perform path-integration match the measured neural responses, while experimental perturbations that selectively disrupt grid cell activity impair performance on path-integration dependent tasks. A more controversial area of work has been top-down normative modelling: why has the brain chosen to compute like this? Floods of ink have been spilt attempting to build a precise link between the population's objective and the measured implementation. The holy grail is a normative link with broad predictive power which generalises to other neural systems. We review this literature and argue that, despite some controversies, the literature largely agrees that grid cells can be explained as a (1) biologically plausible (2) high fidelity, non-linearly decodable code for position that (3) subserves path-integration. As a rare area of neuroscience with mature theoretical and experimental work, this story holds lessons for normative theories of neural computations, and on the risks and rewards of integrating task-optimised neural networks into such theorising.

q-bio.NC↗

arXiv2026

Convex Efficient Coding

Why do neurons encode information the way they do? Normative answers to this question model neural activity as the solution to an optimisation problem; for example, the celebrated efficient coding hypothesis frames neural activity as the optimal encoding of information under efficiency constraints. Successful normative theories have varied dramatically in complexity, from simple linear models (Atick & Redlich '90), to complex deep neural networks (Lindsay '21). What complex models gain in flexibility, they lose in tractability and often understandability. Here, we split the difference by constructing a set of tractable but flexible normative representational theories. Instead of optimising the neural activities directly, following Sengupta et al. '18, we optimise the representational similarity, a matrix formed from the dot products of each pair of neural responses. Using this, we show that a large family of interesting optimisation problems are convex. This family includes problems corresponding to linear and some non-linear neural networks, and problems from the literature not previously recognised as convex, such as modified versions of semi-nonnegative matrix factorisation or nonnegative sparse coding. We put these findings to work in three ways. First, we provide the first necessary and sufficient identifiability result for a form of semi-nonnegative matrix factorisation. Second, we show that if neural tunings are `different enough' then they are uniquely linked to the optimal representational similarity, partially justifying the use of single neuron tuning analysis in neuroscience. Finally, we use the tractable nonlinearity of some of our problems to explain why dense retinal codes, but not sparse cortical codes, optimally split the coding of a single variable into ON & OFF channels. In sum, we identify a space of convex problems, and use them to derive neural coding results.

q-bio.NC↗

arXiv2026

Mapping Connectomic Structure to Function(s) in Cerebellar-like Networks using Kernel Regression

Cerebellar-like networks, in which input activity patterns are separated by projection to a much higher-dimensional space before classification, are a recurring neurobiological motif, present in the cerebellum, dentate gyrus, insect olfactory system, and electrosensory system of the electric fish. Their relatively well-understood design presents a promising test-case for probing principles of biological learning. The circuits' expansive projections have long been modelled as random, enabling effective general purpose pattern separation. However, electron-microscopy studies have discovered interesting hints of structure in both the fly mushroom body and mouse cerebellum. Recent numerical work suggested that this non-random connectivity enables the circuit to prioritise learning of some, presumably natural, tasks over others. Here, rather than numerical results, we present a robust mathematical link between the observed connectivity patterns and the cerebellar circuit's learning ability. In particular, we extend a simplified kernel regression model of the system and use recent machine learning theory results to relate connectivity to learning. We find that the reported structure in the projection weights shapes the network's inductive bias in intuitive ways: functions are easier to learn if they depend on inputs that are oversampled, or on collections of neurons that tend to connect to the same hidden layer neurons. Our approach is analytically tractable and pleasingly simple, and we hope it continues to serve as a model for understanding the functional implications of other processing motifs in cerebellar-like networks.

q-bio.NC↗

arXiv2026

How much neuroscience does a neuroscientist need to know?

How much of the brain's learned algorithms depend on the fact it is a brain? We argue: a lot, but surprisingly few details matter. We point to simple biological details -- e.g. nonnegative firing and energetic/space budgets in connectionist architectures -- which, when mixed with the requirements of solving a task, produce models that predict brain responses down to single-neuron tuning. We understand this as details constraining the set of plausible algorithms, and their implementations, such that only `brain-like' algorithms are learned. In particular, each biological detail breaks a symmetry in connectionist models (scale, rotation, permutation) leading to interpretable single-neuron responses that are meaningfully characteristic of particular algorithms. This view helps us not only understand the brain's choice of algorithm but also infer algorithm from measured neural responses. Further, this perspective aligns computational neuroscience with mechanistic interpretability in AI, suggesting a more unified approach to studying the mechanisms of intelligence, both natural and artificial.

q-bio.NC↗

arXiv2022

Meta-Learning the Inductive Biases of Simple Neural Circuits

Training data is always finite, making it unclear how to generalise to unseen situations. But, animals do generalise, wielding Occam's razor to select a parsimonious explanation of their observations. How they do this is called their inductive bias, and it is implicitly built into the operation of animals' neural circuits. This relationship between an observed circuit and its inductive bias is a useful explanatory window for neuroscience, allowing design choices to be understood normatively. However, it is generally very difficult to map circuit structure to inductive bias. Here, we present a neural network tool to bridge this gap. The tool meta-learns the inductive bias by learning functions that a neural circuit finds easy to generalise, since easy-to-generalise functions are exactly those the circuit chooses to explain incomplete data. In systems with analytically known inductive bias, i.e. linear and kernel regression, our tool recovers it. Generally, we show it can flexibly extract inductive biases from supervised learners, including spiking neural networks, and show how it could be applied to real animals. Finally, we use our tool to interpret recent connectomic data illustrating its intended use: understanding the role of circuit features through the resulting inductive bias.

q-bio.NC↗

arXiv2022

Actionable Neural Representations: Grid Cells from Minimal Constraints

To afford flexible behaviour, the brain must build internal representations that mirror the structure of variables in the external world. For example, 2D space obeys rules: the same set of actions combine in the same way everywhere (step north, then south, and you won't have moved, wherever you start). We suggest the brain must represent this consistent meaning of actions across space, as it allows you to find new short-cuts and navigate in unfamiliar settings. We term this representation an `actionable representation'. We formulate actionable representations using group and representation theory, and show that, when combined with biological and functional constraints - non-negative firing, bounded neural activity, and precise coding - multiple modules of hexagonal grid cells are the optimal representation of 2D space. We support this claim with intuition, analytic justification, and simulations. Our analytic results normatively explain a set of surprising grid cell phenomena, and make testable predictions for future experiments. Lastly, we highlight the generality of our approach beyond just understanding 2D space. Our work characterises a new principle for understanding and designing flexible internal representations: they should be actionable, allowing animals and machines to predict the consequences of their actions, rather than just encode.

q-bio.NC↗

arXiv2020

Acoustic twisted bilayer graphene

Twisted van der Waals (vdW) heterostructures have recently emerged as an attractive platform to study tunable correlated electron systems. However, the quantum mechanical nature of vdW heterostructures makes their theoretical and experimental exploration laborious and expensive. Here we present a simple platform to mimic the behavior of twisted vdW heterostructures using acoustic metamaterials comprising of interconnected air cavities in a steel plate. Our classical analog of twisted bilayer graphene shows much of the same behavior as its quantum counterpart, including mode localization at a magic angle of about 1.1 degrees. By tuning the thickness of the interlayer membrane, we reach a regime of strong interactions more than three times higher than the feasible range of twisted bilayer graphene under pressure. In this regime, we find the magic angle as high as 6.01 degrees, corresponding to a far denser array of localized modes in real space and further increasing their interaction strength. Our results broaden the capabilities for cross-talk between quantum mechanics and acoustics, as vdW metamaterials can be used both as simplified models for exploring quantum systems and as a means for translating interesting quantum effects into acoustics.

cond-mat.mes-hall↗

arXiv2020

A differential Hebbian framework for biologically-plausible motor control

In this paper we explore a neural control architecture that is both biologically plausible, and capable of fully autonomous learning. It consists of feedback controllers that learn to achieve a desired state by selecting the errors that should drive them. This selection happens through a family of differential Hebbian learning rules that, through interaction with the environment, can learn to control systems where the error responds monotonically to the control signal. We next show that in a more general case, neural reinforcement learning can be coupled with a feedback controller to reduce errors that arise non-monotonically from the control signal. The use of feedback control can reduce the complexity of the reinforcement learning problem, because only a desired value must be learned, with the controller handling the details of how it is reached. This makes the function to be learned simpler, potentially allowing learning of more complex actions. We use simple examples to illustrate our approach, and discuss how it could be extended to hierarchical architectures.

q-bio.NC↗

arXiv2020

van der Waals metamaterials

The van der Waals heterostructures are a fertile frontier for discovering emergent phenomena in condensed matter systems. They are constructed by stacking elements of a large library of two-dimensional materials, which couple together through van der Waals interactions. However, the number of possible combinations within this library is staggering, and fully exploring their potential is a daunting task. Here we introduce van der Waals metamaterials to rapidly prototype and screen their quantum counterparts. These layered metamaterials are designed to reshape the flow of ultrasound to mimic electron motion. In particular, we show how to construct analogues of all stacking configurations of bilayer and trilayer graphene through the use of interlayer membranes that emulate van der Waals interactions. By changing the membrane's density and thickness, we reach coupling regimes far beyond that of conventional graphene. We anticipate that van der Waals metamaterials will explore, extend, and inform future electronic devices. Equally, they allow the transfer of useful electronic behavior to acoustic systems, such as flat bands in magic-angle twisted bilayer graphene, which may aid the development of super-resolution ultrasound imagers.

cond-mat.mes-hall↗