arXiv subjects

Lyle Muller

Publications and source records attributed to Lyle Muller.

15 recordsLinked to original sources

arXiv2025

Traveling Waves Integrate Spatial Information Through Time

Traveling waves of neural activity are widely observed in the brain, but their precise computational function remains unclear. One prominent hypothesis is that they enable the transfer and integration of spatial information across neural populations. However, few computational models have explored how traveling waves might be harnessed to perform such integrative processing. Drawing inspiration from the famous "Can one hear the shape of a drum?" problem -- which highlights how normal modes of wave dynamics encode geometric information -- we investigate whether similar principles can be leveraged in artificial neural networks. Specifically, we introduce convolutional recurrent neural networks that learn to produce traveling waves in their hidden states in response to visual stimuli, enabling spatial integration. By then treating these wave-like activation sequences as visual representations themselves, we obtain a powerful representational space that outperforms local feed-forward networks on tasks requiring global spatial context. In particular, we observe that traveling waves effectively expand the receptive field of locally connected neurons, supporting long-range encoding and communication of information. We demonstrate that models equipped with this mechanism solve visual semantic segmentation tasks demanding global integration, significantly outperforming local feed-forward models and rivaling non-local U-Net models with fewer parameters. As a first step toward traveling-wave-based communication and visual representation in artificial networks, our findings suggest wave-dynamics may provide efficiency and training stability benefits, while simultaneously offering a new framework for connecting models to biological recordings of neural activity.

cs.CV↗

arXiv2024

A Spatiotemporal Perspective on Dynamical Computation in Neural Information Processing Systems

Spatiotemporal flows of neural activity, such as traveling waves, have been observed throughout the brain since the earliest recordings; yet there is still little consensus on their functional role. Recent experiments and models have linked traveling waves to visual and physical motion, but these observations have been difficult to reconcile with standard accounts of topographically organized selectivity and feedforward receptive fields. Here, we introduce a theoretical framework that formalizes and generalizes the connection between 'motion' and flowing neural dynamics in the language of equivariant neural network theory. We consider 'motion' not only in physical or visual spaces, but also in more abstract representational spaces, and we argue that recurrent traveling-wave-like dynamics are not just useful but necessary for accurate and stable processing of any signal undergoing such motion. Formally, we show that for any non-trivial recurrent neural network to process a sequence undergoing a flow transformation (such as visual motion) in a structured equivariant manner, its hidden state dynamics must actively realize a homomorphic representation of the same flow through recurrent connectivity. In this ''spatiotemporal perspective on dynamical computation'', traveling waves and related flows are best understood as faithful dynamic representations of stimulus flows; and consequently the natural inclination of biological systems towards such dynamics may be viewed as an innate inductive bias towards efficiency and generalization in the spatiotemporally-structured dynamical world they inhabit.

q-bio.NC↗

arXiv2024

Transformers and Cortical Waves: Encoders for Pulling In Context Across Time

The capabilities of transformer networks such as ChatGPT and other Large Language Models (LLMs) have captured the world's attention. The crucial computational mechanism underlying their performance relies on transforming a complete input sequence - for example, all the words in a sentence - into a long "encoding vector" that allows transformers to learn long-range temporal dependencies in naturalistic sequences. Specifically, "self-attention" applied to this encoding vector enhances temporal context in transformers by computing associations between pairs of words in the input sequence. We suggest that waves of neural activity traveling across single cortical areas or multiple regions at the whole-brain scale could implement a similar encoding principle. By encapsulating recent input history into a single spatial pattern at each moment in time, cortical waves may enable temporal context to be extracted from sequences of sensory inputs, the same computational principle used in transformers.

cs.CL↗

arXiv2023

Traveling Waves Encode the Recent Past and Enhance Sequence Learning

Traveling waves of neural activity have been observed throughout the brain at a diversity of regions and scales; however, their precise computational role is still debated. One physically inspired hypothesis suggests that the cortical sheet may act like a wave-propagating system capable of invertibly storing a short-term memory of sequential stimuli through induced waves traveling across the cortical surface, and indeed many experimental results from neuroscience correlate wave activity with memory tasks. To date, however, the computational implications of this idea have remained hypothetical due to the lack of a simple recurrent neural network architecture capable of exhibiting such waves. In this work, we introduce a model to fill this gap, which we denote the Wave-RNN (wRNN), and demonstrate how such an architecture indeed efficiently encodes the recent past through a suite of synthetic memory tasks where wRNNs learn faster and reach significantly lower error than wave-free counterparts. We further explore the implications of this memory storage system on more complex sequence modeling tasks such as sequential image classification and find that wave-based models not only again outperform comparable wave-free RNNs while using significantly fewer parameters, but additionally perform comparably to more complex gated architectures such as LSTMs and GRUs.

cs.NE↗

arXiv2023

Dynamical properties and mechanisms of metastability: a perspective in neuroscience

Metastability, characterized by a variability of regimes in time, is a ubiquitous type of neural dynamics. It has been formulated in many different ways in the neuroscience literature, however, which may cause some confusion. In this Perspective, we discuss metastability from the point of view of dynamical systems theory. We extract from the literature a very simple but general definition through the concept of metastable regimes as long-lived but transient epochs of activity with unique dynamical properties. This definition serves as an umbrella term that encompasses formulations from other works, and readily connects to concepts from dynamical systems theory. This allows us to examine general dynamical properties of metastable regimes, propose in a didactic manner several dynamics-based mechanisms that generate them, and discuss a theoretical tool to characterize them quantitatively. This perspective leads to insights that help to address issues debated in the literature and also suggest pathways for future research.

q-bio.NC↗

arXiv2023

Predicting Stellar Mass Accretion: An Optimized Echo State Network Approach in Time Series Modeling

Modeling the dynamics of the formation and evolution of protostellar disks as well as the history of stellar mass accretion typically involve the numerical solution of complex systems of coupled differential equations. The resulting mass accretion history of protostars is known to be highly episodic due to recurrent instabilities and also exhibits short timescale flickering. By leveraging the strong predictive abilities of neural networks, we extract some of the critical temporal dynamics experienced during the mass accretion including periods of instability. Particularly, we utilize a novel form of the Echo-State Neural Network (ESN), which has been shown to efficiently deal with data having inherent nonlinearity. We introduce the use of Optimized-ESN (Opt-ESN) to make model-independent time series forecasting of mass accretion rate in the evolution of protostellar disks. We apply the network to multiple hydrodynamic simulations with different initial conditions and exhibiting a variety of temporal dynamics to demonstrate the predictability of the Opt-ESN model. The model is trained on simulation data of $\sim 1-2$ Myr, and achieves predictions with a low normalized mean square error ($\sim 10^{-5}$ to $10^{-3}$) for forecasts ranging between 100 and 3800 yr. This result shows the promise of the application of machine learning based models to time-domain astronomy.

astro-ph.SR↗

arXiv2022

On the Paley graph of a quadratic character

Paley graphs form a nice link between the distribution of quadratic residues and graph theory. These graphs possess remarkable properties which make them useful in several branches of mathematics. Classically, for each prime number $p$ we can construct the corresponding Paley graph using quadratic and non-quadratic residues modulo $p$. Therefore, Paley graphs are naturally associated with the Legendre symbol at $p$ which is a quadratic Dirichlet character of conductor $p$. In this article, we introduce the generalized Paley graphs. These are graphs that are associated with a general quadratic Dirichlet character. We will then provide some of their basic properties. In particular, we describe their spectrum explicitly. We then use those generalized Paley graphs to construct some new families of Ramanujan graphs. Finally, using special values of $L$-functions, we provide an effective upper bound for their Cheeger number.

math.CO↗

arXiv2022

An algebraic approach to spike-time neural codes in the hippocampus

Although temporal coding through spike-time patterns has long been of interest in neuroscience, the specific structures that could be useful for spike-time codes remain highly unclear. Here, we introduce a new analytical approach, using techniques from discrete mathematics, to study spike-time codes. We focus on the phenomenon of ``phase precession'' in the rodent hippocampus. During navigation and learning on a physical track, specific cells in a rodent's brain form a highly structured pattern relative to the oscillation of local population activity. Studies of phase precession largely focus on its well established role in synaptic plasticity and memory formation. Comparatively less attention has been paid to the fact that phase precession represents one of the best candidates for a spike-time neural code. The precise nature of this code remains an open question. Here, we derive an analytical expression for an operator mapping points in physical space, through individual spike times, to complex numbers. The properties of this operator highlight a specific relationship between past and future in hippocampal spike patterns. Importantly, this approach generalizes beyond the specific phenomenon studied here, providing a new technique to study the neural codes within spike-time sequences found during sensory coding and motor behavior. We then introduce a novel spike-based decoding algorithm, based on this operator, that successfully decodes a simulated animal's trajectory using only the animal's initial position and a pattern of spike times. This decoder is robust to noise in spike times and works on a timescale almost an order of magnitude shorter than typically used with decoders that work on average firing rate. These results illustrate the utility of a discrete approach, based on the symmetries in spike patterns, to provide insight into the structure and function of neural systems.

q-bio.NC↗

arXiv2022

On the joins of group rings

Given a collection $\{ G_i\}_{i=1}^d$ of finite groups and a ring $R$, we define a subring of the ring $M_n(R)$ ($n = \sum_{i=1}^d|G_i|)$ that encompasses all the individual group rings $R[G_i]$ along the diagonal blocks as $G_i$-circulant matrices. The precise definition of this ring was inspired by a construction in graph theory known as the joined union of graphs. We call this ring the join of group rings and denote it by $\mathcal{J}_{G_1,\dots, G_d}(R)$. In this paper, we present a systematic study of the algebraic structure of $\mathcal{J}_{G_1,\dots, G_d}(R)$. We show that it has a ring structure and characterize its center, group of units, and Jacobson radical. When $R=k$ is an algebraically closed field, we derive a formula for the number of irreducible modules over $\mathcal{J}_{G_1,\dots, G_d}(k)$. We also show how a blockwise extension of the Fourier transform provides both a generalization of the Circulant Diagonalization Theorem to joins of circulant matrices and an explicit isomorphism between the join algebra and its Wedderburn components.

math.RA↗

arXiv2022

Joins of normal matrices, their spectrum, and applications

Motivated by studies of oscillator networks, we study the spectrum of the join of several normal matrices with constant row sums. We apply our results to compute the characteristic polynomial of the join of several regular graphs. We then use this theorem to study several problems in spectral graph theory. In particular, we provide some simple constructions of Ramanujan graphs and give new proofs for some theorems in the classical book of Cvetkovi\'{c}, Rowlinson, and Slobodan.

math.CO↗

arXiv2021

Spectral perturbation by rank one matrices

Let $A$ be a matrix of size $n \times n$ over an algebraically closed field $F$ and $q(t)$ a monic polynomial of degree $n$. In this article, we describe the necessary and sufficient conditions of $q(t)$ so that there exists a rank one matrix $B$ such that the characteristic polynomial of $A+B$ is $q(t)$.

math.SP↗

arXiv2021

Joins of circulant matrices

We study the spectrum of the join of several circulant matrices. We apply our results to compute explicitly the spectrum of certain graphs obtained by joining several circulant graphs.

math.CO↗

arXiv2021

An algebraic approach to the Kuramoto model

We study the Kuramoto model with attractive sine coupling. We introduce a complex-valued matrix formulation whose argument coincides with the original Kuramoto dynamics. We derive an exact solution for the complex-valued model, which permits analytical insight into individual realizations of the Kuramoto model. The existence of a complex-valued form of the Kuramoto model provides a key demonstration that, in some cases, re-formulations of nonlinear dynamics in higher-order number fields may provide tractable analytical approaches.

math.DS↗

arXiv2014

Characterization and Compensation of Network-Level Anomalies in Mixed-Signal Neuromorphic Modeling Platforms

Advancing the size and complexity of neural network models leads to an ever increasing demand for computational resources for their simulation. Neuromorphic devices offer a number of advantages over conventional computing architectures, such as high emulation speed or low power consumption, but this usually comes at the price of reduced configurability and precision. In this article, we investigate the consequences of several such factors that are common to neuromorphic devices, more specifically limited hardware resources, limited parameter configurability and parameter variations. Our final aim is to provide an array of methods for coping with such inevitable distortion mechanisms. As a platform for testing our proposed strategies, we use an executable system specification (ESS) of the BrainScaleS neuromorphic system, which has been designed as a universal emulation back-end for neuroscientific modeling. We address the most essential limitations of this device in detail and study their effects on three prototypical benchmark network models within a well-defined, systematic workflow. For each network model, we start by defining quantifiable functionality measures by which we then assess the effects of typical hardware-specific distortion mechanisms, both in idealized software simulations and on the ESS. For those effects that cause unacceptable deviations from the original network dynamics, we suggest generic compensation mechanisms and demonstrate their effectiveness. Both the suggested workflow and the investigated compensation mechanisms are largely back-end independent and do not require additional hardware configurability beyond the one required to emulate the benchmark networks in the first place. We hereby provide a generic methodological environment for configurable neuromorphic devices that are targeted at emulating large-scale, functional neural networks.

q-bio.NC↗

arXiv2011

A Comprehensive Workflow for General-Purpose Neural Modeling with Highly Configurable Neuromorphic Hardware Systems

In this paper we present a methodological framework that meets novel requirements emerging from upcoming types of accelerated and highly configurable neuromorphic hardware systems. We describe in detail a device with 45 million programmable and dynamic synapses that is currently under development, and we sketch the conceptual challenges that arise from taking this platform into operation. More specifically, we aim at the establishment of this neuromorphic system as a flexible and neuroscientifically valuable modeling tool that can be used by non-hardware-experts. We consider various functional aspects to be crucial for this purpose, and we introduce a consistent workflow with detailed descriptions of all involved modules that implement the suggested steps: The integration of the hardware interface into the simulator-independent model description language PyNN; a fully automated translation between the PyNN domain and appropriate hardware configurations; an executable specification of the future neuromorphic system that can be seamlessly integrated into this biology-to-hardware mapping process as a test bench for all software layers and possible hardware design modifications; an evaluation scheme that deploys models from a dedicated benchmark library, compares the results generated by virtual or prototype hardware devices with reference software simulations and analyzes the differences. The integration of these components into one hardware-software workflow provides an ecosystem for ongoing preparative studies that support the hardware design process and represents the basis for the maturity of the model-to-hardware mapping software. The functionality and flexibility of the latter is proven with a variety of experimental results.

q-bio.NC↗