arXiv subjects

Manfred Opper

Publications and source records attributed to Manfred Opper.

At least 19 recordsLinked to original sources

arXiv2026

Introduction to Stochastic Differential Equations for Generative Machine Learning: A Variational Perspective

The use of ordinary and stochastic differential equations has led to substantial progress in generative machine learning with applications to, for example, image, video and biomolecule generation. This paper provides a self-contained and informal introduction to the differential equations, the probabilistic framework for using them in generative modeling and the Fokker--Planck equation that governs the temporal evolution of the marginal distribution of the stochastic variables of the differential equations. The variational lower bound on the log-likelihood (the evidence lower bound, ELBO) is derived and used as a general starting point for a discussion of diffusion models, score matching, and flow matching. All of these approaches may be viewed as specific parameterizations of the most general variational approach. A one-dimensional density modeling problem is used as a simple example to compare different parameterizations.

cs.LG↗

arXiv2026

Variational Inference for L\'evy Process-Driven SDEs via Neural Tilting

Modelling extreme events and heavy-tailed phenomena is central to building reliable predictive systems in domains such as finance, climate science, and safety-critical AI. While L\'evy processes provide a natural mathematical framework for capturing jumps and heavy tails, Bayesian inference for L\'evy-driven stochastic differential equations (SDEs) remains intractable with existing methods: Monte Carlo approaches are rigorous but lack scalability, whereas neural variational inference methods are efficient but rely on Gaussian assumptions that fail to capture discontinuities. We address this tension by introducing a neural exponential tilting framework for variational inference in L\'evy-driven SDEs. Our approach constructs a flexible variational family by exponentially reweighting the L\'evy measure using neural networks. This parametrization preserves the jump structure of the underlying process while remaining computationally tractable. To enable efficient inference, we develop a quadratic neural parametrization that yields closed-form normalization of the tilted measure, a conditional Gaussian representation for stable processes that facilitates simulation, and symmetry-aware Monte Carlo estimators for scalable optimization. Empirically, we demonstrate that the method accurately captures jump dynamics and yields reliable posterior inference in regimes where Gaussian-based variational approaches fail, on both synthetic and real-world datasets.

cs.LG↗

arXiv2025

Fractional Diffusion Bridge Models

We present Fractional Diffusion Bridge Models (FDBM), a novel generative diffusion bridge framework driven by an approximation of the rich and non-Markovian fractional Brownian motion (fBM). Real stochastic processes exhibit a degree of memory effects (correlations in time), long-range dependencies, roughness and anomalous diffusion phenomena that are not captured in standard diffusion or bridge modeling due to the use of Brownian motion (BM). As a remedy, leveraging a recent Markovian approximation of fBM (MA-fBM), we construct FDBM that enable tractable inference while preserving the non-Markovian nature of fBM. We prove the existence of a coupling-preserving generative diffusion bridge and leverage it for future state prediction from paired training data. We then extend our formulation to the Schr\"{o}dinger bridge problem and derive a principled loss function to learn the unpaired data translation. We evaluate FDBM on both tasks: predicting future protein conformations from aligned data, and unpaired image translation. In both settings, FDBM achieves superior performance compared to the Brownian baselines, yielding lower root mean squared deviation (RMSD) of C$_\alpha$ atomic positions in protein structure prediction and lower Fr\'echet Inception Distance (FID) in unpaired image translation.

cs.LG↗

arXiv2025

Digital Twins: McKean-Pontryagin Control for Partially Observed Physical Twins

Optimal control for fully observed diffusion processes is well established and has led to numerous numerical implementations based on, for example, Bellman's principle, model free reinforcement learning, Pontryagin's maximum principle, and model predictive control. In contrast, much fewer algorithms are available for optimal control of partially observed processes. However, this scenario is central to the digital twin paradigm, where a physical twin is partially observed and control laws are derived based on a digital twin. In this paper, we contribute to this challenge by combining data assimilation in the form of the ensemble Kalman filter with the recently proposed McKean-Pontryagin approach to stochastic optimal control. We derive forward evolving mean-field evolution equations for states and co-states which simultaneously allow for an online assimilation of data as well as an online computation of control laws. The proposed methodology is therefore perfectly suited for real time applications of digital twins. We present numerical results for controlled Lorenz-63 and Lorenz-96 systems as well as an inverted pendulum.

math.OC↗

arXiv2025

Multimodal Generative Flows for LHC Jets

Generative modeling of high-energy collisions at the Large Hadron Collider (LHC) offers a data-driven route to simulations, anomaly detection, among other applications. A central challenge lies in the hybrid nature of particle-cloud data: each particle carries continuous kinematic features and discrete quantum numbers such as charge and flavor. We introduce a transformer-based multimodal flow that extends flow-matching with a continuous-time Markov jump bridge to jointly model LHC jets with both modalities. Trained on CMS Open Data, our model can generate high fidelity jets with realistic kinematics, jet substructure and flavor composition.

hep-ph↗

arXiv2025

Likelihood-Based Heterogeneity Inference Reveals Non-Stationary Effects in Biohybrid Cell-Cargo Transport

Variability of motility behavior in populations of microbiological agents is a ubiquitous phenomenon even in the case of genetically identical cells. Accordingly, passive objects introduced into such biological systems and driven by them will also exhibit heterogeneous motion patterns. Here, we study a biohybrid system of passive beads driven by active ameboid cells and use a likelihood approach to estimate the heterogeneity of the bead dynamics from their discretely sampled trajectories. We showcase how this approach can deal with information-scarce situations and provides natural uncertainty bounds for heterogeneity estimates. Using these advantages we particularly uncover that the heterogeneity in the system is time-dependent.

cond-mat.soft↗

arXiv2025

On a mean-field Pontryagin minimum principle for stochastic optimal control

This paper outlines a novel extension of the classical Pontryagin minimum (maximum) principle to stochastic optimal control problems. Contrary to the well-known stochastic Pontryagin minimum principle involving forward-backward stochastic differential equations, the proposed formulation is deterministic and of mean-field type. We denote it by the McKean-Pontryagin minimum principle. The Hamiltonian structure of the proposed McKean-Pontryagin minimum principle is achieved via the introduction of a pair of auxiliary functions. A gauge freedom in the choice of one of these two functions can be used to decouple the forward and reverse time equations; hence simplifying the solution of the underlying boundary value problem. We also consider infinite horizon discounted cost optimal control problems. In this case, the mean-field formulation allows one to convert the computation of the desired optimal control law into solving a pair of forward mean-field ordinary differential equations. The McKean-Pontryagin minimum principle is tested numerically for a controlled inverted pendulum, a controlled Lorenz-63 system, and a controlled Lorenz-96 system. Although the focus is on linear-quadratic control problems, the proposed methodology is extendable to more general problems including mean-field type control formulations.

math.OC↗

arXiv2025

Hierarchical Bayesian estimation for continual learning during model-informed precision dosing

Model informed precision dosing (MIPD) is a Bayesian framework to individualize drug therapy based on prior knowledge and patient-specific monitoring data. Typically, prior knowledge results from controlled clinical trials with a more homogeneous patient population compared to the real-world patient population underlying the data to be analysed. Thus, devising algorithms that can learn the distribution underlying the real-world patient population from patient-specific monitoring data is of key importance. Formulating continual learning in MIPD as a hierarchical Bayesian estimation problem, we here investigate different algorithms for the resulting marginal posterior inference problem in a pharmacokinetic context and for different data sparsity scenarios. As an accurate but computationally expensive reference method, a Metropolis-Hastings algorithm adapted to the hierarchical setting was used. Furthermore, several sequential algorithms were investigated: a nested particle filter, a newly developed simplification termed single inner nested particle filter, as well as an approximative parametric method that allows to use Metropolis-within-Gibbs sampling. The single inner nested particle filter showed the best compromise between accuracy and computational complexity. Applications to more challenging MIPD scenarios from cytotoxic chemotherapy and anticoagulation initiation therapy are ongoing.

stat.CO↗

arXiv2025

Efficient Training of Neural SDEs Using Stochastic Optimal Control

We present a hierarchical, control theory inspired method for variational inference (VI) for neural stochastic differential equations (SDEs). While VI for neural SDEs is a promising avenue for uncertainty-aware reasoning in time-series, it is computationally challenging due to the iterative nature of maximizing the ELBO. In this work, we propose to decompose the control term into linear and residual non-linear components and derive an optimal control term for linear SDEs, using stochastic optimal control. Modeling the non-linear component by a neural network, we show how to efficiently train neural SDEs without sacrificing their expressive power. Since the linear part of the control term is optimal and does not need to be learned, the training is initialized at a lower cost and we observe faster convergence.

cs.LG↗

arXiv2024

A Likelihood Approach for Inference of Population Heterogeneity in Particle Ensembles with Second-Order Langevin Dynamics

The inherent complexity of biological agents often leads to motility behavior that appears to have random components. Robust stochastic inference methods are therefore required to understand and predict the motion patterns from time-discrete trajectory data provided by experiments. In many cases, second-order Langevin models are needed to adequately capture the motility. Additionally, population heterogeneity needs to be taken into account when analyzing data from several individual organisms. In this work, we describe a maximum likelihood approach to infer dynamical, stochastic models and, simultaneously, estimate the heterogeneity in a population of motile active particles from discretely sampled, stochastic trajectories. To this end, we propose a method to approximate the likelihood for non-linear second-order Langevin models. We show that this maximum likelihood ansatz outperforms alternative approaches, especially for short trajectories. Additionally, we demonstrate how a measure of uncertainty for the heterogeneity estimate can be derived. We thereby pave the way for the systematic, data-driven inference of dynamical models for actively driven entities based on trajectory data, deciphering temporal fluctuations and inter-particle variability.

cond-mat.soft↗

arXiv2024

Joint Message Detection, Channel, and User Position Estimation for Unsourced Random Access in Cell-Free Networks

We consider unsourced random access (uRA) in user-centric cell-free (CF) wireless networks, where random access users send codewords from a common codebook during specifically dedicated random access channel (RACH) slots. The system is conceptually similar to the so-called 2-step RACH currently discussed in 3GPP standardization. In order to cope with the distributed and CF nature of the network, we propose to partition the network coverage area into zones (referred to as ''locations'') and assign an uRA codebook to each location, such that users in a certain location make use of the associated codebook. The centralized uRA decoder makes use of the multisource AMP algorithm recently proposed by the authors. This yields at once the list of active uRA codewords, an estimate of the corresponding channel vectors, and an estimate of the active users' position. We show excellent performance of this approach and perfect agreement with the rigorous theoretical ''state evolution'' analysis. We also show that the proposed ''location-based'' partitioned codebook approach significantly outperforms a baseline system with a single non-partitioned uRA codebook.

cs.IT↗

arXiv2024

Bridging discrete and continuous state spaces: Exploring the Ehrenfest process in time-continuous diffusion models

Generative modeling via stochastic processes has led to remarkable empirical results as well as to recent advances in their theoretical understanding. In principle, both space and time of the processes can be discrete or continuous. In this work, we study time-continuous Markov jump processes on discrete state spaces and investigate their correspondence to state-continuous diffusion processes given by SDEs. In particular, we revisit the $\textit{Ehrenfest process}$, which converges to an Ornstein-Uhlenbeck process in the infinite state space limit. Likewise, we can show that the time-reversal of the Ehrenfest process converges to the time-reversed Ornstein-Uhlenbeck process. This observation bridges discrete and continuous state spaces and allows to carry over methods from one to the respective other setting. Additionally, we suggest an algorithm for training the time-reversal of Markov jump processes which relies on conditional expectations and can thus be directly related to denoising score matching. We demonstrate our methods in multiple convincing numerical experiments.

stat.ML↗

arXiv2024

A Convergence Analysis of Approximate Message Passing with Non-Separable Functions and Applications to Multi-Class Classification

Motivated by the recent application of approximate message passing (AMP) to the analysis of convex optimizations in multi-class classifications [Loureiro, et. al., 2021], we present a convergence analysis of AMP dynamics with non-separable multivariate nonlinearities. As an application, we present a complete (and independent) analysis of the motivated convex optimization problem.

cs.LG↗

arXiv2023

Generative Fractional Diffusion Models

We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in generative fractional diffusion models (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose augmented score matching to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of rough paths whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

cs.LG↗

arXiv2023

Variational Inference for SDEs Driven by Fractional Noise

We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM). SDEs offer a versatile tool for modeling real-world continuous-time dynamic systems with inherent noise and randomness. Combining SDEs with the powerful inference capabilities of variational methods, enables the learning of representative function distributions through stochastic gradient descent. However, conventional SDEs typically assume the underlying noise to follow a Brownian motion (BM), which hinders their ability to capture long-term dependencies. In contrast, fractional Brownian motion (fBM) extends BM to encompass non-Markovian dynamics, but existing methods for inferring fBM parameters are either computationally demanding or statistically inefficient. In this paper, building upon the Markov approximation of fBM, we derive the evidence lower bound essential for efficient variational inference of posterior path measures, drawing from the well-established field of stochastic analysis. Additionally, we provide a closed-form expression to determine optimal approximation coefficients. Furthermore, we propose the use of neural networks to learn the drift, diffusion and control terms within our variational posterior, leading to the variational training of neural-SDEs. In this framework, we also optimize the Hurst index, governing the nature of our fractional noise. Beyond validation on synthetic data, we contribute a novel architecture for variational latent video prediction,-an approach that, to the best of our knowledge, enables the first variational neural-SDE application to video perception.

cs.LG↗

arXiv2023

Joint Message Detection and Channel Estimation for Unsourced Random Access in Cell-Free User-Centric Wireless Networks

We consider unsourced random access (uRA) in a cell-free (CF) user-centric wireless network, where a large number of potential users compete for a random access slot, while only a finite subset is active. The random access users transmit codewords of length $L$ symbols from a shared codebook, which are received by $B$ geographically distributed radio units (RUs) equipped with $M$ antennas each. Our goal is to devise and analyze a \emph{centralized} decoder to detect the transmitted messages (without prior knowledge of the active users) and estimate the corresponding channel state information. A specific challenge lies in the fact that, due to the geographically distributed nature of the CF network, there is no fixed correspondence between codewords and large-scale fading coefficients (LSFCs). This makes current activity detection approaches which make use of this fixed LSFC-codeword association not directly applicable. To overcome this problem, we propose a scheme where the access codebook is partitioned in location-based subcodes, such that users in a particular location make use of the corresponding subcode. The joint message detection and channel estimation is obtained via a novel {\em Approximated Message Passing} (AMP) algorithm for a linear superposition of matrix-valued sources corrupted by noise. The statistical asymmetry in the fading profile and message activity leads to \emph{different statistics} for the matrix sources, which distinguishes the AMP formulation from previous cases. In the regime where the codebook size scales linearly with $L$, while $B$ and $M$ are fixed, we present a rigorous high-dimensional (but finite-sample) analysis of the proposed AMP algorithm. Exploiting this, we then present a precise (and rigorous) large-system analysis of the message missed-detection and false-alarm rates, as well as the channel estimation mean-square error.

cs.IT↗

arXiv2022

Analysis of Random Sequential Message Passing Algorithms for Approximate Inference

We analyze the dynamics of a random sequential message passing algorithm for approximate inference with large Gaussian latent variable models in a student-teacher scenario. To model nontrivial dependencies between the latent variables, we assume random covariance matrices drawn from rotation invariant ensembles. Moreover, we consider a model mismatching setting, where the teacher model and the one used by the student may be different. By means of dynamical functional approach, we obtain exact dynamical mean-field equations characterizing the dynamics of the inference algorithm. We also derive a range of model parameters for which the sequential algorithm does not converge. The boundary of this parameter range coincides with the de Almeida Thouless (AT) stability condition of the replica symmetric ansatz for the static probabilistic model.

cs.LG↗

arXiv2021

Deterministic particle flows for constraining stochastic nonlinear systems

Devising optimal interventions for constraining stochastic systems is a challenging endeavour that has to confront the interplay between randomness and nonlinearity. Existing methods for identifying the necessary dynamical adjustments resort either to space discretising solutions of ensuing partial differential equations, or to iterative stochastic path sampling schemes. Yet, both approaches become computationally demanding for increasing system dimension. Here, we propose a generally applicable and practically feasible non-iterative methodology for obtaining optimal dynamical interventions for diffusive nonlinear systems. We estimate the necessary controls from an interacting particle approximation to the logarithmic gradient of two forward probability flows evolved following deterministic particle dynamics. Applied to several biologically inspired models, we show that our method provides the necessary optimal controls in settings with terminal-, transient-, or generalised collective-state constraints and arbitrary system dynamics.

cond-mat.stat-mech↗