arXiv subjects

Jonas Kusch

Publications and source records attributed to Jonas Kusch.

At least 19 recordsLinked to original sources

arXiv2026

High-order robust basis-update & Galerkin integrators for dynamical low-rank approximation

Dynamical low-rank approximation has become a widely used numerical method in diverse disciplines. Its main idea is to represent the matrix or tensor-valued solution to a time-dependent differential equation as a low-rank factorization. The evolution of the factorization leads to highly stiff dynamics and requires the derivation of novel time integration methods that are not prone to this stiffness. A promising family of integrators are basis-update \& Galerkin (BUG) integrators as they enable implicit time integration and structure--preservation. However, current BUG integrators are limited to second--order accuracy while general-order BUG integrators are designed as projections of explicit time integration methods, thus severely limiting their use. In this work, we propose general-order BUG integrators that do not rely on an explicit time integration scheme while requiring a smaller number of basis functions to achieve high-order accuracy. We prove a general order error bound for the proposed augmented and parallel BUG integrators and demonstrate their behaviour for a series of stiff and non-stiff numerical benchmarks in which they significantly outperform previous BUG versions.

math.NA↗

arXiv2026

A Dynamical Low-rank Multilevel Monte Carlo Estimator for High-Dimensional Kinetic Equations

Kinetic equations are used to model a wide range of phenomena important for real-world applications. Their applications span astrophysics, nuclear physics, engineering, and social sciences. Due to their high-dimensional phase space, modelling and quantifying uncertainties, relevant for applications, poses a significant challenge even for modern computing infrastructure. In recent years, dynamical low-rank approximation (DLRA) has gained popularity for making fine grid simulations of high-dimensional problems feasible by evolving the solution of a time-dependent PDE as a low-rank factorization. This reduces the computational and memory requirements significantly. In this work, we propose a low-rank multilevel Monte Carlo estimator for kinetic equations based on a probabilistic rank-adaptive DLRA time integrator. The level hierarchy of the low-rank multilevel estimator is constructed through spatial refinement and by ensuring that the low-rank error remains below the spatial discretization error. We demonstrate the efficacy of the estimator through several numerical experiments from radiation transport, radiation therapy, and shallow water flow.

math.NA↗

arXiv2026

Tucker Attention: A generalization of approximate attention mechanisms

The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.

cs.LG↗

arXiv2025

A high-order deterministic dynamical low-rank method for proton transport in heterogeneous media

Dose calculations in proton therapy require the fast and accurate solution of a high-dimensional transport equation for a large number of (pencil) beams with different energies and directions. Deterministically solving this transport problem at a sufficient resolution can however be prohibitively expensive, especially due to highly forward peaked scattering of the protons. We propose using a model order reduction approach, the dynamical low-rank approximation (DLRA), which evolves the solution on the manifold of low-rank matrices in (pseudo-)time. For this, we compare a collided-uncollided split of the linear Boltzmann equation and its Fokker-Planck approximation. We treat the uncollided part using a ray-tracer and combine high-order phase space discretizations and a mixture model for materials with DLRA for the collided equation. Our method reproduces the results of a full-rank reference code at significantly lower rank, and thus computational cost and memory, and further makes computations feasible at much higher resolutions. At higher resolutions, we also achieve good accuracy with respect to TOPAS MC in homogeneous as well as heterogeneous materials. Finally, we demonstrate that several beam sources with different angles can be computed with little cost increase compared to individual beams.

math.NA↗

arXiv2025

An adaptive dynamical low-rank optimizer for solving kinetic parameter identification inverse problems

The numerical solution of parameter identification inverse problems for kinetic equations can exhibit high computational and memory costs. In this paper, we propose a dynamical low-rank scheme for the reconstruction of the scattering parameter in the radiative transfer equation from a number of macroscopic time-independent measurements. We first work through the PDE constrained optimization procedure in a continuous setting and derive the adjoint equations using a Lagrangian reformulation. For the scattering coefficient, a periodic B-spline approximation is introduced and a gradient descent step for updating its coefficients is formulated. After the discretization, a dynamical low-rank approximation (DLRA) is applied. We make use of the rank-adaptive basis update & Galerkin integrator and a line search approach for the adaptive refinement of the gradient descent step size and the DLRA tolerance. We show that the proposed scheme significantly reduces both memory and computational cost. Numerical results computed with different initial conditions validate the accuracy and efficiency of the proposed DLRA scheme compared to solutions computed with a full solver.

math.NA↗

arXiv2025

A geometric framework for momentum-based optimizers for low-rank training

Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used to train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape. To address this, we introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure. Our approach leverages and combines tools from dynamical low-rank approximation and momentum-based optimization to design optimizers that respect the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating faster convergence, and stronger validation metrics at given parameter budgets.

cs.LG↗

arXiv2025

A Parallel, Energy-Stable Low-Rank Integrator for Nonlinear Multi-Scale Thermal Radiative Transfer

Thermal radiative transfer models physical phenomena ranging from supernovas in astrophysics to radiation from a hohlraum striking a fusion target in plasma physics. Transport and absorption of particles in radiative transfer at different rates lead to a complex interaction between the material and particles that involves highly varying time scales. Resolving these effects can require prohibitively small step sizes, which, combined with nonlinear effects and the particle density's high-dimensional phase space, render conventional numerical methods computationally expensive. This work presents an asymptotic--preserving, mass conservative, rank-adaptive, and parallel integrator for a macro--micro decomposition-based dynamical low-rank approximation of the thermal radiative transfer equations. The proposed integrator efficiently incorporates reflection-transmission type boundary conditions in the low-rank factors. It captures the nonlinear effects of thermal radiation and is energy stable with the step size restriction capturing both hyperbolic and parabolic CFL conditions. The efficacy of the proposed integrator is demonstrated with numerical experiments.

math.NA↗

arXiv2025

HADL Framework for Noise Resilient Long-Term Time Series Forecasting

Long-term time series forecasting is critical in domains such as finance, economics, and energy, where accurate and reliable predictions over extended horizons drive strategic decision-making. Despite the progress in machine learning-based models, the impact of temporal noise in extended lookback windows remains underexplored, often degrading model performance and computational efficiency. In this paper, we propose a novel framework that addresses these challenges by integrating the Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT) to perform noise reduction and extract robust long-term features. These transformations enable the separation of meaningful temporal patterns from noise in both the time and frequency domains. To complement this, we introduce a lightweight low-rank linear prediction layer that not only reduces the influence of residual noise but also improves memory efficiency. Our approach demonstrates competitive robustness to noisy input, significantly reduces computational complexity, and achieves competitive or state-of-the-art forecasting performance across diverse benchmark datasets. Extensive experiments reveal that the proposed framework is particularly effective in scenarios with high noise levels or irregular patterns, making it well suited for real-world forecasting tasks. The code is available in https://github.com/forgee-master/HADL.

cs.LG↗

arXiv2025

An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training

Layer factorization has emerged as a widely used technique for training memory-efficient neural networks. However, layer factorization methods face several challenges, particularly a lack of robustness during the training process. To overcome this limitation, dynamical low-rank training methods have been developed, utilizing robust time integration techniques for low-rank matrix differential equations. Although these approaches facilitate efficient training, they still depend on computationally intensive QR and singular value decompositions of matrices with small rank. In this work, we introduce a novel low-rank training method that reduces the number of required QR decompositions. Our approach integrates an augmentation step into a projector-splitting scheme, ensuring convergence to a locally optimal solution. We provide a rigorous theoretical analysis of the proposed method and demonstrate its effectiveness across multiple benchmarks.

math.NA↗

arXiv2025

An energy stable and conservative multiplicative dynamical low-rank discretization for the Su-Olson problem

Computing numerical solutions of the thermal radiative transfer equations on a finely resolved grid can be costly due to high computational and memory requirements. A numerical reduced order method that has recently been applied to a wide variety of kinetic partial differential equations is the concept of dynamical low-rank approximation (DLRA). In this paper, we consider the thermal radiative transfer equations with Su-Olson closure, leading to a linearized kinetic model. For the conducted theoretical and practical considerations we use a multiplicative splitting of the distribution function that poses additional challenges in finding an energy stable discretization and deriving a hyperbolic Courant-Friedrichs-Lewy (CFL) condition. We propose such an energy stable DLRA scheme that makes use of the augmented basis update & Galerkin integrator. This integrator allows for additional basis augmentations, enabling us to give a mathematically rigorous proof of energy stability and local mass conservation. Numerical examples confirm the derived properties and show the computational advantages of the DLRA scheme compared to a numerical solution of the full system of equations.

math.NA↗

arXiv2025

Low-rank variance reduction for uncertain radiative transfer with control variates

The radiative transfer equation models various physical processes ranging from plasma simulations to radiation therapy. In practice, these phenomena are often subject to uncertainties. Modeling and propagating these uncertainties requires accurate and efficient solvers for the radiative transfer equations. Due to the equation's high-dimensional phase space, fine-grid solutions of the radiative transfer equation are computationally expensive and memory-intensive. In recent years, dynamical low-rank approximation has become a popular method for solving kinetic equations due to the development of computationally inexpensive, memory-efficient and robust algorithms like the augmented basis update \& Galerkin integrator. In this work, we propose a low-rank Monte Carlo estimator and combine it with a control variate strategy based on multi-fidelity low-rank approximations for variance reduction. We investigate the error analytically and numerically and find that a joint approach to balance rank and grid size is necessary. Numerical experiments further show that the efficiency of estimators can be improved using dynamical low-rank approximation, especially in the context of control variates.

math.NA↗

arXiv2024

A Deterministic Dynamical Low-rank Approach for Charged Particle Transport

Deterministically solving charged particle transport problems at a sufficient spatial and angular resolution is often prohibitively expensive, especially due to their highly forward peaked scattering. We propose a model order reduction approach which evolves the solution on a low-rank manifold in time, making computations feasible at much higher resolutions and reducing the overall run-time and memory footprint. For this, we use a hybrid dynamical low-rank approach based on a collided-uncollided split, i.e., the transport equation is split through a collision source method. Uncollided particles are described using a ray tracer, facilitating the inclusion of boundary conditions and straggling, whereas collided particles are represented using a moment method combined with the dynamical low-rank approximation. Here the energy is treated as a pseudo-time and a rank adaptive integrator is chosen to dynamically adapt the rank in energy. We can reproduce the results of a full-rank reference code at a much lower rank and thus computational cost and memory usage. The solution further achieves comparable accuracy with respect to TOPAS MC as previous deterministic approaches.

math.NA↗

arXiv2024

A review of low-rank methods for time-dependent kinetic simulations

Time-dependent kinetic models are ubiquitous in computational science and engineering. The underlying integro-differential equations in these models are high-dimensional, comprised of a six--dimensional phase space, making simulations of such phenomena extremely expensive. In this article we demonstrate that in many situations, the solution to kinetics problems lives on a low dimensional manifold that can be described by a low-rank matrix or tensor approximation. We then review the recent development of so-called low-rank methods that evolve the solution on this manifold. The two classes of methods we review are the dynamical low-rank (DLR) method, which derives differential equations for the low-rank factors, and a Step-and-Truncate (SAT) approach, which projects the solution onto the low-rank representation after each time step. Thorough discussions of time integrators, tensor decompositions, and method properties such as structure preservation and computational efficiency are included. We further show examples of low-rank methods as applied to particle transport and plasma dynamics.

math.NA↗

arXiv2024

A parallel Basis Update and Galerkin Integrator for Tree Tensor Networks

Computing the numerical solution to high-dimensional tensor differential equations can lead to prohibitive computational costs and memory requirements. To reduce the memory and computational footprint, dynamical low-rank approximation (DLRA) has proven to be a promising approach. DLRA represents the solution as a low-rank tensor factorization and evolves the resulting low-rank factors in time. A central challenge in DLRA is to find time integration schemes that are robust to the arising small singular values. A robust parallel basis update & Galerkin integrator, which simultaneously evolves all low-rank factors, has recently been derived for matrix differential equations. This work extends the parallel low-rank matrix integrator to Tucker tensors and general tree tensor networks, yielding an algorithm in which all bases and connecting tensors are evolved in parallel over a time step. We formulate the algorithm, provide a robust error bound, and demonstrate the efficiency of the new integrators for problems in quantum many-body physics, uncertainty quantification, and radiative transfer.

math.NA↗

arXiv2024

A stable multiplicative dynamical low-rank discretization for the linear Boltzmann-BGK equation

The numerical method of dynamical low-rank approximation (DLRA) has recently been applied to various kinetic equations showing a significant reduction of the computational effort. In this paper, we apply this concept to the linear Boltzmann-Bhatnagar-Gross-Krook (Boltzmann-BGK) equation which due its high dimensionality is challenging to solve. Inspired by the special structure of the non-linear Boltzmann-BGK problem, we consider a multiplicative splitting of the distribution function. We propose a rank-adaptive DLRA scheme making use of the basis update & Galerkin integrator and combine it with an additional basis augmentation to ensure numerical stability, for which an analytical proof is given and a classical hyperbolic Courant-Friedrichs-Lewy (CFL) condition is derived. This allows for a further acceleration of computational times and a better understanding of the underlying problem in finding a suitable discretization of the system. Numerical results of a series of different test examples confirm the accuracy and efficiency of the proposed method compared to the numerical solution of the full system.

math.NA↗

arXiv2024

GeoLoRA: Geometric integration for parameter efficient fine-tuning

Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.

cs.LG↗

arXiv2024

Second-order robust parallel integrators for dynamical low-rank approximation

Due to its reduced memory and computational demands, dynamical low-rank approximation (DLRA) has sparked significant interest in multiple research communities. A central challenge in DLRA is the development of time integrators that are robust to the curvature of the manifold of low-rank matrices. Recently, a parallel robust time integrator that permits dynamic rank adaptation and enables a fully parallel update of all low-rank factors was introduced. Despite its favorable computational efficiency, the construction as a first-order approximation to the augmented basis-update & Galerkin integrator restricts the parallel integrator's accuracy to order one. In this work, an extension to higher order is proposed by a careful basis augmentation before solving the matrix differential equations of the factorized solution. A robust error bound with an improved dependence on normal components of the vector field together with a norm preservation property up to small terms is derived. These analytic results are complemented and demonstrated through a series of numerical experiments.

math.NA↗

arXiv2024

Asymptotic-preserving and energy stable dynamical low-rank approximation for thermal radiative transfer equations

The thermal radiative transfer equations model temperature evolution through a background medium as a result of radiation. When a large number of particles are absorbed in a short time scale, the dynamics tend to a non-linear diffusion-type equation called the Rosseland approximation. The main challenges for constructing numerical schemes that exhibit the correct limiting behavior are posed by the solution's high-dimensional phase space and multi-scale effects. In this work, we propose an asymptotic-preserving and rank-adaptive dynamical low-rank approximation scheme based on the macro-micro decomposition of the particle density and a modified augmented basis-update \& Galerkin integrator. We show that this scheme, for linear particle emission by the material, dissipates energy over time under a step size restriction that captures the hyperbolic and parabolic CFL conditions. We demonstrate the efficacy of the proposed method in a series of numerical experiments.

math.NA↗