arXiv subjects

James Demmel

Publications and source records attributed to James Demmel.

At least 19 recordsLinked to original sources

arXiv2026

Scaling H2 Factorization to Billion-Unknown Three-Dimensional Problems

Strong recursive skeletonization provides approximate direct solvers for dense matrices from integral equations and kernel methods, but distributed implementations face substantial communication costs and limited parallelism, especially in three dimensions. We present a distributed-memory $\mathcal{H}^2$ factorization solver for symmetric matrices that combines spatial box coloring, delayed Schur complement updates, streamed sketching-based interpolative decomposition, and BLAS-3 and shared-memory kernels. Spatial coloring enables every process to participate in each boundary stage, while delayed updates and streamed sketching reduce dense storage and communication. To our knowledge, this is the first distributed-memory strong recursive skeletonization factorization demonstrated for three-dimensional volume integral equations at billion-unknown scale. Experiments on up to 512 compute nodes demonstrate factorization of a three-dimensional Laplace system with 3.62 billion unknowns, alongside three-dimensional Helmholtz tests and a two-dimensional Matérn system with 268 million unknowns. Combining streamed sketching and fully delayed updates reduces factorization time by $6.9\times$ for Laplace and $8.5\times$ for Helmholtz relative to the unsketched, eager configuration, while reducing maximum per-process resident memory by 57\% and 54\%, respectively.

math.NA↗

arXiv2026

DBDSVDMR3: An MRRR-Based Bidiagonal SVD Solver Built from DSTEGR

The optimal algorithm for the symmetric tridiagonal eigenvalue problem is Multiple Relatively Robust Representations (MRRR or MR$^3$), which is capable of computing $k$ eigenvalue/eigenvector pairs in only $O(kn)$ floating-point operations. While MRRR is nearly 30 years old, it has never successfully been adapted to the bidiagonal SVD. This is not for lack of trying; two previous attempts produced implementations that were considered for LAPACK but eventually abandoned, a result of the fact that neither was reliable enough when run on test problems. The question therefore remained: is there a problem with the theory of MRRR for the bidiagonal SVD or were there simply bugs in the previous implementations that went undetected? This paper resolves this question by presenting DBDSVDMR3 -- a new implementation of the MRRR-based algorithm for the bidiagonal SVD originally proposed by Willems and Lang [ETNA 2012], which, for the first time, performs well on a variety of test problems while achieving significant speedups over both LAPACK's bisection-based solver and, in certain cases, divide-and-conquer for the bidiagonal SVD (thereby indicating that the theory is sound). DBDSVDMR3 will be available in the next release of LAPACK. In this paper, we discuss the design of our implementation, which was obtained with the help of Claude Code, and exhibit its performance on a diverse set of test problems.

math.NA↗

arXiv2026

When Context Changes: Understanding Update Failures in LLMs

As preferences, goals, and facts change, LLM agents must use the current state while earlier versions remain in context. Yet they can answer with an old value of the same variable, a failure that we call stale binding. To study when models use outdated information and why, we introduce Controlled In-Context Memory (CICM), a benchmark for tracking and using updated information in conversations and agent logs. We observe that even frontier reasoning models can fail to recover the current state. We find that in open-source models probes can still recover the updated value when the model answers with an old one, pointing to a failure to select information that remains available. Component tests in Qwen and Pythia identify a mechanism for this selection failure: attention drift, where attention favors old values over the current one when producing an answer. We study a one-layer transformer to mathematically understand how this phenomenon happens: when attention scores are similar, several old values can together receive more attention than the current value. Guided by this explanation, we redirect attention toward the current value without further training. When the current value is requested directly, adjusting this intervention for each input corrects most old-value errors across various model families while preserving nearly all initially correct answers. Reliable context management therefore requires more than remembering updated information: models must use it to guide their answers.

cs.AI↗

arXiv2026

How to grade the accuracy of the BLAS

Motivated by accelerating machine learning (ML), many computer vendors and chip manufacturers are building accelerators for matrix multiplication, which save time and energy by operating in the lower precisions needed for ML. This has in turn motivated many efforts to use these accelerators to provide faster matrix multiplication implementations with the higher precision required by many other linear algebra applications. Motivated by the large design space of algorithms for approximating higher precision, with significant performance/accuracy tradeoffs, we provide a benchmark to "grade the accuracy" of a matrix multiplication implementation (or the BLAS more generally), ranging from an "A" for attaining the classic floating point error bound, to a "C" for attaining a weaker but still useful bound, that is satisfied by Strassen-like algorithms. We also propose "ungameable" tests that vendors or users can run to verify their promised accuracy, and describe how these different grades impact the accuracy of applications like LU, QR, and Cholesky decomposition. Our test code is publicly available at github.com/Reference-LAPACK/grading-the-BLAS for developers and users, and we also plan to publicly release all our test results.

math.NA↗

arXiv2026

Tackling Parallelization Challenges of Randomized Preconditioners With Dependency Tracking

Incomplete Cholesky (IC) preconditioners are the standard black-box, memory-efficient first resort for large graph Laplacian systems, but classical IC can break down on irregular graphs. Approximate Cholesky (AC) addresses this with randomized dropping that is provably breakdown-free. Its stochastic fill-in pattern, however, resists static parallelization, creating a "setup bottleneck." We present ParAC, which parallelizes the sequential rchol algorithm on CPUs and GPUs using dynamic dependency tracking and architecture-aware memory management, without altering its numerics. Total time-to-solution speedups over deterministic IC reach $57.3\times$ on CPU and $52.2\times$ on GPU. Algebraic multigrid (AMG) remains preferable on structured PDE discretizations; on irregular graph Laplacians where AMG struggles, ParAC attains up to $7.2\times$ and $6.0\times$. On 200 adversarial SDD systems, ParAC converges on every instance, whereas the AMG solvers fail on 15 and 19, on CPU and GPU, respectively.

cs.DC↗

arXiv2026

Minimizing the Arithmetic and Communication Complexity of Jacobi's Method for Eigenvalues and Singular Values: Part Two -- Parallel Algorithms

This paper presents several parallel versions of Jacobi's method for the symmetric eigenvalue problem and the SVD. A continuation of [Demmel, Luo, Schneider, & Wang 2025], we develop parallel Jacobi algorithms whose arithmetic cost is optimal and whose bandwidth or latency can match the corresponding lower bounds of parallel matrix multiplication. Our focus is a standard distributed-memory setting with variable processor layouts, including both 2D and 2.5D processor grids. In the 2D case, we demonstrate that a standard implementation of parallel Jacobi achieves a perfect speedup in arithmetic cost -- i.e., complexity $O(n^3/P)$ when done with $P$ processors -- while hitting the 2D matrix-multiplication lower bound for bandwidth and (nearly) the lower bound for latency. By employing a 2.5D processor grid and leveraging 2.5D matrix multiplication, equivalently by increasing the memory per processor, we demonstrate that parallel Jacobi can achieve even lower bandwidth/latency, though we also prove that these costs cannot simultaneously match the best-known bounds for parallel matrix multiplication in any Jacobi algorithm. Finally, we extend our results to one-sided Jacobi SVD.

math.NA↗

arXiv2026

Structured Divide-and-Conquer for the Definite Generalized Eigenvalue Problem

This paper presents a fast, randomized divide-and-conquer algorithm for the definite generalized eigenvalue problem, which corresponds to pencils $(A,B)$ in which $A$ and $B$ are Hermitian and the Crawford number $γ(A,B) = \min_{\|x\|_2 = 1} |x^H(A+iB)x|$ is positive. Adapted from the fastest known method for diagonalizing arbitrary matrix pencils [Foundations of Computational Mathematics 2024], the algorithm is both inverse-free and highly parallel. As in the general case, randomization takes the form of perturbations applied to the input matrices, which regularize the problem for compatibility with fast, divide-and-conquer eigensolvers -- i.e., the now well-established phenomenon of pseudospectral shattering. We demonstrate that this high-level approach to diagonalization can be executed in a structure-aware fashion by (1) extending pseudospectral shattering to definite pencils under structured perturbations (either random diagonal or sampled from the Gaussian unitary ensemble) and (2) formulating the divide-and-conquer procedure in a way that maintains definiteness. The result is a specialized solver whose complexity, when applied to definite pencils, is provably lower than that of general divide-and-conquer.

math.NA↗

arXiv2026

Minimizing the Arithmetic and Communication Complexity of Jacobi's Method for Eigenvalues and Singular Values: Part One -- Serial Algorithms

We analyze several versions of Jacobi's method for the symmetric eigenvalue problem. Our goal is to reduce the asymptotic cost of the algorithm as much as possible, as measured by the number of arithmetic operations performed and associated (serial or parallel) communication, i.e., the amount of data moved between slow and fast memory or between processors in a network. The first half of this effort, which considers the serial setting, is presented here; this paper contains rigorous complexity bounds for a variety of serial Jacobi algorithms, built on both classic $O(n^3)$ matrix multiplication and fast, Strassen-like $O(n^{ω_0})$ alternatives. In the classical case, we show that a blocked implementation of Jacobi's method attains the communication lower bound for $O(n^3)$ matrix multiplication (and is therefore expected to be communication optimal among $O(n^3)$ eigensolvers). In the fast setting, we demonstrate that a recursive version of blocked Jacobi can go further, reaching essentially optimal complexity in both measures. We also derive analogous complexity bounds for (one-sided) Jacobi SVD algorithms. A forthcoming sequel to this paper will extend our complexity analysis to the parallel case.

math.NA↗

arXiv2026

A Trust-Region Interior-Point Stochastic Sequential Quadratic Programming Method

In this paper, we propose a trust-region interior-point stochastic sequential quadratic programming (TR-IP-SSQP) method for solving optimization problems with a stochastic objective and deterministic nonlinear equality and inequality constraints. In this setting, exact evaluations of the objective function and its gradient are unavailable, but their stochastic estimates can be constructed. In particular, at each iteration our method builds stochastic oracles, which estimate the objective value and gradient to satisfy proper adaptive accuracy conditions with a fixed probability. To handle inequality constraints, we adopt an interior-point method (IPM), in which the barrier parameter follows a prescribed decaying sequence. Under standard assumptions, we establish global almost-sure convergence of the proposed method to first-order stationary points. We implement the method on a subset of problems from the CUTEst test set, as well as on logistic regression problems, to demonstrate its practical performance.

math.OC↗

arXiv2026

Parallelizing the Approximate Minimum Degree Ordering Algorithm: Strategies and Evaluation

The approximate minimum degree algorithm is widely used before numerical factorization to reduce fill-in for sparse matrices. While considerable attention has been given to the numerical factorization process, less focus has been placed on parallelizing the approximate minimum degree algorithm itself. In this paper, we explore different parallelization strategies, and introduce a novel parallel framework that leverages multiple elimination on distance-2 independent sets. Our evaluation shows that parallelism within individual elimination steps is limited due to low computational workload and significant memory contention. In contrast, our proposed framework overcomes these challenges by parallelizing the work across elimination steps. To the best of our knowledge, our implementation is the first scalable shared memory implementation of the approximate minimum degree algorithm. Experimental results show that we achieve up to a 7.29x speedup using 64 threads over the state-of-the-art sequential implementation in SuiteSparse.

cs.DC↗

arXiv2026

Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise

We develop a worst-case complexity theory for stochastically preconditioned stochastic gradient descent (SPSGD) and its accelerated variants under heavy-tailed noise, a setting that encompasses widely used adaptive methods such as Adam, RMSProp, and Shampoo. We assume the stochastic gradient noise has a finite $p$-th moment for some $p \in (1,2]$, and measure convergence after $T$ iterations. While clipping and normalization are parallel tools for stabilizing training of SGD under heavy-tailed noise, there is a fundamental separation in their worst-case properties in stochastically preconditioned settings. We demonstrate that normalization guarantees convergence to a first-order stationary point at rate $\mathcal{O}(T^{-\frac{p-1}{3p-2}})$ when problem parameters are known, and $\mathcal{O}(T^{-\frac{p-1}{2p}})$ when problem parameters are unknown, matching the optimal rates for normalized SGD, respectively. In contrast, we prove that clipping may fail to converge in the worst case due to the statistical dependence between the stochastic preconditioner and the gradient estimates. To enable the analysis, we develop a novel vector-valued Burkholder-type inequality that may be of independent interest. These results provide a theoretical explanation for the empirical preference for normalization over clipping in large-scale model training.

cs.LG↗

arXiv2026

3DGS$^2$-TR: Scalable Second-Order Trust-Region Method for 3D Gaussian Splatting

We propose 3DGS$^2$-TR,a second-order optimizer for accelerating the scene training problem in 3D Gaussian Splatting (3DGS). Unlike existing second-order approaches that rely on explicit or dense curvature representations, such as 3DGS-LM (Höllein et al., 2025) or 3DGS2 (Lan et al., 2025), our method approximates curvature using only the diagonal of the Hessian matrix, efficiently via Hutchinson's method. Our approach is fully matrix-free and has the same complexity as ADAM (Kingma, 2024), $O(n)$ in both computation and memory costs. To ensure stable optimization in the presence of strong nonlinearity in the 3DGS rasterization process, we introduce a parameter-wise trust-region technique based on the squared Hellinger distance, regularizing updates to Gaussian parameters. Under identical parameter initialization and without densification, 3DGS$^2$-TR is able to achieve better reconstruction quality on standard datasets, using 50% fewer training iterations compared to ADAM, while incurring less than 1GB of peak GPU memory overhead (17% more than ADAM and 85% less than 3DGS-LM), enabling scalability to very large scenes and potentially to distributed training settings.

cs.CV↗

arXiv2026

Fast and Inverse-Free Algorithms for Deflating Subspaces

This paper explores a key question in numerical linear algebra: how can we compute projectors onto the deflating subspaces of a regular matrix pencil $(A,B)$, in particular without using matrix inversion or defaulting to an expensive Schur decomposition? We focus specifically on spectral projectors, whose associated deflating subspaces correspond to sets of eigenvalues/eigenvectors. In this work, we present a high-level approach to computing these projectors, which combines rational function approximation with an inverse-free arithmetic of Benner and Byers [Numerische Mathematik 2006]. The result is a numerical framework that captures existing inverse-free methods, generates an array of new options, and provides straightforward tools for pursuing efficiency on structured problems (e.g., definite pencils). To exhibit the efficacy of this framework, we consider a handful of methods in detail, including Implicit Repeated Squaring and iterations based on the matrix sign function. In an appendix, we demonstrate that recent, randomized divide-and-conquer eigensolvers -- which are built on fast methods for individual projectors -- can be adapted to produce the generalized Schur form of any matrix pencil in nearly matrix multiplication time.

math.NA↗

arXiv2025

StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training

Training Transformer models on long sequences in a distributed setting poses significant challenges in terms of efficiency and scalability. Current methods are either constrained by the number of attention heads or excessive communication overheads. To address this problem, we propose StarTrail, a multi-dimensional concentric distributed training system for long sequences, fostering an efficient communication paradigm and providing additional tuning flexibility for communication arrangements. Specifically, StarTrail introduces an extra parallel dimension and divides the peer-to-peer communication into sub-rings to substantially reduce communication volume and avoid bandwidth bottlenecks. Through comprehensive experiments across diverse hardware environments and on both Natural Language Processing (NLP) and Computer Vision (CV) tasks, we demonstrate that our approach significantly surpasses state-of-the-art methods that support Long sequence lengths, achieving performance improvements of up to 77.12% on GPT-style models and up to 114.33% on DiT (Diffusion Transformer) models without affecting the computations results.

cs.DC↗

arXiv2025

Anatomy of High-Performance Column-Pivoted QR Decomposition

We introduce an algorithmic framework for performing QR factorization with column pivoting (QRCP) on general matrices. The framework enables the design of practical QRCP algorithms through user-controlled choices for the core subroutines. We provide a comprehensive overview of how to navigate these choices on modern hardware platforms, offering detailed descriptions of alternative methods for both CPUs and GPUs. The practical QRCP algorithms developed within this framework are implemented as part of the open-source RandLAPACK library. Our empirical evaluation demonstrates that, on a dual AMD EPYC 9734 system, the proposed method achieves performance improvements of up to two orders of magnitude over LAPACK's standard QRCP routine and greatly surpasses the performance of the current state-of-the-art randomized QRCP algorithm. Additionally, on an NVIDIA H100 GPU, our method attains approximately 65 percent of the performance of cuSOLVER's unpivoted QR factorization.

cs.MS↗

arXiv2025

An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks

Rotation equivariant graph neural networks, i.e. networks designed to guarantee certain geometric relations between their inputs and outputs, yield state of the art performance on spatial deep learning tasks. They exhibit high data efficiency during training and significantly reduced inference time for interatomic potential calculations compared to classical approaches. Key to these models is the Clebsch-Gordon (CG) tensor product, a kernel that contracts two dense feature vectors with a highly-structured sparse tensor to produce a dense output vector. The operation, which may be repeated millions of times for typical equivariant models, is a costly and inefficient bottleneck. We introduce a GPU sparse kernel generator for the CG tensor product that provides significant speedups over the best existing open and closed-source implementations. Our implementation achieves high performance by carefully managing the limited GPU shared memory through static analysis at model compile-time, minimizing reads and writes to global memory. We break the tensor product into a series of smaller kernels with operands that fit entirely into registers, enabling us to emit long arithmetic instruction streams that maximize instruction-level parallelism. By fusing the CG tensor product with a subsequent graph convolution, we reduce both intermediate storage and global memory traffic over naive approaches that duplicate input data. We also provide optimized kernels for the gradient of the CG tensor product and a novel identity for the higher partial derivatives required to predict interatomic forces. Our kernels offer up to 1.3x speedup over NVIDIA's closed-source cuEquivariance package, as well as 10x speedup over the widely-used e3nn package. In FP64 precision, we offer up to 6.2x inference-time speedup for the MACE chemistry foundation model over the original unoptimized version.

cs.LG↗

arXiv2025

CholeskyQR with Randomization and Pivoting for Tall Matrices (CQRRPT)

This paper develops and analyzes a new algorithm for QR decomposition with column pivoting (QRCP) of rectangular matrices with many more rows than columns. The algorithm carefully combines methods from randomized numerical linear algebra to accelerate pivot decisions for the input matrix and the process of decomposing the pivoted matrix into the QR form. The source of the latter improvement is CholeskyQR with randomized preconditioning. Comprehensive analysis is provided in both exact and finite-precision arithmetic to characterize the algorithm's rank-revealing properties and its numerical stability granted probabilistic assumptions of the sketching operator. An implementation of the proposed algorithm is described and made available inside the open-source RandLAPACK library, which itself relies on RandBLAS. Experiments with this implementation on an Intel Xeon Gold 6248R CPU demonstrate order-of-magnitude speedups over LAPACK's standard function for QRCP, and comparable performance to a specialized algorithm for unpivoted QR of tall matrices, which lacks the strong rank-revealing properties of the proposed method.

math.NA↗

arXiv2024

Large Scale Multi-GPU Based Parallel Traffic Simulation for Accelerated Traffic Assignment and Propagation

Traffic propagation simulation is crucial for urban planning, enabling congestion analysis, travel time estimation, and route optimization. Traditional micro-simulation frameworks are limited to main roads due to the complexity of urban mobility and large-scale data. We introduce the Large Scale Multi-GPU Parallel Computing based Regional Scale Traffic Simulation Framework (LPSim), a scalable tool that leverages GPU parallel computing to simulate extensive traffic networks with high fidelity and reduced computation time. LPSim performs millions of vehicle dynamics simulations simultaneously, outperforming CPU-based methods. It can complete simulations of 2.82 million trips in 6.28 minutes using a single GPU, and 9.01 million trips in 21.16 minutes on dual GPUs. LPSim is also tested on dual NVIDIA A100 GPUs, achieving simulations about 113 times faster than traditional CPU methods. This demonstrates its scalability and efficiency for large-scale applications, making LPSim a valuable resource for researchers and planners. Code: https://github.com/Xuan-1998/LPSim

cs.DC↗