arXiv subjects

Ulrich Rüde

Publications and source records attributed to Ulrich Rüde.

At least 19 recordsLinked to original sources

arXiv2026

TERRA-NG v1.0: Extreme-Scale, GPU-accelerated Mantle Convection

We present TERRA-NG, a portable, GPU-accelerated, matrix-free mantle-convection code. A single Kokkos C++ implementation runs at scale on NVIDIA, AMD, and Intel GPU supercomputers. TERRA-NG has a deliberately narrow design: built on a radially extruded mesh of spherical wedges, tailored to the spherical shell geometry, which enables domain-specific optimizations like single quadrature-point integral-evaluations, radial coordinate storage compression and radial shared-memory tiling. The corresponding low-order $W_1$-iso-$W_2/W_1$ wedge-based Stokes--energy discretisation is verified against the Zhong et al.(2008) spherical-shell convection benchmark suite. We showcase TERRA-NG through strong- and weak-scaling on the JUWELS Booster (NVIDIA A100), MareNostrum 5 (NVIDIA H100), LUMI-G (AMD MI250X), Hunter (AMD MI300A APU), and SuperMUC-NG Phase 2 (Intel PVC) supercomputers. Coupled mantle convection simulations at $\sim\!11$ km and $\sim\!5.6$ km radial spacing ($\sim 2.8$ B and $\sim 22$ B DoFs) can be run routinely on standard node partitions of all considered systems. Global $\sim\!1$ km-per-gridpoint mantle convection ($\sim 1.4$ T DoFs) is feasible on an extreme-scale allocation, and a sub-km hero-run at $\sim\!0.7$ km grid spacing scaling up to $\sim 11,000$ GPUs of LUMI-G ($\sim 11$ T DoFs) shows the potential of the code on future, larger machines.

cs.CE↗

arXiv2026

Performance Analysis of Low-Order, GPU-accelerated Finite Element Kernels using Kokkos

We study performance portability for low-order, matrix-free finite element kernels, using the example of a vectorial, variable-coefficient PDE operator originating in geophysical models. Written in Kokkos, the kernel is compared on NVIDIA H100, AMD MI250X, AMD MI300A and Intel PVC Max 1550 GPUs. Owing to its low order and to optimizations that reduce the arithmetic, the kernel has a low arithmetic intensity, so that its performance is determined by how the finite element assembly is mapped onto the memory hierarchy. This is a dimension in which the architectures differ even within one vendor family, causing different performance characteristics. We examine how Kokkos' hierarchical parallelism and shared scratch memory, which are used for the shared degrees of freedom of the conforming discretization, behave on each device. Finally, we show how portability gaps can be narrowed with tuning levers such as the size of the thread groups, the balance between occupancy and register use, and the atomic accumulation strategy at the end of the kernel.

cs.PF↗

arXiv2026

Mixed precision multigrid with smoothing based on incomplete Cholesky factorization

Multigrid methods are popular iterative methods for solving large-scale sparse systems of linear equations. We present a mixed precision formulation of the multigrid V-cycle method with general assumptions on the finite precision errors coming from the application of coarsest-level solver and smoothing. Inspired by existing analysis, we derive a bound on the relative finite precision error of the V-cycle which gives insight into how the finite precision errors from the individual components of the method may affect the overall finite precision error. We use the result to study V-cycle methods with smoothing based on incomplete Cholesky factorization. The results imply that in certain settings, the precisions used for applying the incomplete Cholesky smoothing can be significantly lower than the precision used for computing the residual, restriction, prolongation, and correction on a given level. We perform numerical experiments using simulated floating point arithmetic with the MATLAB Advanpix toolbox as well as experiments on GPUs using the Ginkgo library. The experiments illustrate the theoretical findings and show that in the considered settings, the incomplete Cholesky smoothing can be applied in relatively low precisions, resulting in significant improvements in the execution time (up to 28% less), energy savings (up to 26% less) and required memory (up to 16% less) in comparison with the uniform double precision variant.

math.NA↗

arXiv2026

Automated Grammar-based Algebraic Multigrid Design With Evolutionary Algorithms

Although multigrid is asymptotically optimal for solving many important partial differential equations, its efficiency relies heavily on the careful selection of the individual algorithmic components. In contrast to recent approaches that can optimize certain multigrid components using deep learning techniques, we adopt a complementary strategy, employing evolutionary algorithms to construct efficient multigrid cycles from proven algorithmic building blocks. Here, we will present its application to generate efficient algebraic multigrid methods with so-called \emph{flexible cycling}, that is, level-specific smoothing sequences and non-recursive cycling patterns. The search space with such non-standard cycles is intractable to navigate manually, and is generated using genetic programming (GP) guided by context-free grammars. Numerical experiments with the linear algebra library, \emph{hypre}, demonstrate the potential of these non-standard GP cycles to improve multigrid performance both as a solver and a preconditioner.

cs.CE↗

arXiv2026

Conjugate Gradient Methods are Not Efficient: Experimental Study of the Locality Limitation

The convergence of the Conjugate Gradient method is subject to a locality limitation which imposes a lower bound on the number of iterations required before a qualitatively accurate approximation can be obtained. This limitation originates from the restricted transport of information in the graph induced by the sparsity pattern of the system matrix. In each iteration, information from the right-hand side can propagate only across directly connected graph nodes. The diameter of this graph therefore determines a minimum number of iterations that is necessary to achieve an acceptable level of accuracy.

math.NA↗

arXiv2025

Large-scale Multigrid with Adaptive Galerkin Coarsening

We propose a robust, adaptive coarse-grid correction scheme for matrix-free geometric multigrid targeting PDEs with strongly varying coefficients. The method combines uniform geometric coarsening of the underlying grid with heterogeneous coarse-grid operators: Galerkin coarse grid approximation is applied locally in regions with large coefficient gradients, while lightweight, direct coarse grid approximation is used elsewhere. This selective application ensures that local Galerkin operators are computed and stored only where necessary, minimizing memory requirements while maintaining robust convergence. We demonstrate the method on a suite of sinker benchmark problems for the generalized Stokes equation, including grid-aligned and unaligned viscosity jumps, smoothly varying viscosity functions with large gradients, and different viscosity evaluation techniques. We analytically quantify the solver's memory consumption and demonstrate its efficiency by solving Stokes problems with $10^{10}$ degrees of freedom, viscosity jumps of $10^{6}$ magnitude, and more than 100{,}000 parallel processes.

cs.PF↗

arXiv2025

$k\ell$-refinement: An adaptive mesh refinement scheme for hiearchical hybrid grids

This work introduces an adaptive mesh refinement technique for hierarchical hybrid grids with the goal to reach scalability and maintain excellent performance on massively parallel computer systems. On the block structured hierarchical hybrid grids, this is accomplished by using classical, unstructured refinement only on the coarsest level of the hierarchy, while keeping the number of structured refinement levels constant on the whole domain. This leads to a compromise where the excellent performance characteristics of hierarchical hybrid grids can be maintained at the price that the flexibility of generating locally refined meshes is constrained. Furthermore, mesh adaptivity often relies on a posteriori error estimators or error indicators that tend to become computationally expensive. Again with the goal of preserving scalability and performance, a method is proposed that leverages the grid hierarchy and the full multigrid scheme that generates a natural sequence of approximations on the nested hierarchy of grids. This permits to compute a cheap error estimator that is well-suited for large-scale parallel computing. We present the theoretical foundations for both global and local error estimates and present a rigorous analysis of their effectivity. The proposed method, including error estimator and the adaptive coarse grid refinement, is implemented in the finite element framework HyTeG. Extensive numerical experiments are conducted to validate the effectiveness, as well as performance and scalability.

math.NA↗

arXiv2024

Efficiency and scalability of fully-resolved fluid-particle simulations on heterogeneous CPU-GPU architectures

Current supercomputers often have a heterogeneous architecture using both CPUs and GPUs. At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on CPUs. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale flow simulation results in only moderate slowdowns due to the efficiency of the CUDA-aware MPI communication, combined with communication hiding techniques. On 1024 A100 GPUs, a parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers. Additionally, an a priori estimate of the speedup for hybrid implementations is suggested.

cs.CE↗

arXiv2024

Architecture Specific Generation of Large Scale Lattice Boltzmann Methods for Sparse Complex Geometries

We implement and analyse a sparse / indirect-addressing data structure for the Lattice Boltzmann Method to support efficient compute kernels for fluid dynamics problems with a high number of non-fluid nodes in the domain, such as in porous media flows. The data structure is integrated into a code generation pipeline to enable sparse Lattice Boltzmann Methods with a variety of stencils and collision operators and to generate efficient code for kernels for CPU as well as for AMD and NVIDIA accelerator cards. We optimize these sparse kernels with an in-place streaming pattern to save memory accesses and memory consumption and we implement a communication hiding technique to prove scalability. We present single GPU performance results with up to 99% of maximal bandwidth utilization. We integrate the optimized generated kernels in the high performance framework WALBERLA and achieve a scaling efficiency of at least 82% on up to 1024 NVIDIA A100 GPUs and up to 4096 AMD MI250X GPUs on modern HPC systems. Further, we set up three different applications to test the sparse data structure for realistic demonstrator problems. We show performance results for flow through porous media, free flow over a particle bed, and blood flow in a coronary artery. We achieve a maximal performance speed-up of 2 and a significantly reduced memory consumption by up to 75% with the sparse / indirect-addressing data structure compared to the direct-addressing data structure for these applications.

cs.DC↗

arXiv2024

Code Generation and Performance Engineering for Matrix-Free Finite Element Methods on Hybrid Tetrahedral Grids

This paper introduces a code generator designed for node-level optimized, extreme-scalable, matrix-free finite element operators on hybrid tetrahedral grids. It optimizes the local evaluation of bilinear forms through various techniques including tabulation, relocation of loop invariants, and inter-element vectorization - implemented as transformations of an abstract syntax tree. A key contribution is the development, analysis, and generation of efficient loop patterns that leverage the local structure of the underlying tetrahedral grid. These significantly enhance cache locality and arithmetic intensity, mitigating bandwidth-pressure associated with compute-sparse, low-order operators. The paper demonstrates the generator's capabilities through a comprehensive educational cycle of performance analysis, bottleneck identification, and emission of dedicated optimizations. For three differential operators ($-Δ$, $-\nabla \cdot (k(\mathbf{x})\, \nabla\,)$, $α(\mathbf{x})\, \mathbf{curl}\ \mathbf{curl} + β(\mathbf{x}) $), we determine the set of most effective optimizations. Applied by the generator, they result in speed-ups of up to 58$\times$ compared to reference implementations. Detailed node-level performance analysis yields matrix-free operators with a throughput of 1.3 to 2.1 GDoF/s, achieving up to 62% peak performance on a 36-core Intel Ice Lake socket. Finally, the solution of the curl-curl problem with more than a trillion ($ 10^{12}$) degrees of freedom on 21504 processes in less than 50 seconds demonstrates the generated operators' performance and extreme-scalability as part of a full multigrid solver.

cs.CE↗

arXiv2024

A Continuous Benchmarking Infrastructure for High-Performance Computing Applications

For scientific software, especially those used for large-scale simulations, achieving good performance and efficiently using the available hardware resources is essential. It is important to regularly perform benchmarks to ensure the efficient use of hardware and software when systems are changing and the software evolves. However, this can become quickly very tedious when many options for parameters, solvers, and hardware architectures are available. We present a continuous benchmarking strategy that automates benchmarking new code changes on high-performance computing clusters. This makes it possible to track how each code change affects the performance and how it evolves.

cs.PF↗

arXiv2023

waLBerla-wind: a lattice-Boltzmann-based high-performance flow solver for wind energy applications

This article presents the development of a new wind turbine simulation software to study wake flow physics. To this end, the design and development of waLBerla-wind, a new simulator based on the lattice-Boltzmann method that is known for its excellent performance and scaling properties, will be presented. Here it will be used for large eddy simulations (LES) coupled with actuator wind turbine models. Due to its modular software design, waLBerla-wind is flexible and extensible with regard to turbine configurations. Additionally it is performance portable across different hardware architectures, another critical design goal. The new solver is validated by presenting force distributions and velocity profiles and comparing them with experimental data and a vortex solver. Furthermore, waLBerla-wind's performance is \revision{compared to a theoretical peak performance}, and analysed with weak and strong scaling benchmarks on CPU and GPU systems. This analysis demonstrates the suitability for large-scale applications and future cost-effective full wind farm simulations.

cs.CE↗

arXiv2023

Fundamental Data Structures for Matrix-Free Finite Elements on Hybrid Tetrahedral Grids

This paper presents efficient data structures for the implementation of matrix-free finite element methods on block-structured, hybrid tetrahedral grids. It provides a complete categorization of all geometric sub-objects that emerge from the regular refinement of the unstructured, tetrahedral coarse grid and describes efficient iteration patterns and analytical linearization functions for the mapping of coefficients to memory addresses. This foundation enables the implementation of fast, extreme-scalable, matrix-free, iterative solvers, and in particular geometric multigrid methods by design. Their application to the variable-coefficient Stokes system subject to an enriched Galerkin discretization and to the curl-curl problem discretized with Nédélec edge elements showcases the flexibility of the implementation. Eventually, the solution of a curl-curl problem with $1.6 \cdot 10^{11}$ (more than one hundred billion) unknowns on more than $32000$ processes with a matrix-free full multigrid solver demonstrates its extreme-scalability.

cs.CE↗

arXiv2023

Model-Based Performance Analysis of the HyTeG Finite Element Framework

In this work, we present how code generation techniques significantly improve the performance of the computational kernels in the HyTeG software framework. This HPC framework combines the performance and memory advantages of matrix-free multigrid solvers with the flexibility of unstructured meshes. The pystencils code generation toolbox is used to replace the original abstract C++ kernels with highly optimized loop nests. The performance of one of those kernels (the matrix-vector multiplication) is thoroughly analyzed using the Execution-Cache-Memory (ECM) performance model. We validate these predictions by measurements on the SuperMUC-NG supercomputer. The experiments show that the performance mostly matches the predictions. In cases where the prediction does not match, we discuss the discrepancies. Additionally, we conduct a node-level scaling study which shows the expected behavior for a memory-bound compute kernel.

cs.PF↗

arXiv2023

Particle-resolved simulation of antidunes in free-surface flows

The interaction of supercritical turbulent flows with granular sediment beds is challenging to study both experimentally and numerically; this challenging task has hampered the advances in understanding antidunes, the most characteristic bedform of supercritical flows. This article presents the first numerical attempt to simulate upstream-migrating antidunes with geometrically resolved particles and a liquid-gas interface. Our simulations provide data at a resolution higher than laboratory experiments, and they can therefore provide new insights into the mechanisms of antidune migration and contribute to a deeper understanding of the underlying physics. To manage the simulations' computational costs and physical complexity, we employ the cumulant lattice Boltzmann method in conjunction with a discrete element method for particle interactions, as well as a volume of fluid scheme to track the deformable free surface of the fluid. By reproducing two flow configurations of previous experiments (Pascal et al., Earth Surf. Proc. Land., vol. 46(9), 2021, 1750-1765), we demonstrate that our approach is robust and accurately predicts the antidunes' amplitude, wavelength, and celerity. Furthermore, the simulated wall-shear stress, a key parameter governing sediment transport, is in excellent agreement with the experimental measurements. The highly resolved data of fluid and particle motion from our simulation approach open new perspectives for detailed studies of morphodynamics in shallow supercritical flows.

physics.flu-dyn↗

arXiv2023

Analysis and comparison of boundary condition variants in the free-surface lattice Boltzmann method

The accuracy of the free-surface lattice Boltzmann method (FSLBM) depends significantly on the boundary condition employed at the free interface. Ideally, the chosen boundary condition balances the forces exerted by the liquid and gas pressure. Different variants of the same boundary condition are possible, depending on the number and choice of the particle distribution functions (PDFs) to which it is applied. This study analyzes and compares four variants, in which (i) the boundary condition is applied to all PDFs oriented in the opposite direction of the free interface's normal vector, including or (ii) excluding the central PDF. While these variants overwrite existing information, the boundary condition can also be applied (iii) to only missing PDFs without dropping available data or (iv) to only missing PDFs but at least three PDFs as suggested in the literature. It is shown that neither variant generally balances the forces exerted by the liquid and gas pressure at the free surface. The four variants' accuracy was compared in five different numerical experiments covering various applications. These include a standing gravity wave, a rectangular and cylindrical dam break, a rising Taylor bubble, and a droplet impacting a thin pool of liquid. Overall, variant (iii) was substantially more accurate than the other variants in the numerical experiments performed in this study.

physics.flu-dyn↗

arXiv2022

Comparison of free-surface and conservative Allen-Cahn phase-field lattice Boltzmann method

This study compares the free-surface lattice Boltzmann method (FSLBM) with the conservative Allen-Cahn phase-field lattice Boltzmann method (PFLBM) in their ability to model two-phase flows in which the behavior of the system is dominated by the heavy phase. Both models are introduced and their individual properties, strengths and weaknesses are thoroughly discussed. Six numerical benchmark cases were simulated with both models, including (i) a standing gravity and (ii) capillary wave, (iii) an unconfined rising gas bubble in liquid, (iv) a Taylor bubble in a cylindrical tube, and (v) the vertical and (vi) oblique impact of a drop into a pool of liquid. Comparing the simulation results with either analytical models or experimental data from the literature, four major observations were made. Firstly, the PFLBM selected was able to simulate flows purely governed by surface tension with reasonable accuracy. Secondly, the FSLBM, a sharp interface model, generally requires a lower resolution than the PFLBM, a diffuse interface model. However, in the limit case of a standing wave, this was not observed. Thirdly, in simulations of a bubble moving in a liquid, the FSLBM accurately predicted the bubble's shape and rise velocity with low computational resolution. Finally, the PFLBM's accuracy is found to be sensitive to the choice of the model's mobility parameter and interface width.

physics.flu-dyn↗

arXiv2022

Comparison of refilling schemes in the free-surface lattice Boltzmann method

Simulating mobile liquid-gas interfaces with the free-surface lattice Boltzmann method (FSLBM) requires frequent re-initialization of fluid flow information in computational cells that convert from gas to liquid. The corresponding algorithm, here referred to as the refilling scheme, is crucial for the successful application of the FSLBM in terms of accuracy and numerical stability. This study compares five refilling schemes that extract information from the surrounding liquid and interface cells by averaging, extrapolating, or assuming one of the three different equilibrium states. Six numerical experiments were performed, covering a broad spectrum of possible scenarios. These include a standing gravity wave, a rectangular and cylindrical dam break, a Taylor bubble, a drop impact into liquid, and a bubbly plane Poiseuille flow. In some simulations, the averaging, extrapolation, and one equilibrium-based scheme were numerically unstable. Overall, the results have shown that the simplest equilibrium-based scheme should be preferred in terms of numerical stability, computational cost, accuracy, and ease of implementation.

physics.flu-dyn↗