MFC 5.0: An exascale many-physics flow solver

Abstract

Many problems of interest in engineering, medicine, and the fundamental sciences rely on high-fidelity flow simulation, making performant computational fluid dynamics solvers a mainstay of the open-source software community. Previous work, MFC 3.0, was published, documented, and made open-source by Bryngelson et al. CPC (2021) features numerous physical features, numerical methods, and scalable infrastructure. MFC 5.0 is a significant update to MFC 3.0, featuring a broad set of well-established and novel physical models and numerical methods, as well as the introduction of GPU and APU (or superchip) acceleration. We exhibit state-of-the-art performance and ideal scaling on the first two exascale supercomputers, OLCF's Frontier and LLNL's El Capitan. Combined with MFC's single-accelerator performance, MFC achieves exascale computation in practice and has achieved the largest-to-date public CFD simulation at 200 trillion grid points, earning it a 2025 ACM Gordon Bell Prize finalist. New physical features include the immersed boundary method, $N$-fluid phase change, Euler-Euler and Euler-Lagrange sub-grid bubble models, fluid-structure interaction, hypo- and hyper-elastic materials, chemically reacting flow, two-material surface tension, magnetohydrodynamics (MHD), and more. Numerical techniques now represent the current state-of-the-art, including general relaxation characteristic boundary conditions, WENO variants, Strang splitting for stiff sub-grid flow features, and low Mach number treatments. Weak scaling to tens of thousands of GPUs on OLCF's Summit and Frontier, and LLNL's El Capitan, achieves efficiencies within 5% of ideal to over 90% of their respective system sizes. Strong scaling results for a 16-fold increase in device count show parallel efficiencies exceeding 90% on OLCF Frontier.