arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.22769

One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble

Lenore Mullin·Gaetan Hains
View original source ↗Read source document ↗Export metadata ↓

Abstract

The companion MPI-extension paper validated the first communication-bearing machine-shape ensemble: four NVIDIA A100 ranks and four CPU ranks on one NCSA Delta node under one Open MPI job, reassembled by MPI_Gatherv. This paper extends it to a multi-vendor communicator. A Slurm heterogeneous job spanning Delta's A100 and AMD MI100 partitions places A100, MI100 and CPU ranks (nine in all) under one communicator for the first time. We give the theory (a w_i proportional to 1/t_i weight split for three concurrent realizations; a het-job specification as an admissible directive) and report real results, including what the run exposed: the 1/t split assumes time linear in rows, and this fails for the MI100 rank (6 rows took 597 ms against a linear prediction of 1.6 ms). A launch-geometry model, with predictions recorded before measuring, succeeds for the A100 (time flat from 128 to 3456 rows, stepping at exactly 3520 and 6976) and places the MI100 floor in the key loop; changing its launch settings cut about 600 ms to 14.6 ms and the energy per call at 6 rows by 32x, though the MI100 remains about 95x slower than an A100 at 8192 rows. Controlled reruns show that the companion paper's 8.1x A100 result and 22 ms Gatherv were artifacts of four ranks sharing one GPU under Open MPI's default transport. A key/value-split partition whose message sizes are derived from shapes alone is then run: the ring Allreduce of the 8.5 MB message took 11.3, 16.0 and 15.3 ms against a predicted 11-12 ms, the small Allreduce missed its prediction in every run, and compute times reproduce to about 1% while communication times vary up to 4x between runs. Each idea is introduced with a figure and closed with worked examples.

Explore related subjects

cs.DCcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Lenore Mullin, Gaetan Hains. 2026-10-16. One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble. https://arxiv.org/abs/2610.22769

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Load Balancing with Evolving Workloads in Large-Scale Language Model Serving

Serving large language models requires assigning requests whose workloads grow during processing and whose completion times are unknown. With a synchronization barrier, lightly loaded workers wait for the slowest worker at every decode iteration. We model this loss through aggregate workload imbalance and formulate a joint admission and assignment problem with sticky requests and capacity limits. Short lookahead incorporates predicted completions of ongoing requests. For exact optimization without lookahead, we prove a comparison under independent geometric decode lengths, bounded independent nondegenerate prompt sizes, and batched releases satisfying a cumulative supply condition. The long-run expected imbalance improvement over first come, first served admission with request-count join-shortest-queue grows at least as the square root of the product of worker capacity and the logarithm of worker count, in the stated scaling regime. The comparison extends to capacity-aware round robin, random routing, and power of two choices. BalanceRoute approximations connect the formulation to practical routing. Experiments using up to 144 NPUs with proprietary and public traces show lower workload spread and higher throughput than the evaluated baselines. These empirical results complement the guarantee for exact optimization.

cs.DC↗
arXiv2026

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

The landscape of large language models (LLMs) has become increasingly fragmented, with no single model dominating the field. Meanwhile, cloud providers offer a diverse array of mid-tier and older-generation GPUs. These heterogeneous resources provide better availability and deliver comparable performance-per-dollar to top-tier hardware. To efficiently harness these GPUs for concurrent multi-LLM serving, we introduce Coral: an adaptive, heterogeneity-aware serving system. The key idea behind Coral is the joint optimization of resource allocation and replica serving strategies across all models. To keep pace with shifting throughput demands and resource availability, Coral employs a lossless, two-stage decomposition. This approach preserves joint optimality while reducing online solve times from hours to mere tens of seconds. Evaluations across 6 models and 20 GPU configurations demonstrate that Coral reduces serving costs by up to 2.79$\times$ compared to the strongest baseline, while delivering up to 2.39$\times$ higher goodput under scarce resource constraints.

cs.DC↗
arXiv2026

Domain Extension of Lock-Freedom and Wait-Freedom for Group Computations

A domain extension of a definition refers to broadening the scope of a definition so that it applies to a larger set of cases than originally specified. The notion of lock-free and wait-free computation is designed for the domain of tasks that are completed by a single thread (in competition with other threads). The goal of this paper is to extend the definition of lock-freedom and wait-freedom to group-computations (denoted by gl-freedom and gw-freedom) that require that the task at hand must be completed by a collaboration between multiple threads. When extending a definition, certain constraints must be respected: the new domain must remain logically consistent with the original meaning, the extension should not introduce contradictions or ambiguities, and it must preserve the essential properties that make the definition valid and useful. We demonstrate this by showing that our extended definition is consistent with the original definition when the group consists of a single thread. We note that extension allows us to characterize programs in a new domain (distributed computing, NUMA computation systems, systems with private data for different threads, etc.) instead of relegating them to be in the same category (deadlock/livelock-free) without regard to the actual properties of that program. We also illustrate this definition with various examples.

cs.DC↗