arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.26609

CQTwin: A Digital-Twin Toolkit for Operational Analytics of HPC Schedulers

Kanglin Xu·Yash Kurkure·Yihe Zhang·Michael E. Papka·Zhiling Lan
View original source ↗Read source document ↗Export metadata ↓

Abstract

HPC facilities collect years of job scheduler records, yet much of this operational data remains underused because accessible tools for turning it into actionable insight are scarce. We present CQTwin, an open-source digital-twin toolkit for operational analytics of HPC scheduling data. CQTwin addresses this gap through two complementary modes: offline trace replay and online event-driven simulation. Built on the Python-based CQSim discrete-event simulator, CQTwin combines a fidelity-validated, high-performance C++ engine with an interactive browser-based dashboard and a Snapshot Lab, enabling facility staff, operators, and researchers to characterize workloads, compare scheduling policies, and explore what-if scenarios. As an offline twin, CQTwin replays archived workload traces under alternative policies to show how the same workload would have unfolded under different scheduling decisions. As an online twin, it consumes job events as they arrive and projects how alternative policies would process the currently queued jobs, providing guidance for real-time decision-making. The dashboard supports the complete analysis workflow, including workload characterization, experiment configuration and monitoring, and multidimensional comparison of simulation results, allowing users to determine visually which policies best suit a workload and what trade-offs they entail. We validate CQTwin using production traces from the Argonne Leadership Computing Facility spanning 2016-2023.

Explore related subjects

cs.DCcs.PF

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kanglin Xu, Yash Kurkure, Yihe Zhang, Michael E. Papka, Zhiling Lan. 2026-10-19. CQTwin: A Digital-Twin Toolkit for Operational Analytics of HPC Schedulers. https://arxiv.org/abs/2610.26609

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Bitcoin-IPC: Scaling Bitcoin with a Network of Proof-of-Stake Subnets

Despite holding over $1.7T in value, Bitcoin scales poorly: Layer-1 (L1) processes only a few transactions per second, and Layer-2 (L2) solutions suffer from operational and liquidity fragmentation that requires external bridges for cross-chain connectivity. We present Bitcoin-IPC, a protocol that horizontally scales Bitcoin via permissionless, interconnected, programmable Proof-of-Stake L2 subnets staked in L1 BTC. Subnets leverage Bitcoin L1 for interconnectivity, settlement, and security, including Sybil protection, equivocating validator slashing, unilateral exit for honest stake, long-range attack protection, and compromised subnet firewalling. Bitcoin-IPC cross-subnet transfers involve only source/destination subnet validators and L1, with no external bridges or per-path liquidity lock-up. Embedded and batched in Bitcoin's witness mechanism, cross-subnet settlements can approach 6 vB per transfer, 23x less than a native L1 payment. Filling Bitcoin blocks with such batches corresponds to L1 monetary throughput of about 273 transfers per second, without modifying Bitcoin.

cs.DC↗
arXiv2026

Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods

Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, which can be interpreted as a method with a backup-worker strategy, and theoretically show that they are nearly optimal in many heterogeneous computation and communication scenarios, which is somewhat unexpected. We analyze the synchronous methods under random times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show theoretically that they are sufficient for many modern heterogeneous computation and communication scenarios.

cs.DC↗
arXiv2026

SIGMA: A Versatile Streaming Graph Partitioner for Vertex- and Edge-Balanced Distributed GNN Training

Distributed Graph Neural Network (GNN) training depends critically on how the underlying graph is partitioned across compute resources. Existing graph partitioners focus either on vertex partitioning or edge partitioning and typically optimize only a single communication objective (edge cut vertex cut) under a single balance constraint (vertex balance edge balance). We present SIGMA (Streaming Integrated Graph Partitioning with Multi-objective Awareness), a versatile streaming graph partitioner that supports both vertex and edge partitioning within a unified multi-objective, multi-constraint framework. Depending on the target distributed GNN system, SIGMA can be configured for edge-cut-oriented vertex partitioning or vertex-cut-oriented edge partitioning while simultaneously accounting for vertex and edge balancing. A clustering-based preprocessing stage incorporates global graph structure to improve partition quality while preserving the efficiency and scalability advantages of streaming partitioning. We evaluate SIGMA on six benchmark graphs spanning diverse domains and scales using two distributed GNN training systems: DistGNN (edge-partitioned) and DistDGL (vertex-partitioned). Across both settings, SIGMA consistently achieves strong performance, showing its ability to navigate complex trade-offs between partition quality, training efficiency, and memory consumption, frequently outperforming established streaming approaches and delivering competitive results against high-quality in-memory partitioners, with particularly strong results in edge-partitioned GNN training. These results demonstrate that a unified streaming partitioner can effectively address the communication, compute, and memory challenges of distributed GNN training across fundamentally different system architectures.

cs.DC↗