arXiv subjects

Daniel Milroy

Publications and source records attributed to Daniel Milroy.

10 recordsLinked to original sources

arXiv2026

Descriptive Dispatch of Computational Work

Agents powered by AI/ML are becoming ingrained in orchestration. Dispatch of work is the task of receiving a request, transforming it for a workload manager, and successfully submitting it. Running scientific workflows across multi-cluster environments introduces substantial challenges of dynamic job transformation, dispatch, and submission to heterogeneous clusters. These tasks are well-suited to agents, which can receive textual instructions for work, prepare job specifications, and dispatch. In this work, we assess the reliability of a dispatch agent across 432 runs, testing all possible combinations of five feature dimensions across four prompt styles. The agent is highly reliable (97.9% success). We test a full orchestration to submit, queue, match, score, select, transform, and dispatch in a multi-cluster experiment. We find that descriptive metadata increases successful execution from 48% to 87% of 220 submitted jobs, eliminating architecture mismatch, and improving performance for five of ten measurable applications by up to 3.3x.

cs.DC↗

arXiv2026

Hierarchical Server Architecture for Agentic Science

Agentic science is transforming the landscape of computational work, and is applied to scientific pipelines and workload managers. Scientific workloads require specialized hardware within and between institutions. Automated resource discovery is an essential step for scheduling workloads with specific hardware and environmental requirements. In this paper, we present a hierarchical, dynamic architecture and accompanying software to discover resources across diverse cloud, edge, and HPC systems. The design enables concurrent, asynchronous negotiation, selection, and dispatch of requests for work using secretary agents. The agents probe and discover 51 real and simulated providers across 7 categories. We perform 19,973 negotiation and 6,952 selection simulations to assess reliability of decisions, demonstrating high (87.71%) negotiation accuracy and selection costs comparable to more traditional strategies. Designed for extensibility and currently supporting the US DOE Genesis Mission, this architecture exemplifies the importance of careful coordination between agents, discovery tools, and infrastructure for agentic science.

cs.DC↗

arXiv2026

Examining QRMI as a Unified Interface for Quantum-HPC Integration

The efficient and scalable integration of quantum resources into high-performance computing (HPC) environments requires standardized mechanisms for resource management, scheduling, and workflow orchestration across diverse and heterogeneous infrastructures. The Quantum Resource Management Interface (QRMI) addresses this challenge through a thin, vendor-agnostic middleware layer that provides standardized APIs for scheduling, executing, and monitoring quantum workloads while exposing quantum resources as first-class schedulable resources alongside CPUs and GPUs. Although previous work demonstrated QRMI integration with the Slurm workload manager, its applicability across other workload managers remained unexamined. This paper extends the validation of QRMI to a broad range of workload managers, including PBS, LSF, Grid Engine, Kubernetes, and the Flux Framework, encompassing traditional batch schedulers, a cloud-native orchestration platform, and a graph-based scheduler. We examine the integration patterns, implementation requirements, and scheduler-specific considerations associated with each environment and compare QRMI with alternative approaches to quantum resource integration. We demonstrate that QRMI provides a portable and flexible abstraction layer that minimizes scheduler-specific modifications while enabling consistent access to heterogeneous quantum resources across both on-premises and cloud environments.

cs.ET↗

arXiv2026

Descriptive Execution of HPC Applications and Workflows

The means to execute and orchestrate software components has changed from human-written code to descriptive prose. In high performance computing, this transition is represented in application orchestration, workload management, and system monitoring and debugging, to name a few. The underlying means to enable descriptive definition of tasks is the use of the Large Language Model with associated tool functions and resources. A combination of a model with access to such resources, modeled in software, encompasses an autonomous framework. As fully automated and agentic frameworks are developed for science, it is important to assess reliability and strategies scoped to specific tasks. In this work, we assess the extent to which an agentic framework can optimize and run an HPC scaling study with a low latency network in Amazon Web Services, accurately transform HPC job specifications between workload managers, and design and run an entire biosciences workflow. We find that the framework completes all three tasks while surfacing task-specific failure modes. In the scaling study, agents deploy and optimize applications but monitor running jobs inefficiently, preferring conservative fixed waits over event subscriptions. In job translation, they convert specifications between Slurm and Flux with high accuracy, with processor-affinity flags the most common error. In the bioscience workflow, the agent reproduces an expert-written variant-calling pipeline almost exactly -- agreeing with the reference call set in 18 of 19 completed runs -- and reaches this result through many distinct yet functionally equivalent workflow implementations. This information is invaluable moving forward to developing multi-cluster setups with scheduling and transformation handled by agents.

cs.DC↗

arXiv2026

Hybrid Quantum and Classical Workload Management with Graph-based Scheduling

High Performance Computing (HPC) centers are expanding to integrate quantum resources, enabling hybrid quantum-classical workflows for complex optimization. Integrating quantum processing units (QPUs) into workload managers poses an orchestration challenge: a remote QPU introduces a second queue - a "two-queue problem" - alongside the scheduler's own. We present Fluence, a Kubernetes scheduler plugin backed by the Fluxion graph-based scheduler, enabling gang-scheduled placement for quantum-classical workloads and custom resources. First, under contention, Fluence's atomic gang placement eliminates the node-time a default scheduler wastes on partially placed gangs. Second, a synchronization primitive gates consumers behind a single producer's shared quantum task, cutting worker idle time roughly 1.2-12x under short queues and orders of magnitude under long ones. Third, policy-aware backend selection cuts mean per-run cost roughly 72x and time-to-result from hours to under two minutes. Together, these results show that quantum-awareness can be added to a cloud-native scheduler without modifying user containers.

quant-ph↗

arXiv2026

Agentic Orchestration of HPC Applications in Cloud

Large Language Models (LLMs) are serving as a catalyst of change for research practices, touching the daily lives of staff scientists, software engineers, and system administrators. The developments promise new degrees of autonomy, where categories of human work and decision making are replaced by autonomous, goal-oriented systems. This transition necessitates novel architectural paradigms and solid understanding of the strengths and limitations of LLMs. In this work, we design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study. We pursue this task for four well-known HPC applications to build multi-platform images and optimize across 21 instance types in Kubernetes. We demonstrate successful linear scaling with patterns approved by human experts, designs that improve work time to completion, and review suggested best practices for agentic design and collaboration.

cs.DC↗

arXiv2025

Possible Futures for Cloud Cost Models

Cloud is now the leading software and computing hardware innovator, and is changing the landscape of compute to one that is optimized for artificial intelligence and machine learning (AI/ML). Computing innovation was initially driven to meet the needs of scientific computing. As industry and consumer usage of computing proliferated, there was a shift to satisfy a multipolar customer base. Demand for AI/ML now dominates modern computing and innovation has centralized on cloud. As a result, cost and resource models designed to serve AI/ML use cases are not currently well suited for science. If resource contention resulting from a unipole consumer makes access to contended resources harder for scientific users, a likely future is running scientific workloads where they were not intended. In this article, we discuss the past, current, and possible futures of cloud cost models for the continued support of discovery and science.

cs.DC↗

arXiv2025

Usability Evaluation of Cloud for HPC Applications

The rise of AI and the economic dominance of cloud computing have created a new nexus of innovation for high performance computing (HPC), which has a long history of driving scientific discovery. In addition to performance needs, scientific workflows increasingly demand capabilities of cloud environments: portability, reproducibility, dynamism, and automation. As converged cloud environments emerge, there is growing need to study their fit for HPC use cases. Here we present a cross-platform usability study that assesses 11 different HPC proxy applications and benchmarks across three clouds (Microsoft Azure, Amazon Web Services, and Google Cloud), six environments, and two compute configurations (CPU and GPU) against on-premises HPC clusters at a major center. We perform scaling tests of applications in all environments up to 28,672 CPUs and 256 GPUs. We present methodology and results to guide future study and provide a foundation to define best practices for running HPC workloads in cloud.

cs.DC↗

arXiv2024

A Contention-Free Model for Converged Kubernetes on HPC

High performance computing (HPC) and cloud have traditionally been separate, and presented in an adversarial light. The conflict arises from disparate beginnings that led to two drastically different cultures, incentive structures, and communities that are now in direct competition with one another for resources, talent, and speed of innovation. With the emergence of converged computing, a new paradigm of computing has entered the space that advocates for bringing together the best of both worlds from a technological and cultural standpoint. This movement has emerged due to economic and practical needs. Emerging heterogeneous, complex scientific workloads that require an orchestration of services, simulation, and reaction to state can no longer be served by traditional HPC paradigms. However, while cloud offers automation, portability, and orchestration, as it stands now it cannot deliver the network performance, fine-grained resource mapping, or scalability that these same simulations require. These novel requirements call for change not just in workflow software or design, but also in the underlying infrastructure to support them. This is one of the goals of converged computing. While the future of traditional HPC and commercial cloud cannot be entirely known, a reasonable approach to take is one that focuses on new models of convergence, and a collaborative mindset. In this paper, we introduce a new paradigm for compute -- a traditional HPC workload manager, Flux Framework, running seamlessly with a user-space Kubernetes "Usernetes" to bring a service-oriented, modular, and portable architecture directly to on-premises HPC clusters. We present experiments that assess HPC application performance and networking between the environments, and provide a reproducible setup for the larger community to do exactly that.

cs.DC↗

arXiv2023

The Flux Operator

Converged computing brings together the best of both worlds for high performance computing (HPC) and cloud-native communities. In fact, the economic impact of cloud-computing, and need for portability, flexibility, and manageability make it not important, but inevitable. Navigating this uncharted territory requires not just innovation in the technology space, but also effort toward collaboration and sharing of ideas. With these goals in mind, this work first tackles the central component of running batch workflows, whether in cloud or HPC: the workload manager. For cloud, Kubernetes has become the de facto tool for this kind of batch orchestration. For HPC, the next-generation HPC workload manager Flux Framework is analogous -- combining fully hierarchical resource management and graph-based scheduling to support intelligent scheduling and job management. Convergence of these managers would mean the implementation of Flux inside of Kubernetes, allowing for hierarchical resource management and scheduling that scales impressively without burdening the Kubernetes scheduler itself. This paper introduces the Flux Operator -- an on-demand HPC workload manager that is easily deployed in Kubernetes. The work here highlights design decisions, mapping of components between environments, experimental features, and shares the results of experiments that compare performance with an equivalent operator in the space, the MPI Operator. Finally, discussion closes with a review of challenges remaining, and hopes for the future for improved technological innovation and collaboration.

cs.DC↗