arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.38196

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

Qi Qi·Yifan Sui·Hanfei Yu·Hong Zhang·Hao Wang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.

Explore related subjects

cs.DCcs.LGcs.OS

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang. 2026-10-28. GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading. https://arxiv.org/abs/2610.38196

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Rank-Aware CPU Reservation and In-Place Adjustment for MPI CFD on Kubernetes

Non-uniform meshes in computational fluid dynamics (CFD) can leave some Message Passing Interface (MPI) ranks waiting while others determine execution time. On Kubernetes, reserving a full CPU for every rank can then restrict the admission of further simulations. We investigate rank-aware CPU reservation and in-place request adjustment using OpenFOAM on three Kubernetes testbeds. The framework lowers reservations without hard CPU caps and uses online MPI profiles to adjust each rank's CPU request while the solver runs. We reduced the per-job CPU reservation from 16.0 to 5.9 vCPU, with single-job wall-clock time increasing by at most 6%. With these reservations on twelve nodes, six co-located simulations reached 4.09 times the throughput of sequential execution. Matched controls showed no consistent benefit from proportional request shaping. Beyond the nominal CPU capacity, throughput collapsed with default busy-waiting; yielding waits softened the collapse. In one six-simulation run, the online controller performed 120 in-place resizes of running solvers with zero pod restarts; the pipeline reached 2.5 times the throughput of sequential execution. In a separate comparison on 16 physical cores, reallocating ranks from the far field to the near-wall zone reduced fixed-iteration wall-clock time by factors of 2.0 to 4.2 relative to the original concentric allocation, depending on mesh and iteration horizon. The reallocation also outperformed balanced Scotch partitions. The implementation requires no changes to OpenFOAM, the MPI library, or the Kubernetes scheduler.

cs.DC↗
arXiv2026

The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures

Power capping is a common GPU energy lever in LLM serving, and it appears to work: the cap is set, the GPU reports drawing less than it, and the energy budget is met. We show the cap does little for the phase that dominates LLM serving: autoregressive decode. Across four attention paradigms (GQA, MLA, Gated DeltaNet, and Mamba2) on an NVIDIA H200 under vLLM eager execution, decode kernels wait on HBM and the GPU idles between kernel launches, so decode draws less than half of the GPU's 700W rating and throughput stays within a few percent under every cap we set. The cap engages only when a large batch runs a long generation. Locking the SM clock instead lowers power without slowing memory-bound decode and recovers up to 32% of decode energy at minimal throughput loss. We identify three architecture-dependent frequency-scaling classes and find that, once prefill is measured without prefix caching, the recurrent architectures pay no prefill penalty: their linear-time mixing undercuts quadratic attention on long prompts, so their efficient decode adds to the prefill saving and lowers total request energy from the first output token at large batch.

cs.DC↗
arXiv2026

Simple-IT: Practical Low-Latency Signature-Free BFT Consensus

Recent advances in quantum computing pose a looming threat to most current Byzantine fault-tolerant (BFT) consensus protocols, which rely on quantum-vulnerable public-key signature schemes such as Ed25519 and BLS12-381. Instead of switching to much more expensive post-quantum secure signature schemes, an alternative is to use signature-free protocols, which rely only on cheap, post-quantum secure authenticated channels. In this paper, we ask whether signature-free BFT consensus protocols can match the performance of current state-of-the-art, quantum-vulnerable BFT consensus protocols. While previous work on the Sailfish++ protocol showed that state-of-the-art throughput is attainable signature-free, the question of latency is still open. Several recent signature-free protocols have low latency in theory, but they are all very intricate, and no practical implementation has so far been presented. In this work, we propose Simple-IT, a new leader-based, signature-free BFT consensus protocol that achieves a theoretical latency of 4 message delays (one more than the optimum), and only 3 on its optimistic path. Crucially, Simple-IT is simple enough to be amenable to implementation and to practical optimizations such as speculative pipelining, and, as we show experimentally in a geo-distributed testbed, it achieves both throughput and latency competitive with state-of-the-art quantum-vulnerable protocols.

cs.DC↗