arXiv subjects

Kui Luo

Publications and source records attributed to Kui Luo.

6 recordsLinked to original sources

arXiv2026

FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving

In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capacity wall and results in low concurrency. Offloading the KV cache to host dynamic random-access memory (DRAM) frees HBM space and raises concurrency, but the additional cache management and host-to-device (H2D) load overhead inflates inference latency, higher concurrency fails to translate into effective throughput. We present FlashOffloading, a host-DRAM-centric inference architecture that fully offloads the KV cache in both the prefill and decode phases. FlashOffloading builds rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero extra miss-lookup overhead, and a compute--load overlapped kernel, optimizing KV offloading latency across the entire pipeline and delivering throughput gains under a time-per-output-token (TPOT) service-level objective (SLO). In experiments deploying GLM-5.2 with multi-token prediction (MTP3) speculative decoding on an Ascend910C prefill--decode (PD)-disaggregated platform, FlashOffloading improves prefill throughput by up to 1.38$\times$ and decode throughput by up to 1.91$\times$ under a 30 ms TPOT constraint. FlashOffloading is integrated into an open-source serving framework: https://github.com/vllm-project/vllm-ascend/tree/releases/v0.30.0.

cs.AI↗

arXiv2026

MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference

Sparse expert activation reduces MoE models' computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memory-constrained MoE inference. It stages complete expert layers in alternating buffers during prefill. During decode, it combines nonuniform layer-wise cache capacity, domain-informed initialization, routing-history-aware replacement, and cross-layer prefetching. The main configuration executes router-selected experts exactly; an optional score-based substitution path handles eligible low-score misses. The main comparison uses 1K- and 128-token output caps for MoE-CORE and vLLM Prefetch, respectively. Across five workloads per model, MoE-CORE records a mean time per output token (TPOT) of 38.0-44.8 ms versus 1268.9-1269.1 ms for the evaluated vLLM Prefetch configuration on DeepSeek-V4-Flash-W4A8; the corresponding values on GLM-5.2-W4A8C8 are 206.6-220.5 and 5941.5-5941.8 ms. Under an 84-GB NPU-memory cap, the best measured DeepSeek GSM8K configuration achieves a TPOT of 21.5 ms with approximate expert substitution and multi-token prediction (MTP) at depth 2. These results support coordinated expert residency and transfer scheduling under a device-memory constraint. The code is here.

cs.DC↗

arXiv2026

AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs). Traditional task-collocated RL frameworks suffer from significant scalability bottlenecks, while task-separated RL frameworks face challenges in managing complex dataflows and resolving resource idling. Furthermore, most existing frameworks are tightly coupled with LLM training or inference engines, making them difficult to support custom-designed engines. To address these challenges, we propose AsyncFlow, an asynchronous streaming RL framework tailored for efficient post-training. Specifically, we introduce a distributed data storage and transfer module that provides panoramic data management and fine-grained scheduling capabilities in a fully streamed manner. This architecture inherently enables automated pipeline overlapping among RL tasks and dynamic load-balancing. Moreover, we propose an asynchronous producer-consumer workflow, which is engineered to minimize computational idleness by strategically deferring the parameter update process within staleness thresholds. Finally, the core capabilities of AsyncFlow are architecturally decoupled from underlying training and inference engines and encapsulated by service-oriented user interfaces, offering a modular and customizable user experience. Extensive experiments demonstrate an average throughput of 1.59x compared to the state-of-the-art baseline. The architecture presented in this work provides actionable insights for designing next-generation RL training systems.

cs.LG↗

arXiv2026

VPP: Virtual Pipeline Parallelism for Efficient Chunked Prefill in Long-Context LLM Inference

Chunked prefill pipeline parallelism (CPP) is a key technique for LLM inference. However, equal-size chunks exhibit imbalanced latency, as later chunks attend longer prefix KV caches and incur higher attention costs, leading to pipeline bubbles. Existing approaches mitigate this imbalance through dynamic chunk resizing (Dynamic CPP, DCPP), but our measurements show that this trades scheduling overhead for load balancing, which becomes unfavorable on long sequences. In this study, we propose Virtual Pipeline Parallelism (VPP), which keeps chunk sizes fixed and optimizes the pipeline layout through virtual stages. A V-shaped virtual-stage traversal overlaps each chunk's expensive middle stages with the lighter head and tail stages of its neighbors, while asynchronous communication and pipelined packing further reduce communication stalls and cross-request drain bubbles. We implement VPP in vLLM-Ascend and evaluate it on three MoE-based LLMs with sequences up to 1M tokens on 16 Ascend 910C NPUs. VPP improves throughput by up to 13.1% over DCPP on long sequences and 6.7% on mixed workloads, while preserving performance on short sequences. On a 512K-token DeepSeek-V3.1 prefill workload, VPP reduces the pipeline bubble ratio from 6.4% to 0.1%, achieving a 98.0% reduction compared with DCPP.

cs.DC↗

arXiv2025

Controlled Evolution-Based Day-Ahead Robust Dispatch Considering Frequency Security with Frequency Regulation Loads and Curtailable Loads

With the extensive integration of volatile and uncertain renewable energy, power systems face significant challenges in primary frequency regulation due to instantaneous power fluctuations. However, the maximum frequency deviation constraint is inherently non-convex, and commonly used two-stage dispatch methods overlook causality, potentially resulting in infeasible day-ahead decisions. This paper presents a controlled evolution-based day-ahead robust dispatch method to address these issues. First, we suggest the convex relaxation technique to transform the maximum frequency deviation constraint to facilitate optimization. Then, an evolution-based robust dispatch framework is introduced to align day-ahead decisions with intraday strategies, ensuring both frequency security and power supply reliability. Additionally, a novel controlled evolution-based algorithm is developed to solve this framework efficiently. Case studies on a modified IEEE 14-bus system demonstrate the superiority of the proposed method in enhancing frequency security and system reliability.

eess.SY↗

arXiv2019

Small strain induced large piezoelectric coefficient in $α$-AsP monolayer

Strain engineering can effectively tune the electronic, topological and piezoelectric properties of materials. In this work, the small strain (-4\% to 4\%) effects on piezoelectric properties of $α$-AsP monolayer are studied by density functional theory (DFT). The piezoelectric stress tensors $e_{ij}$ and elastic stiffness tensors $C_{ij}$ are reported by using density functional perturbation theory (DFPT) and finite difference method (FDM). It is found that the Young's modulus of $α$-AsP monolayer shows very strong anisotropy, and the armchair direction is very softer than zigzag direction, which provides possibility for tuning easily piezoelectric coefficients along the armchair direction. In considered strain range, uniaxial compressive (tensile) strain along the armchair (zigzag) direction is found to raise observably both the $e_{22}$ and $d_{22}$ (absolute value). In fact, both compressive strain along the armchair direction and tensile strain along the zigzag direction essentially reduce the lattice constants along the armchair direction, which can enhance the piezoelectric coefficients. The $e_{ij}$ of $β$-AsP monolayer as a function of strain is also studied to illustrate the importance of particular puckered structure of $α$-AsP in enhancing the piezoelectric coefficients. A classic SnSe monolayer with puckered structure is used to further declare that small strain along the armchair direction can effectively improve the piezoelectric coefficients. For example, the $d_{22}$ of SnSe monolayer at -3.5\% strain is up to 628.8 pm/V from unstrained 175.3 pm/V. For SnSe monolayer, a large peak is observed for $e_{22}$, which is due to a structural phase transition. For $e_{16}$ of SnSe monolayer, a large peak is also observed due to the cross of lattice constants $a$ along the zigzag direction and $b$ along the armchair direction.

cond-mat.mtrl-sci↗