Enda Yu | arXiv Science

arXiv2025

LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers

Mixture-of-Experts (MoE) models face memory and PCIe latency bottlenecks when deployed on commodity hardware. Offloading expert weights to CPU memory results in PCIe transfer latency that exceeds GPU computation by several folds. We present PreScope, a prediction-driven expert scheduling system that addresses three key challenges: inaccurate activation prediction, PCIe bandwidth competition, and cross-device scheduling complexity. Our solution includes: 1) Learnable Layer-Aware Predictor (LLaPor) that captures layer-specific expert activation patterns; 2) Prefetch-Aware Cross-Layer Scheduling (PreSched) that generates globally optimal plans balancing prefetching costs and loading overhead; 3) Asynchronous I/O Optimizer (AsyncIO) that decouples I/O from computation, eliminating waiting bubbles. PreScope achieves 141% higher throughput and 74.6% lower latency than state-of-the-art solutions.

cs.LG↗

arXiv2024

Nezha: Breaking Multi-Rail Network Barriers for Distributed DNN Training

In distributed deep learning, communication remains a critical bottleneck. While modern hardware advances rapidly, over 60 percent of production HPC systems still rely on legacy infrastructure (V100 GPUs, multi-plane Ethernet/InfiniBand), necessitating communication optimization without hardware upgrades. Existing approaches face three key limitations: (1) static single-rail binding underutilizes multi-rail bandwidth, (2) protocol heterogeneity (TCP-RDMA coexistence) causes synchronization delays, and (3) mainstream libraries (NCCL/MPI) lack cross-protocol coordination. We present Nezha, the first protocol-agnostic system for multi-rail networks. Our contributions include: (1) Hardware-agnostic cross-protocol coordination: A unified abstraction enabling seamless collaboration between in-network computing (SHARP), adaptive RDMA (GLEX), and TCP, achieving 1.7 to 4.3 times lower latency than Gloo. (2) Protocol-aware dynamic load balancing: A hybrid scheduling strategy with cold/hot start state machine for heterogeneous protocols, reducing startup latency for small payloads while enhancing throughput for large transfers. (3) Fault-tolerant multi-rail collaboration: A self-recovery mechanism that reroutes data flows within 200 milliseconds upon single-rail failures, ensuring uninterrupted training. Experiments on 8-node clusters demonstrate Nezha achieves 74 percent and 80 percent higher throughput than MPTCP in homogeneous (TCP-TCP) and heterogeneous (TCP-SHARP) networks, respectively. On 128-node supercomputers, Nezha delivers 2.36 times higher training efficiency than Gloo. By bridging modern DNN communication demands with legacy infrastructure, Nezha proves that systematic multi-rail optimization can unlock the potential of aging clusters.

cs.DC↗

arXiv2021

CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation

Communication overhead is the key challenge for distributed training. Gradient compression is a widely used approach to reduce communication traffic. When combining with parallel communication mechanism method like pipeline, gradient compression technique can greatly alleviate the impact of communication overhead. However, there exists two problems of gradient compression technique to be solved. Firstly, gradient compression brings in extra computation cost, which will delay the next training iteration. Secondly, gradient compression usually leads to the decrease of convergence accuracy.

cs.LG↗