Distributed Load Balancing with Workload-Dependent Service Rates

Abstract

Modern service systems, including cloud platforms and large language model inference endpoints, must distribute jobs across servers whose processing speeds depend on current workloads. At scale, centralized coordination is costly, while naive distributed policies can perform arbitrarily poorly. We study how to design a simple distributed load balancing policy that achieves globally optimal latency performance in such settings. We model the system as a bipartite queueing network with an arbitrary compatibility graph and servers with concave, workload-dependent service rates. We propose the Greatest Marginal Service Rate (GMSR) policy, which routes jobs to a connected server where it has the largest marginal impact on service rate. In a discrete-time stochastic model, we show that as time discretization is refined (shrinking time step and job size proportionally), the scaled workload process converges almost surely to a fluid limit governed by a differential inclusion. In the fluid regime, GMSR reaches an $\epsilon$-suboptimal solution in $\mathcal{O}(\delta + \log(1/\epsilon))$ time from any $\delta$-suboptimal initial state, implying global convergence to the centrally optimal routing. When the system is overloaded, GMSR maximizes throughput, maximizes the number of stabilized backends among throughput-optimal policies, and minimizes total workload over those stabilized backends. GMSR yields a practical routing rule that requires neither demand-rate knowledge nor centralized coordination. By relying only on local information, service providers can achieve near-optimal latency performance through decentralized decisions, making the policy well suited to large-scale cloud computing, LLM serving, and other distributed service environments where centralized control is costly or infeasible.