arXiv Science⌕ Search

arXiv · 2611.09474

Nebula: Distributed Position-Independent Caching for LLM Serving

Abstract

Modern LLM workflows, such as retrieval-augmented generation (RAG) and LLM agents, build long prompts from multiple independent text segments, making prefill a major source of computation. Reusing KV caches is therefore important for reducing serving costs and latency: distributed caching systems spread KV cache across a cluster to increase aggregate capacity, while position-independent caching (PIC) enables reuse of recurring segments under different prefixes. However, existing systems distribute KV caches but still rely on prefix caching (PC), while existing PIC methods enable position-independent reuse without cluster-wide resource management. We present Nebula, the first distributed LLM serving framework to treat PIC segments as first-class citizens, enabling remote PIC reuse and parallel prefill. To track PIC's fine-grained, asynchronous state changes, Nebula combines optimistic accounting with event-driven updates to provide a timely and truthful cluster view for cache management and prefill scheduling. To accommodate diverse reuse patterns across workloads, Nebula leverages cluster-wide ghost-hit feedback to automatically tune the allocation of shared cache capacity between PC and PIC. Finally, Nebula incorporates queueing, computation, and KV transfer costs into a unified prefill completion time model to guide elastic parallelism adjustment and locality-aware compute placement. Across four models and four real-world RAG and agentic workloads, Nebula reduces median TTFT by an average of 61.4% compared with the strongest existing baseline and recovers an average of 76.1% of the tokens missed by prefix caching.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yifei Liu, Junbin Pan, Junhao Hu, Yang Liu, Junyi Chen, Juntong Wu, Chen Chen, Haiyuan Zhao, Xiaoxu Chen, Weihang Chen. 2026-11-06. Nebula: Distributed Position-Independent Caching for LLM Serving. https://arxiv.org/abs/2611.09474

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Online Graph Exploration under T-Interval Connectivity

We study deterministic exploration by a single agent in $T$-interval-connected graphs, a standard model of dynamic networks in which, for every time window of length $T$, the intersection of the graphs within the window is connected. The agent does not know the window size $T$, nor the number of nodes $n$ or edges $m$, and must visit all nodes of the graph. We consider two visibility models: in $KT_0$, the agent observes only its current node's identifier and port information; in $KT_1$, it additionally observes the identifiers of its currently available neighbors. We investigate two fundamental questions: the minimum window size that guarantees exploration, and the optimal exploration time under sufficiently large window size. For both models, we show that a window size $T = Ω(m)$ is necessary, and we present deterministic algorithms whose required window size is $O(ξ(n,m)\cdot m + n \log^2 n)$, where $ξ(n,m) = \frac{\ln n}{1 + \ln m - \ln n}$. This window-size bound is tight for a wide range of $m$, in particular when $m = n^{1+Θ(1)}$. Separately, for exploration time, we prove lower bounds of $Ω((m - n + 1)n)$ in the $KT_0$ model and $Ω(m)$ in the $KT_1$ model, even when the dynamic graph is $\infty$-interval-connected (i.e., even when all snapshots share a common spanning tree), and our algorithms match these time bounds up to a polylogarithmic factor. These bounds are also tight when parameterized solely by $n$: $Θ(n^3)$ for $KT_0$ and $Θ(n^2)$ for $KT_1$.

cs.DC↗

Greedy-Like Defective Coloring: Distributed Algorithms and Applications

A $d$-defective $c$-coloring of a graph $G=(V,E)$ is a coloring of the nodes $V$ with $c$ colors such that every node has at most $d$ neighbors of the same color. Distributed algorithms for computing different variants of defective coloring are at the core of most deterministic state-of-the-art distributed coloring algorithms, and they are also an important tool in many other distributed graph algorithms. In several cases, the overall complexity could be improved if some version of defective coloring could be solved more efficiently. Barenboim and Elkin [STOC '09] introduced a two-pass greedy algorithm that uses $p^2$ colors with defect $\lfloor Δ/p\rfloor$ in $O(Δ+\log^{\ast} n)$ rounds. This remains the best defect/color tradeoff for $O(\log^{\ast} n)$-time algorithms in bounded-degree graphs. This paper expands the capabilities of this two-pass algorithm. First, we generalize it to the \emph{list defective coloring} problem (Fuchs and Kuhn, [DISC '23]). Consequently, we obtain an alternative algorithm for computing a proper $(Δ+1)$-coloring in $\tilde{O}(\sqrtΔ) + O(\log^{\ast} n)$ rounds in the CONGEST model. Second, we analyze a generalized two-pass algorithm for standard defective colorings. We prove that if the number of colors $c$ is not a perfect square, we can improve the state-of-the-art defect for distributed $c$-colorings by a constant factor in most cases. However, we also prove a limitation: for any $c\geq 1$, this generalized algorithm cannot achieve a $c$-coloring with defect below $(1-o(1))\cdotΔ/\sqrt{c}$.

cs.DC↗

Rendezvous under Variable Disorientation:The Algorithmic Power of Fixed Unit Distance

We study deterministic Rendezvous of two robots with lights under weak geometric and synchronization assumptions. Our focus is on Variable Disorientation (VD), where each robot may arbitrarily rotate, reflect, and rescale its local coordinate system at every \Look, and on VD+Fixed-Unit-Distance (FUD), where each robot's local unit distance remains fixed. We consider the one-sided visibility models FSTA and FCOM, and schedulers ranging from the energy-restricted RSYNCH and R-RSYNCH to fully asynchronous executions. Under VD, we prove color-independent impossibility results for FSTA under SSYNCH and for FCOM under ASYNCH, and establish sharp two-color boundaries under RSYNCH and R-RSYNCH. Under VD+FUD, we introduce periodic distance classes, which turn multipl icative changes of the physical distance into predictable cyclic phase shifts. This technique yields a two-color self-stabilizing non-$L$-Rendezvous algorithm for both FSTA and FCOM under RSYNCH and R-RSYNCH even with Non-Rigid movement. Under Rigid movement, periodic distance classes further yield an optimal three-color non-quasi-self-stabilizing algorithm for FSTA under SSYNCH, a four-color FSTA algorithm under $M$-atomic ASYNCH, and a four-color FCOM algorithm under CM-atomic ASYNCH. By guarding the latter phase mechanism with two additional colors and replacing its initial swap by passive initialization, we obtain a six-color FCOM algorithm under full ASYNCH, improving the previous twelve-color upper bound by a factor of two. These results show that a fixed local metric can serve as persistent cyclic memory, and clarify how metric stability, one-sided visibility, and scheduler atomicity jointly determine Rendezvous solvability and color complexity.

cs.DC↗