arXiv Science⌕ Search

arXiv subjects

Junning Zhu

Publications and source records attributed to Junning Zhu.

3 recordsLinked to original sources

Ostinato: Fast CPU-GPU MoE Inference for Local Agents via Resident Experts in Every Forward Pass

Language-model agents increasingly run on workstations with a few consumer GPUs, where large mixture-of-experts (MoE) models keep part of their experts in host memory. With existing CPU-GPU engines, these agents get fast prefill or fast decode, not both: llama.cpp's expert cache skips forward passes of more than 32 tokens to keep the experts of decode cached, so an agent waits 2.1-2.8 s after each tool output, whereas its default layer placement shortens this wait but is 1.5-1.6$\times$ slower in decode. Our key insight is that a short prefill touches too many distinct experts to upload, yet sends 84% of its expert activations (routes) to a stable set of experts that fits in GPU memory. We present Ostinato, a CPU-GPU MoE inference engine that computes these experts on the GPUs in every forward pass without changing what the model computes. Short prefills split their routes, leaving fewer than one of a token's six routed experts per layer to the CPU; long prefills upload each distinct cold expert once; and each decode step runs as one fixed-shape CUDA graph per GPU. On real coding and customer-service agent sessions with DeepSeek-V4-Flash on four RTX 4090D GPUs, Ostinato is the fastest engine in both phases and finishes every session sooner than all six baselines. Against llama.cpp and KTransformers, it cuts the median wait after a tool output by 22-55% and finishes sessions 1.4-2.5$\times$ sooner.

cs.DC↗

GenCoord: Executable Commitments for Embodied LLM Agents under Private Information

Embodied language-model agents must coordinate interdependent actions while each holds private task information. In dialogue-based coordination, models generate messages and interpret replies to decide what each agent should do, spending inference before the team acts. We propose GenCoord, a decentralized protocol that makes the division of work itself the message. A proposal pairs the sender's skill path with its partner's continuation in Short DSL, a compact language for expressing joint work. Compatible paths form commitments linking the sender's delivery to the partner's work after verified receipt. We check fulfillment separately from original-task success. Complete requests also let a deterministic receiver proceed without another model call. In Minecraft, trained local 0.8B agents match the task success of controlled free-form communication while reducing median commitment time by 68.2%. The same-field full-key JSON control approaches free-form output length and delay, indicating that the amount generated matters even when messages are structured. With the same prompted DeepSeek model, both protocols complete nearly all tasks, but dialogue generates 2.8 times as many output tokens. Multistep commitments improve held-out-template success from 91.3% to 98.1% with fewer decisions. These results point to a design principle for embodied cooperation: make the division of work the message, and spend inference on choosing and revising the work.

cs.AI↗

Position: Recommender Systems Should Move Beyond Platform-Centric Ranking toward Personal Agent-Mediated Recommendation

Recommender systems are usually framed as ranking systems: platforms observe users, construct candidate sets, and select items on their behalf. This framing hides a deeper allocation of control, in which platforms also determine candidate access, evidence boundaries, explanations, and the path from user need to recommended output. We argue that the next bottleneck in recommendation is not only preference modeling, but control over evidence acquisition and disclosure. We argue for \textbf{Personal Agent-Mediated Recommendation} (PAMR), a paradigm in which a user-facing personal agent represents the user in discovering, filtering, aggregating, and governing recommendation evidence across distributed sources. The central shift is not simply from one ranking model to another, but from platform-side item ranking to user-side evidence mediation. As a position paper, we define PAMR as a new recommendation paradigm, establish its boundary criteria, identify its core mediation decisions, and propose a mediation-centered evaluation framework. A proof-of-concept study on hard Yelp restaurant recommendation tasks further shows that, under a shared LLM ranker, source selection and controlled disclosure provide the strongest observed utility--traceability--exposure--cost operating point.

cs.IR↗