arXiv Science⌕ Search

arXiv · 2611.10022

How Much Skill Orchestration Can LLM Agents Compile Away? Execution-Grounded Specialization for Token-Efficient Skill Use

Abstract

Reusable skills guide large language model (LLM) agents through recurring requests, but each invocation still requires the model to interpret the skill and reconstruct its orchestration. Can agents automatically compile this orchestration into an executable fast path without human intervention? We first study zero-shot skill specialization, which compiles without execution evidence, and find that despite the skill's procedural guidance, incorrect assumptions about runtime bindings can make generated plans brittle and recovery costly. This motivates SkillCache, a one-trace skill specialization method that compiles a completed invocation into a parameterized capsule of programs and natural-language steps, replays it before reuse, and drives it with a plan executor that returns control to the model only when needed, keeping the source skill for recovery. With the Claude Code harness, warm execution of the same request reduces tokens on 87 SkillsBench tasks by 81% with Claude Sonnet 4.6 and by 74% with Claude Opus 4.8 at comparable reward, and capsules stay effective under controlled request changes. Capsules also transfer across models, raising three open-weight models' mean reward by 12-25 percentage points while reducing their tokens by 72-81%, suggesting that execution-grounded compilation is a promising approach to efficient repeated skill use and cross-model transfer.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jiayi Ni, Zac Wang, Alex Zhang, Zheng Liu, Roopa Prabhu. 2026-11-05. How Much Skill Orchestration Can LLM Agents Compile Away? Execution-Grounded Specialization for Token-Efficient Skill Use. https://arxiv.org/abs/2611.10022

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Understanding Developer Information Needs for Software with Embedded Natural Language Prompts

Developers use foundation models (FMs) to create novel forms of intelligent software by writing natural language prompts embedded within a software's implementation---referred to as embedded prompts---that are also passed as a string within an FM's API. Developing software with embedded prompts requires developers to engage in iterative embedded prompt changes to obtain the desired behavior. While the information needs of traditional software have been investigated, the information needs of modern developers for embedded prompt changes are underexplored, despite these needs shaping how developers plan and justify embedded prompt changes. In this work, we perform a mixed-method exploratory study on the information needs of developers making changes to embedded prompts. We create, refine, validate, and apply a list of 50 information needs that developers have while making a change to an embedded prompt by interviewing 16 developers, observing 8 developers make prompt changes, surveying 50 developers, and evaluating 72 research and commercial tools. We find that embedded prompts are not well-supported by existing tools: 16 of the 50 information needs, including four of the most important ones, are unaddressed. We conclude with important design opportunities that should be addressed by future developer tools for embedded prompts.

cs.SE↗

Enhancing Neural Code Representation with Additional Context

Automated program comprehension underpins many software engineering tasks, from code summarisation to clone detection. While recent deep learning models have achieved strong results, they typically rely on source code alone, overlooking contextual signals such as version history or structural relationships. We conduct an empirical study of how enriching code representations with such signals affects model performance on four tasks: code clone detection, code classification, code summarisation, and vulnerability detection, using three datasets (SeSaMe, CodeSearchNet-Java, Vul4J). We evaluate five foundational code models (CodeBERT, GraphCodeBERT, CodeT5, PLBART, ASTNN) and seven code-specialised LLMs (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder), comparing prompt-based context injection against fine-tuned use of context. Context improves classification for all five foundational models and, after fine-tuning, for most LLMs. For summarisation, the effect depends on the model family: version history improves all four foundational models on every metric, whereas prompt-level context helps two of the seven LLMs and hurts the other five. Combining context types reaches up to +21.48% macro-F1 in code classification and +15.04% F1 in clone detection. A human evaluation of CodeT5 summaries on 100 Java snippets ranks context-augmented summaries significantly higher for Accuracy and Content Adequacy. An exploratory vulnerability detection evaluation provides initial evidence of transfer to security-oriented tasks.

cs.SE↗

Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

LLM agents are explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Evaluating eleven LLMs on Cloud-OpsBench reveals that strong outcome accuracy often masks incomplete investigations: while Joint RCA Accuracy (JRA) reaches up to 0.78 on OnlineBoutique and 0.68 on TrainTicket, models identify the correct root cause without achieving evidence closure (e.g., in 33.1% and 29.4% of evaluated cases for GPT-5). These results demonstrate that final-answer accuracy alone cannot establish whether an agent has traced the failure and collected the evidence needed to confirm its root cause.

cs.SE↗