arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.14477

What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

Shaikh Akib Shahriyar·MD. Akil Raihan Iftee·Armon Barton·Matthew Wright
View original source ↗Read source document ↗Export metadata ↓

Abstract

Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the agent by what it does. The result tells us about the original task only if the changed test case is still a valid test of that task. Many evaluations decide this with weak rules, such as accepting any well-formed edit or any edit that stays close to the original in embedding space. We propose executable task contracts. A contract states which facts define a task, which edits are allowed, and what finishing the task means, and it checks each changed test case before the agent runs. We compare validity rules by scoring the same 14,336 stored agent runs under each rule. Across four encoders, embedding thresholds allowed 75-100% of edits that changed the task, such as swapping the payee, yet rejected 24-45% of valid hostile notes. With a MiniLM cosine threshold, Mistral appeared vulnerable on all 64 tasks, but only 26 had a successful valid attack. Replaying a saved solution and allowing edits to benchmark fields fail in similar ways. Evaluations should state, and check, what counts as the same task.

Explore related subjects

cs.SEcs.AIcs.CRcs.MA

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Shaikh Akib Shahriyar, MD. Akil Raihan Iftee, Armon Barton, Matthew Wright. 2026-10-10. What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents. https://arxiv.org/abs/2610.14477

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Enhancing Knowledge Density via Hierarchical Context Pruning for Retrieval-Augmented Repository-Level Code Completion

Retrieval-Augmented Generation (RAG) has advanced repository-level code completion by retrieving external code snippets, API documentation, or type definitions to expand the model's contextual awareness. However, under the finite token budget of the context window, existing retrieval-augmented methods often saturate the budget with highly similar or even redundant code, which not only wastes tokens but also suppresses critical cross-file dependencies, preventing synergistic interaction between semantically similar reference snippets and cross-file dependency definitions. Therefore, further gains are limited by low knowledge density in the retrieved context rather than by model capacity. We propose SaraCoder, a retrieval-augmented framework that enhances knowledge density via hierarchical context pruning under a fixed context budget. A Hierarchical Feature Optimization module prunes an over-complete candidate pool in four stages. The freed tokens are then allocated to an External-Aware Identifier Disambiguator, which selectively injects only the most necessary cross-file dependency definitions. SaraCoder transforms context construction into a budget-aware optimization problem, balancing reference snippets and cross-file dependencies to build a context that is both knowledge-dense and semantically complete. Experiments on CrossCodeEval and RepoEval-Updated, across multiple backbone code LLMs, show that SaraCoder consistently outperforms strong RAG baselines such as GraphCoder and Repocoder in both code-level and identifier-level metrics, without increasing token usage, and yields synergistic gains when combined with other cross-file methods.

cs.SE↗
arXiv2026

Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths

Large Language Models (LLMs) are central to automated code generation, yet most approaches use one model at every stage, from writing code to repairing and optimizing it. LLMs have complementary strengths: their failures do not fully overlap across programming languages, algorithmic categories, and development stages. For example, in Python, the model with the highest code-generation accuracy ranks last at code repair. Using one model to generate code, another to repair it, and a third to refine it can therefore achieve higher correctness and better execution performance than a single model throughout. We propose PerfOrch, a multi-LLM multi-agent system with four agents for categorization, generation, debugging, and refinement. The generation, debugging, and refinement agents select models through a Memory module that ranks candidate LLMs by their measured performance for each stage, programming language, and algorithmic category. We evaluate PerfOrch with five candidate LLMs on 4,640 problems from four benchmarks covering five programming languages (HumanEval-X, EffiBench-X, BigCodeBench, and APPS) against single-model direct generation, single-model generate-debug-refine pipelines (PerfCodeGen), and the state-of-the-art multi-agent systems MapCoder and CodeSIM. PerfOrch achieves the highest pass@1 in all twelve benchmark-language settings, exceeding the strongest baseline by an average of 8.19 percentage points, and reduces execution time for an average of 75.03% of solved problems with an average speedup of 1.18x. The Memory rankings remain effective on three unseen benchmarks without re-profiling. Sequential refinement matches exhaustive execution-time refinement coverage with 66% fewer refinement-stage tokens (57% for average-memory optimization). The complementary strengths persist after all five LLMs are replaced with their current-generation successors.

cs.SE↗
arXiv2026

Smart Hiring Redefined: An Intelligent Recruitment Management Platform

Against the backdrop of deepening digital and intelligent transformation in human resource management, traditional recruitment models struggle to fully meet enterprises' growing demand for precise talent acquisition due to limited efficiency, high costs, and information asymmetry. As a vital tool for optimizing recruitment processes, reducing labor and time costs, and enhancing core competitiveness, intelligent recruitment management systems have become an indispensable component of modern organizational talent strategies. Compared with the labor intensive tasks of resume screening, candidate position matching, and interview coordination in traditional manual recruitment, intelligent recruitment systems significantly enhance the efficiency and accuracy of the hiring process through automation and data driven approaches. These systems enable rapid parsing of massive resume volumes, intelligent matching of candidates to positions, and automated scheduling of interview processes. This substantially reduces the workload on human resources departments while improving recruitment quality and response speed. This research leverages the Java technology framework to design and implement an intelligent recruitment management system tailored for campus recruitment scenarios. The system establishes a collaborative platform connecting students, enterprises, and administrators through information technology and intelligent solutions, offering comprehensive functionalities including job posting distribution, resume submission, candidate position matching, and process management. Guided by the vision of Smart Campus Recruitment, the project delivers a more convenient job seeking experience for students and provides enterprises with more efficient talent screening and recruitment management services, thereby driving high quality development in university enterprise collaboration.

cs.SE↗