arXiv Science⌕ Search

arXiv subjects

Radin Shayanfar

Publications and source records attributed to Radin Shayanfar.

6 recordsLinked to original sources

Choosing Coding-Agent Benchmarks for Practice: Lessons from Task Demands and Agent Behaviour

Software teams evaluating coding agents need evidence that reflects the changes they expect those agents to make. Benchmark scores summarize how often an agent resolves tasks under broad labels such as bug fixing and feature implementation, but provide little guidance on whether the evaluated work resembles a team's workload. To support this decision, we characterize 2,487 tasks from five public benchmarks and analyze 14,922 runs of six agent configurations. We introduce the Spread-Novelty-Centrality (SNC) profile to describe how the reference solution is distributed across files and directories, how much it adds relative to what it removes, and the dependencies, history, and complexity of the code it touches. We also compare the scope of agent patches and their file exploration. Every pair of benchmarks differs on at least two SNC axes, including benchmarks with the same task label. Among tasks resolved by Claude Opus 4.6, patches are larger than the reference on FeatBench and smaller on FeatureBench. Across the six configurations, resolved runs tend to involve less distributed changes and code with lower values on the Centrality indicators, while the patch sizes associated with resolution differ between the Claude and Qwen configurations. These observations suggest that practitioners should select evaluation tasks based on their change characteristics, inspect the information provided in task descriptions, and assess patch scope alongside test outcomes. We explain how teams can apply this analysis to completed changes and identify the additional evidence needed before using it for task assignment or review automation.

cs.SE↗

The Software Engineering Factory: Engineering Accountability and Control in the Age of Abundant Code

AI agents can now plan, write, test, review, and revise code at a volume no human organization can inspect line by line. Final code review therefore cannot remain the primary organizational control for owning code changes. The Software Engineering (SE) Factory moves control into the process that produces the code. The Factory protocols define how work moves, what evidence must exist before it can proceed, and who owns the result. A deterministic protocol engine, implemented in conventional code rather than prompts, enforces those protocols by controlling stage transitions, requiring evidence, and invoking gates and human escalation, where needed, before work can advance. The SE Factory preserves human authority over consequential decisions and makes accountable acceptance possible without requiring humans to supervise every agent action. The SE Factory is not a dark factory. Humans remain in control by defining the scope and granularity of delegation, the boundaries and escalation conditions, and the consequential decisions that require accountable human judgment. We draw on our experience building three distinct SE Factory implementations, all of which have undergone multiple iterations and two of which are publicly available as open-source systems, into ten principles for conceiving, building, and operating SE Factories at scale, grounded in the organizational and technical realities of our implementations. We do not claim that these principles are exhaustive or constitute the only viable foundation for an SE Factory. Rather, we offer them as a conceptual scaffold and shared vocabulary for reasoning about SE Factories, evaluating their designs, and guiding their evolution. Our goal is to catalyze a community-wide dialogue that moves software engineering beyond controls designed for human-scale code production toward a disciplined, scalable, and trustworthy agentic future.

cs.SE↗

Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation

Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.

cs.AI↗

RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates

We introduce RedDebate, a novel multi-agent debate framework that provides the foundation for Large Language Models (LLMs) to identify and mitigate their unsafe behaviours. AI safety approaches often rely on costly human evaluation or isolated single-model assessment, both constrained by scalability and prone to oversight failures. RedDebate employs collaborative argumentation among multiple LLMs across diverse debate scenarios, enabling them to critically evaluate one another's reasoning and systematically uncover unsafe failure modes through fully automated red-teaming. To support this, we propose designing distinct long-term memory modules that preserve safety-relevant insights from debate interactions and leverage them during subsequent inference, facilitating continuous refinement of model behaviour. Empirical evaluation on safety benchmarks across a diverse set of models demonstrates that RedDebate substantially reduces unsafe outputs. While debate alone allows LLMs to refine their behaviour, the addition of memory yields further error reductions. To the best of our knowledge, RedDebate is the first fully automated framework to unify multi-agent debate and red-teaming to progressively enhance LLM safety without human intervention.

cs.CL↗

CoDial: Interpretable Task-Oriented Dialogue Systems Through Dialogue Flow Alignment

Building Task-Oriented Dialogue (TOD) systems that generalize across different tasks remains a challenging problem. Data-driven approaches often struggle to transfer effectively to unseen tasks. While recent schema-based TOD frameworks improve generalization by decoupling task logic from language understanding, their reliance on neural or generative models often obscures how task schemas influence behaviour and hence impair interpretability. In this work, we introduce a novel framework, CoDial (Code for Dialogue), at the core of which is converting a predefined task schema to a structured heterogeneous graph and then to programmatic LLM guardrailing code, such as NVIDIA's Colang. The pipeline enables efficient and interpretable alignment of dialogue policies during inference. We introduce two paradigms for LLM guardrailing code generation, $\text{CoDial}_{\text{free}}$ and $\text{CoDial}_{\text{structured}}$, and propose a mechanism that integrates human feedback to iteratively improve the generated code. Empirically, CoDial achieves state-of-the-art (SOTA) performance on the widely used benchmark datasets, while providing inherent interpretability in the design. We additionally demonstrate CoDial's iterative improvement via manual and LLM-aided feedback, making it a practical tool for human-guided alignment of LLMs in unseen domains.

cs.CL↗

Misinformation with Legal Consequences (MisLC): A New Task Towards Harnessing Societal Harm of Misinformation

Misinformation, defined as false or inaccurate information, can result in significant societal harm when it is spread with malicious or even innocuous intent. The rapid online information exchange necessitates advanced detection mechanisms to mitigate misinformation-induced harm. Existing research, however, has predominantly focused on assessing veracity, overlooking the legal implications and social consequences of misinformation. In this work, we take a novel angle to consolidate the definition of misinformation detection using legal issues as a measurement of societal ramifications, aiming to bring interdisciplinary efforts to tackle misinformation and its consequence. We introduce a new task: Misinformation with Legal Consequence (MisLC), which leverages definitions from a wide range of legal domains covering 4 broader legal topics and 11 fine-grained legal issues, including hate speech, election laws, and privacy regulations. For this task, we advocate a two-step dataset curation approach that utilizes crowd-sourced checkworthiness and expert evaluations of misinformation. We provide insights about the MisLC task through empirical evidence, from the problem definition to experiments and expert involvement. While the latest large language models and retrieval-augmented generation are effective baselines for the task, we find they are still far from replicating expert performance.

cs.CL↗