arXiv subjects

Yicheng Cai

Publications and source records attributed to Yicheng Cai.

3 recordsLinked to original sources

arXiv2026

Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation

Evaluations of automated program repair (APR) systems commonly report the benchmark, the number of repaired defects, and the tests used for final patch validation, but these items no longer fully specify the repair task presented to a system. Recent LLM-based systems differ in the information supplied before repair, the repository and testing operations permitted during repair, and the feedback returned after unsuccessful attempts, allowing the same benchmark to instantiate substantially different repair tasks ranging from localized patch generation to repository-level diagnosis and iterative repair. We present a framework for explicitly specifying the experimental settings associated with reported APR results. We analyze reported experimental settings from systems evaluated on Defects4J and SWE-bench and characterize each result by its task unit, fault-localization assumptions, initial input, tool access, repair-time feedback, final validation, and resource budget. Our analysis shows that benchmark identity alone is insufficient to reconstruct the evaluated task or determine the appropriate scope of comparison across reported repair rates. We therefore introduce a machine-readable schema for specifying each experimental setting to improve reproducibility and make the scope of cross-system comparisons explicit.

cs.SE↗

arXiv2026

Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to achieve more autonomous SOCs (security operation centers) and reduce manual effort. In particular, the AI and cybersecurity communities have recently developed several benchmarks for evaluating the red team capabilities of multi-agent AI systems. However, because the operations in SOCs are dominated by blue team operations, the capabilities of AI systems & agents to achieve more autonomous SOCs cannot be evaluated without a benchmark focused on blue team operations. To our best knowledge, no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature. Existing blue team benchmarks focus on a particular task. The goal of this work is to develop a set of design principles for the construction of a benchmark, which is denoted as SOC-bench, to evaluate the blue team capabilities of AI. Following these design principles, we have developed a conceptual design of SOC-bench, which consists of a family of five blue team tasks in the context of large-scale ransomware attack incident response.

cs.CR↗

arXiv2026

2DESR: a two-dimensional Fourier-space gyrokinetic eigenvalue code for the ion-temperature-gradient modes in tokamaks

A two-dimensional (2D) gyrokinetic eigenvalue solver, 2DESR, has been developed to solve the 2D gyrokinetic eigenvalue problem in the poloidal Fourier space for the ion-temperature-gradient (ITG) modes in tokamaks. With full kinetic effects of ions retained, the 2D gyrokinetic eigenvalue equations in the poloidal Fourier space have been derived and numerically solved in the 2DESR code. In the linear ITG Cyclone test with adiabatic electrons, the 2DESR code benchmarks well against the gyrokinetic initial-value codes GENE and NLT. It is found that two branches of ITG modes coexist in the system.

physics.plasm-ph↗