arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2507.02976

How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench

Amirali Sajadi·Kostadin Damevski·Preetha Chatterjee
View original source ↗Read source document ↗Export metadata ↓

Abstract

Large language models (LLMs) and their agentic frameworks are increasingly adopted to perform development tasks such as automated program repair (APR). While prior work has identified security risks in LLM-generated code, most have focused on synthetic, simplified, or isolated tasks that lack the complexity of real-world program repair. In this study, we present the first large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues. We evaluate patches proposed by developers, a standalone LLM (Llama 3.3 Instruct-70B), and three top-performing agentic frameworks (OpenHands, AutoCodeRover, HoneyComb). Finally, we analyze a wide range of code, issue, and project-level factors to understand the conditions under which generating insecure patches is more likely. Our findings reveal that Llama introduces many new vulnerabilities, exhibiting unique patterns not found in developers' code. Agentic workflows also generate a number of vulnerabilities, particularly when given more autonomy. We find that vulnerabilities in LLM-generated patches are associated with distinctive code characteristics and are commonly observed in issues missing specific types of information. These results suggest that contextual factors play a critical role in the security of the generated patches and point toward the need for proactive risk assessment methods that account for both issue and code-level information.

Explore related subjects

cs.CRcs.LGcs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Amirali Sajadi, Kostadin Damevski, Preetha Chatterjee. 2025-12-29. How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench. https://arxiv.org/abs/2507.02976

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks

Large Language Models (LLMs) are increasingly deployed in interactive settings, where user intent commonly unfolds through multi-turn dialogue. Multi-turn jailbreaks exploit this pattern by advancing a harmful intent across turns, so that no single message exposes the full objective. However, existing work treats these attacks as a loose collection of prompt patterns and does not analyze how the adversary organizes and advances harmful intent across an interaction. We develop a four-part, intent-oriented taxonomy that organizes multi-turn jailbreaks by adversarial intent structure. Through controlled ablations, we find that effectiveness is driven by how deliberately intent is organized across turns rather than by context length or query count. We further show that the way intent is organized determines the level at which it becomes detectable, pushing the required detection surface outward from the turn level to the session level to the cross-session level. These findings indicate that turn-local safety mechanisms are structurally insufficient and that single-point evaluation overlooks how intent is organized, motivating evaluation protocols aligned to the level at which harmful intent becomes observable. The code is available at: https://github.com/SiyuanLi00/INTACT.

cs.CR↗
arXiv2026

Concept drift mitigation through community and spectral graph analysis for the detection of cyberattacks in network traffic

In network traffic, legitimate behaviours and attack techniques evolve jointly - the phenomenon known as 'concept drift' [1]. Every detector is thereby left obsolete between two updates, and always one step behind adversaries. In this work, we propose to move the point of intervention from the model, repaired after the drift, to the feature space, selected before learning. We therefore introduce t-robustness, a stability score defined for each feature independently of any detection model, comparable across an entire feature space. It combines the step-by-step distance between successive statistical states of a feature, and its cumulative divergence from its initial state, so that a slow monotonic drift cannot pass for stability. The candidates are drawn from abnormal network connectivity patterns left by scans, DoS and communications between endpoints, read through graph community metrics and spectral metrics. The evaluation is performed on the UGR16 dataset, across three learning scenarios and a control scenario, as well as without model update, and demonstrate that t-robust feature spaces sustain detection where the baselines collapse: retained expectancy at the last test interval reaches 0.6025, against 0.5230 for graph community features and 0.3831 for the base NetFlow features.

cs.CR↗
arXiv2026

EI-DDLGN: Efficient Encrypted Inference with Deep Differentiable Logic Gate Networks under TFHE

Privacy-preserving inference via Torus Fully Homomorphic Encryption (TFHE) provides strong protection for sensitive data in outsourced deep learning applications. However, most TFHE-compatible neural network frameworks remain based on arithmetic neural architectures, resulting in high inference latency due to programmable bootstrapping (PBS), accumulator growth, and circuit bit-width sensitivity. In this work, we investigate Deep Differentiable Logic Gate Networks (DDLGNs) as a Boolean-native alternative for encrypted inference under TFHE. Because DDLGNs learn Boolean computations directly and discretize into fixed logic gate networks, their inference procedure is naturally aligned with TFHE's Boolean execution model and avoids arithmetic accumulation in hidden layers. We present EI-DDLGN, the first in-depth study of TFHE-based DDLGN inference, and characterize how encrypted execution cost depends on model size, learned Boolean-function distribution, and propagated wire status. We also introduce Model-Fixed-Wire PBS Bypass (MFW-PBS Bypass), a semantics-preserving execution strategy that eliminates unnecessary PBS operations without modifying the learned network topology. Evaluations across 72 depth-width configurations on MNIST, FashionMNIST, and UCI Phishing show that DDLGNs constitute an efficient alternative to arithmetic TFHE inference, achieving substantially improved accuracy-latency trade-offs. Notably, on MNIST, EI-DDLGN-Small matches the accuracy of QAT-FCNN-4 while reducing encrypted inference latency by 13.4x. Our implementation is available at https://github.com/Carleton-SCI/EI-DDLGN

cs.CR↗