Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

Aman Priyanshu; Supriti Vijay; Kimia Majd; Xuhong He; Fraser Burch; Takahiro Matsumoto; Jianliang He; Baturay Saglam; Arthur Goldblatt; Zhuoran Yang; Amin Karbasi

arXiv · 2609.15939

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

Aman Priyanshu·Supriti Vijay·Kimia Majd·Xuhong He·Fraser Burch·Takahiro Matsumoto·Jianliang He·Baturay Saglam·Arthur Goldblatt·Zhuoran Yang·Amin Karbasi

View original source ↗Read source document ↗Export metadata ↓

Abstract

Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness. We introduce the Vulnerability Localization Benchmark (VLoc Bench), comprising 500 real world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. We evaluate 27 language models and four static-analysis tools under a common agent interface. Repository-scale vulnerability localization remains difficult: the strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model. We further find that stronger localization does not imply reliable behavior after remediation: systems that identify vulnerable files effectively can still report unsupported locations on patched repositories. These results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.

Explore related subjects

cs.CR cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi. 2026-09-14. Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale. https://arxiv.org/abs/2609.15939

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

MIRANDA: short signatures from a leakage-free full-domain-hash scheme

We present $\mathsf{Miranda}$, the first family of full-domain-hash signatures based on matrix codes. This signature scheme fulfils the paradigm of Gentry, Peikert and Vaikuntanathan ($\mathsf{GPV}$), which gives strong security guarantees. Our trapdoor is very simple and generic: if we propose it with matrix codes, it can actually be instantiated in many other ways since it only involves a subcode of a decodable code (or lattice) in a unique decoding regime of parameters. Though $\mathsf{Miranda}$ signing algorithm relies on a decoding task where there is exactly one solution, there are many possible signatures given a message to sign and we ensure that signatures are not leaking information on their underlying trapdoor by means of a very simple procedure involving the drawing of a small number of uniform bits. In particular $\mathsf{Miranda}$ does not use a rejection sampling procedure which makes its implementation a very simple task contrary to other $\mathsf{GPV}$-like signatures schemes such as $\mathsf{Falcon}$ or even $\mathsf{Wave}$. We instantiate $\mathsf{Miranda}$ with the famous family of Gabidulin codes represented as spaces of matrices and we study thoroughly its security (in the EUF-CMA security model). For~$128$ bits of classical security, the signature sizes are as low as~$90$ bytes and the public key sizes are in the order of~$2.6$ megabytes.

cs.CR↗

arXiv2026

SteganoBackdoor: Evading Data-Poisoning Defenses via Steganographic Backdoors

Transformer-based models are highly susceptible to backdoor attacks via supervised fine-tuning (SFT). To red-team existing data-poisoning defenses, prior work has increasingly focused on stylized triggers, synthetic artifacts, and token-level perturbations designed to evade detection. However, this trend has shifted threat models away from naturally occurring semantic triggers and realistic low-budget poisoning settings. Addressing this gap, we introduce SteganoBackdoor, an optimization-based framework that transforms semantic-trigger seeds through autoregressive token replacement, sequentially minimizing embedding overlap with the inference-time trigger while preserving a strong per-sample training-time payload. The resulting SteganoPoisons maintain linguistic fluency and encode the payload across ordinary tokens, such that no individual token carries a concentrated signal and the full payload instead emerges from their exact combination and ordering. Across 18 encoder-based and decoder-only models spanning 120M to 14B parameters, SteganoBackdoor achieves high attack success under sub-percent poisoning budgets and exposes limitations in existing data-poisoning defenses.

cs.CR↗

arXiv2026

Foundations and Design Principles of Lightweight Cryptography for IoT Systems

The successful deployment of the Internet of Things (IoT) applications relies heavily on their robust security, and lightweight cryptography is considered an emerging solution in this context. While existing surveys have been examining lightweight cryptographic techniques from the perspective of hardware and software implementations or performance evaluation, there is a significant gap in addressing different security aspects, such as design principles, specific to the IoT environment. This study aims to bridge this gap. This research presents an examination with focusing on the security evaluation of symmetric lightweight ciphers commonly used in IoT systems. The objective of this study is to provide a concise overview of lightweight ciphers with emphasizing on their security challenges which is an essential consideration for real-time and resource-constrained applications.

cs.CR↗