arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2609.32601

VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents

Yikun Li·Jinfeng Jiang·Yuheng Yieh·Ting Zhang·Yide Yin·Wen Bin Leow·Eng Lieh Ouh·Lwin Khin Shar·David Lo
View original source ↗Read source document ↗Export metadata ↓

Abstract

Vulnerability-detection benchmarks score the verdict an agent reaches, not the evidence it gathered. A model that recalls a CVE from pretraining therefore scores the same as one that traced the data flow. We study a task where this difference matters, deciding whether a commit introduces a vulnerability. Instead of scoring the verdict, we score whether the agent retrieved the code its conclusion depends on. We present VulContextBench, a benchmark of 111 vulnerability-introducing commits (VICs) across 83 repositories, 63 CWEs, and five languages. Existing datasets label such commits by tracing a fix back through the version history, which often points to the wrong commit. We therefore audit every case by hand against an explicit four-criterion definition of a VIC, so the benchmark does not inherit that label noise. Each case is annotated with gold context, 464 code blocks in total, each tagged by its role in the evidence for the vulnerability. We evaluate seven frontier models with precision, recall and F1 at three granularities (file, block, and line), scored separately on the context an agent viewed while exploring and on the context it finally declared as evidence. The gap between the two is the main finding. Every model opens most of the gold context while exploring, but reports only part of it as evidence. At the level of code blocks, the share of the gold context a model reports is 37 to 73 percentage points below the share it viewed. Qwen3-Coder-Next views 86.3% of the lines in annotated code blocks but cites only 12.9% in its final report. GPT-5.5, which cites the most, views 73% and reports 36%. These results highlight a gap between finding relevant code and selecting it for the final report, which verdict-level benchmarks cannot reveal.

Explore related subjects

cs.CR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yikun Li, Jinfeng Jiang, Yuheng Yieh, Ting Zhang, Yide Yin, Wen Bin Leow, Eng Lieh Ouh, Lwin Khin Shar, David Lo. 2026-09-26. VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents. https://arxiv.org/abs/2609.32601

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

SoK: Cryptocurrency Mixing and Anonymity - Architectures, Threat Models, Operational Aspects and Security

Public blockchains record transaction histories that enable address clustering, taint analysis, and cross-service attribution, thereby motivating the development of mixers and privacy layers. Our work presents a structured scoping review of 22 representative systems, defining a common unlinkability objective and five adversary archetypes. We evaluate these systems against a taxonomy of attack surfaces, including chain analysis, timing inference, custodial compromise, coordination abuse, network metadata, and trusted execution compromise. While nominal anonymity-set size and cryptographic strength characterize privacy in theory, effective anonymity in practice depends on transaction denominations, cover traffic, relayer behavior, and compliance-interface design. Distinguishing nominal from effective anonymity, we derive four core lessons: (1) Privacy is strongest when integrated into everyday transactions, since standalone mixing creates an easily profiled user subset; (2) Trust points, including operators, peer quorums, and hardware enclaves, must be explicit so users know who can break privacy; (3) Network metadata, including gas funding and timing, must be treated formally as protocol data in privacy evaluations; and (4) Compliance should use auditable cryptographic predicates for selective disclosure rather than broad operator discretion. Ultimately, our systematization clarifies the strengths, failures, and future requirements of blockchain privacy architectures.

cs.CR↗
arXiv2026

Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Models

Large Language Models (LLMs) have been extensively used across diverse domains, including virtual assistants, automated code generation, and scientific research. However, they remain vulnerable to jailbreak attacks, which manipulate the models into generating harmful responses despite safety alignment. Recent studies have shown that current safety-aligned LLMs undergo shallow safety alignment. In this work, we conduct an in-depth investigation into the underlying mechanism of this phenomenon and reveal that it manifests through learned ''safety trigger tokens'' that activate the model's safety patterns when paired with the specific input. Through both analysis and empirical verification, we further demonstrate the high similarity of the safety trigger tokens across different harmful inputs. Accordingly, we propose D-STT, a simple yet effective defense algorithm that identifies and explicitly decodes safety trigger tokens of the given safety-aligned LLM to activate the model's learned safety patterns. In this process, the safety trigger is constrained to a single token, which effectively preserves model usability by introducing minimum intervention in the decoding process. Extensive experiments across diverse jailbreak attacks and benign prompts demonstrate that D-STT significantly reduces output harmfulness while preserving model usability and incurring negligible response time overhead, outperforming ten baseline methods.

cs.CR↗
arXiv2026

A traffic analysis attack against Introduction Protocol and Onion Services

Tor onion services rely on long-lived introduction circuits to support anonymous rendezvous between clients and services. Although Tor incorporates defenses against traffic analysis, the introduction protocol retains deterministic routing structure that can be exploited by an adversary. We present a practical intersection attack against Tor introduction circuits that over repeated interactions can identify each hop from the introduction point toward the onion service while requiring observation at only one relay per stage. The attack repeatedly probes the target service and intersects sets of destination IP addresses observed within narrowly bounded INTRODUCE1-RENDEZVOUS2 intervals, without assuming global visibility or access to packet payloads. Our traffic-analysis technique identifies with certainty the next relay in the path to target at each stage, thereby revealing a gap in Tor's privacy model, which is intended to resist traffic-analysis attacks in which an adversary uses traffic patterns to determine which points in the network to observe or attack. We evaluate the attack's feasibility through live-network experiments using a self-operated onion service and relays. To support data minimization, we implement a Tor-compatible plugin that computes intersections online over pseudonymized data retained only in volatile memory. Our experiments show reliable convergence in practice, with convergence rate influenced by relay consensus weight and time-varying background traffic. We further assess practicality under a partial-global adversary model and discuss the implications of geographic concentration in Tor relay selection weight across cooperating jurisdictions.

cs.CR↗