arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2604.13764

RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code

John Pellew·Faizan Raza
View original source ↗Read source document ↗Export metadata ↓

Abstract

How do security scanners perform on real-world code? We present RealVuln, the first open-source benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on 26 intentionally vulnerable Python repositories (educational and Capture-The-Flag applications) with 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps). We test 15 scanners (3 Rule-Based SAST, 10 General-Purpose LLM, 2 Security-Specialized) and rank them by F3 score (beta=3, weighting recall 9x over precision). A clear three-tier ranking emerges under all metrics. Under F3, the Security-Specialized scanner Kolega.Dev (73.0) leads, followed by the best General-Purpose LLM, Claude Sonnet 4.6 (51.7), which in turn scores nearly 3x higher than the best Rule-Based tool, Semgrep (17.7). Under F1, Sonnet 4.6 leads (60.9) with Kolega.Dev at 52.4. Rankings within tiers shift with beta, but the three-tier hierarchy holds across all weightings. All code, ground-truth data, scanner outputs, and scoring scripts are released under an open-source license. An interactive dashboard is at https://realvuln.kolega.dev/. RealVuln is a living benchmark: versioned, community-driven, with a roadmap toward multi-language coverage.

Explore related subjects

cs.CR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

John Pellew, Faizan Raza. 2026-04-15. RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code. https://arxiv.org/abs/2604.13764

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking

Speaker verification models are trained on large-scale public datasets whose licenses usually prohibit unauthorized commercial use, yet such infringement is difficult to detect or deter. Dataset ownership verification (DOV) is the mainstream countermeasure: it can watermark a dataset with backdoor attacks so that models trained on it exhibit owner-specified behaviors. However, existing DOV methods presuppose a closed label space fixed at watermarking time, whereas in open-set speaker verification the identities that a deployed model accepts are enrolled by third parties after release and are never observed by the dataset owner. We show that straightforward adaptations fail in two characteristic modes, and accordingly distill three requirements for an effective watermark, namely identity agnosticism, coverage, and fidelity, together with an intrinsic tension between the latter two. Our clustering-based backdoor watermark (CBW) resolves this tension by partitioning training speakers into clusters by feature similarity and implanting a distinct trigger for each cluster, so that each trigger covers one region of the speaker embedding space while the trigger set is designed to jointly cover it. We further develop paired hypothesis tests for ownership verification under both the similarity-available and the decision-only black-box settings at the 1-to-1 and 1-to-$N$ enrollment scales, and theoretically characterize when the audit succeeds, including an exact small-sample certificate and the effect of the enrollment size. Extensive experiments on benchmark datasets and representative models verify the effectiveness of our CBW, its resistance to watermark-removal attacks, and its transferability across model structures. Code is at https://github.com/Radiant0726/CBW/tree/master.

cs.CR↗
arXiv2026

Sockeye: Bug-finding and proofs for platform configurations and hardware based on reference manuals

The ever increasing complexity of hardware platforms poses a challenge to systems programmers. Correctly programming a multitude of components, providing functionality and security, is difficult: semantics of individual units are described in prose, underspecified, and prone to inaccuracies. Rigorous statements about platform security are often impossible. We are the first to address this problem for closed-source hardware platforms, by introducing a domain-specific language to formally describe hardware semantics based on hardware reference manuals, assumptions about software behavior, and desired security properties. We demonstrate the practicality of our approach by creating machine-readable specifications for a diverse set of eight platforms from their reference manuals, and formally proving their (in-)security. In addition to security proofs about memory confidentiality and integrity, we discover a handful of documentation errors. Finally, our analysis also revealed a vulnerability on a real-world server chip, which was confirmed by the vendor to apply to a wide family of deployed network appliances. Our tooling offers system integrators a way of formally describing security properties for whole platforms, and the means to find counterexamples, or proving them correct.

cs.CR↗
arXiv2026

Do LLMs Make Neural Distinguishers Wise?

Neural distinguishers are a cryptanalysis method for symmetric-key cryptography that trains machine learning models on pairs of plaintexts and ciphertexts with specific differences in order to recover a secret key. To the best of our knowledge, no existing work has explored the use of large language models (LLMs) for neural distinguishers. In this paper, we propose LLM-based neural distinguishers through a prompt design and conduct extensive experiments with them on SPECK-32/64 to investigate whether LLMs can strengthen neural distinguishers. We then found three key insights. First, by comparing the results of LLM-based neural distinguishers with ResNet in the existing work, we demonstrate that LLMs provide no observable improvement in the performance of neural distinguishers. Second, we confirm that, at high rounds, the choice of differences is no longer effective for LLM-based neural distinguishers as well as ResNet. Third, we show that the performance of LLM-based neural distinguishers can be significantly improved by incorporating only the XOR operation results as a prompt design.

cs.CR↗