arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2505.06827

Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking

Fabrice Y Harel-Canada·Boran Erol·Connor Choi·Jason Liu·Gary Jiarui Song·Nanyun Peng·Amit Sahai
View original source ↗Read source document ↗Export metadata ↓

Abstract

Watermarking AI-generated text is critical for combating misuse. Yet recent theoretical work argues that any watermark can be erased via random walk attacks that perturb text while preserving quality. However, such attacks rely on two key assumptions: (1) rapid mixing (watermarks dissolve quickly under perturbations) and (2) reliable quality preservation (automated quality oracles perfectly guide edits). Through large-scale experiments and human-validated assessments, we find mixing is slow: 100% of perturbed texts retain traces of their origin after hundreds of edits, defying rapid mixing. Oracles falter, as state-of-the-art quality detectors misjudge edits (77% accuracy), compounding errors during attacks. Ultimately, attacks underperform: automated walks remove watermarks just 26% of the time -- dropping to 10% under human quality review. These findings challenge the inevitability of watermark removal. Instead, practical barriers -- slow mixing and imperfect quality control -- reveal watermarking to be far more robust than theoretical models suggest. The gap between idealized attacks and real-world feasibility underscores the need for stronger watermarking methods and more realistic attack models.

Explore related subjects

cs.CRcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Fabrice Y Harel-Canada, Boran Erol, Connor Choi, Jason Liu, Gary Jiarui Song, Nanyun Peng, Amit Sahai. 2025-05-11. Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking. https://arxiv.org/abs/2505.06827

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Effective and Efficient Threat Hunting with Small Language Models

Analysts in Security Operations Centers query massive telemetry streams using Kusto Query Language (KQL), but writing correct KQL demands specialized expertise that bottlenecks scaling security teams. We investigate how Small Language Models (SLMs) can enable accurate, cost-effective translation from natural language queries (NLQs) to KQL. We propose a three-knob framework spanning prompting, fine-tuning, and architecture. First, we adapt NL2KQL for SLMs with lightweight retrieval and introduce error-aware prompting that targets common parser failures with a handful of mined tips, at a fraction of the tokens KQL's full rule set would require. Second, we apply LoRA fine-tuning with rationale distillation augmenting each NLQ-KQL pair with a brief chain-of-thought to transfer teacher reasoning. This yields an informative negative result, as neither variant surpasses targeted prompting. Third, we propose a two-stage architecture pairing an SLM drafter with a low-cost LLM judge for schema-aware refinement. We evaluate nine models (five SLMs, four LLMs) on syntax correctness, semantic accuracy, table selection, filter precision, latency, and token cost. On Microsoft's NL2KQL Defender Evaluation dataset, our two-stage approach reaches 0.987 syntax and 0.906 schema-valid ("semantic") accuracy, exceeding every baseline we run under equivalent infrastructure, and it generalizes to independently authored queries over the same schema (0.964 syntax, 0.831 schema-valid). The only baselines within 0.05 schema-valid are NL2KQL+GPT-4o (0.878) and NL2KQL+GPT-5 (0.861), which cost USD 2.998 and USD 2.018 for 230 queries against USD 0.213 for ours, a 9.5-14x reduction at matched accuracy. These results establish SLMs as a practical foundation for natural-language querying in security operations.

cs.CR↗
arXiv2026

KUDA: Knowledge Unlearning by Deviating Representation for Large Language Models

Large language models (LLMs) acquire a large amount of knowledge through pre-training on vast and diverse corpora. While this endows LLMs with strong capabilities in generation and reasoning, it amplifies risks associated with sensitive, copyrighted, or harmful content in training data. LLM unlearning, which aims to remove specific knowledge encoded within models, is a promising technique to reduce these risks. However, existing LLM unlearning methods often force LLMs to generate random or incoherent answers due to their inability to alter the encoded knowledge precisely. To achieve effective unlearning at the knowledge level of LLMs, we propose Knowledge Unlearning by Deviating representAtion (KUDA). We first utilize causal tracing to locate specific layers for target knowledge storage. We then design a new unlearning objective that induces the model's representations to deviate from its original position in the phase of knowledge removal, thus disrupting the ability to associate with the target knowledge. To resolve the optimization conflicts between forgetting and retention, we employ a relaxation null-space projection mechanism to mitigate the disruption to the representation space of retaining knowledge. Extensive experiments on representative benchmarks, WMDP and MUSE, demonstrate that KUDA outperforms most existing baselines by effectively balancing knowledge removal and model utility retention. Source code is available at https://github.com/AceNagi/KUDA.

cs.CR↗
arXiv2026

Self-State Attacks on Self-Hosted AI Agents: How Far Can OS Defenses Go?

Self-hosted AI agents maintain persistent memory, instructions, and configuration that influence their future behavior. If an agent is compromised, an attacker can exploit the agent's legitimate write permissions to corrupt this self-state, making malicious and benign updates difficult to distinguish at the operating system (OS) level. We investigate how far existing OS mechanisms can prevent, detect, and recover from such self-state attacks. We formalize an attack space and evaluate representative OS defenses using four agent workloads and a Linux telemetry pipeline. Our results show a consistent limitation across defense dimensions. File-level controls either leave alternative mutation paths open or, when complete over the tested operations, also block corresponding legitimate updates. Detectors flag a substantial part of legitimate activity, while more selective methods cover only part of the attack space. Finally, protected backups successfully restore corrupted state, but require a trusted recovery point and may incur rollback cost. Overall, our results show that the main limitation is not OS observability. Indeed, the OS can enforce, observe, attribute, and recover self-state changes. Yet, generic OS defenses lack the decision context needed to combine broad operation coverage with selective decisions. Effective protection therefore requires self-state-aware mechanisms that exploit additional context beyond generic file and syscall behavior.

cs.CR↗