Jona te Lintelo | arXiv Science

arXiv2026

BadPatches: Routing-Aware Backdoor Attacks on Vision Mixture-of-Experts

Mixture-of-Experts (MoE) architectures have gained significant traction for reducing computational costs in deep neural networks by activating only a sparse subset of parameters during inference. While this efficiency makes MoE highly attractive for scaling vision tasks, its patch-based processing mechanism inherently disrupts traditional, routing-agnostic backdoor attacks by fragmenting or discarding adversarial triggers. To expose the vulnerabilities of this architecture, we introduce BadPatches, a novel routing-aware trigger application strategy specifically designed for patch-based MoE (pMoE) models and MoE-based vision transformers. Rather than applying a global pattern across the entire image, BadPatches encapsulates triggers within targeted image patches, ensuring they are consistently routed to and processed by the active experts. Our evaluations demonstrate that BadPatches achieves a high Attack Success Rate (ASR) at lower poisoning rates than routing-agnostic triggers, reaching over 83.2% ASR with a poisoning rate of only 0.01%, and scaling to a 96.8% ASR at 0.05%, while preserving the model's clean accuracy. Furthermore, the attack remains effective in gray-box scenarios where the adversary lacks complete knowledge of the model's patch routing configuration. Finally, we evaluate fine-pruning as a potential defense mechanism, revealing that pruning alone is insufficient to mitigate the attack; successful backdoor removal strictly requires the fine-tuning stage. These findings highlight the fragility of sparse vision architectures and underscore the need for routing-aware defenses.

cs.CR↗

arXiv2026

WoE Wrote It? Watermarking Mixture-of-Experts LLMs for Black-Box Text Provenance

Large Language Model (LLM) watermarks provide a mechanism for text provenance, enabling model owners to identify machine-generated content and attribute it to a specific watermarked model. However, current LLM watermarking approaches predominantly rely on inference-time sampler methods and focus their analysis on dense models. Inference-time methods are only effective when the text is explicitly generated via the model owner's controlled API; they fail in a post-compromise scenario. An adversary who steals or leaks the model weights gains complete control over inference and can simply run an unmodified sampler, bypassing the watermark and preventing post-theft attribution. In this work, we introduce Watermarking of Experts (WoE), a novel black-box text provenance method that leverages the unique structural properties of sparse Mixture-of-Experts (MoE) models. WoE biases the vocabulary of specific experts and shifts the watermark signal embedding away from unenforceable inference wrappers. This approach ensures the watermark remains intrinsic to the model parameters, enabling defenders to attribute text generated by stolen weights, leaked checkpoints, and secondary dense models distilled from the stolen architecture without needing access to the adversary's deployment or weights. We evaluate WoE across eight MoE models, demonstrating successful watermark detection from suspect text, achieving an average true positive rate of 90.1% at a 1% false positive rate, reaching up to 94.9%, while largely preserving general model utility. Furthermore, WoE remains detectable under adversarial supervised fine-tuning, model extraction, and output-level paraphrasing, forcing malicious actors into a trade-off in which weakening the attribution signal requires additional model adaptation or text-rewriting operations, or compromises the utility of the resulting output.

cs.CR↗