arXiv2026
Concept erasure aims to suppress undesirable concepts in text-to-image diffusion models while preserving their generative capabilities. Existing methods often redirect target concepts toward predefined semantic anchors, but the influence of anchor selection on the scope of concept erasure remains insufficiently understood. Through causal tracing, we reveal that concepts with more dispersed causal effects tend to exhibit higher anchor sensitivity and lower erasure robustness. Controlled anchor interventions demonstrate that anchor selection affects target suppression and collateral damage to surrounding semantic concepts. These findings motivate us to formulate anchor selection as semantic erasure boundary construction. We propose SELECT (Selective Erasure via Local Exclusive Counterfactual Targets), a plug-and-play framework based on Sibling Exclusive Concepts (SECs), which are semantically close to the target but mutually exclusive in its defining attributes. SELECT jointly constructs a counterfactual redirection endpoint and a local retention set to balance target suppression and semantic preservation. SELECT filters candidates using contextual activation, and uses semantic coherence to identify context compatible redirection endpoints, while the remaining SECs serve as local preservation constraints. Experiments on object, celebrity, artist-style, and NSFW erasure show that SELECT strengthens target suppression over fixed general anchors and achieves suppression-preservation trade-offs, while reducing concept re-emergence and erosion.