arXiv subjects

Shudong Zhang

Publications and source records attributed to Shudong Zhang.

9 recordsLinked to original sources

arXiv2026

Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

Large Vision-Language Models (LVLMs) have achieved impressive performance across multimodal understanding and reasoning tasks, yet their internal safety mechanisms remain opaque and poorly controlled. In this work, we present a comprehensive framework for diagnosing and repairing unsafe channels within LVLMs (CARE). We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors. Based on these findings, we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities through generalized eigen-decomposition between benign and malicious activations. During inference, activations are dynamically projected toward these safety subspaces via a hybrid fusion mechanism that adaptively balances visual and textual corrections, effectively suppressing unsafe features while preserving semantic fidelity. Extensive experiments on multiple safety benchmarks demonstrate that our causal-subspace repair framework significantly enhances safety robustness without degrading general multimodal capabilities, outperforming prior activation steering and alignment-based baselines. Additionally, our method exhibits good transferability, defending against unseen attacks.

cs.CV↗

arXiv2025

Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion

We present Adversarial Object Fusion (AdvOF), a novel attack framework targeting vision-and-language navigation (VLN) agents in service-oriented environments by generating adversarial 3D objects. While foundational models like Large Language Models (LLMs) and Vision Language Models (VLMs) have enhanced service-oriented navigation systems through improved perception and decision-making, their integration introduces vulnerabilities in mission-critical service workflows. Existing adversarial attacks fail to address service computing contexts, where reliability and quality-of-service (QoS) are paramount. We utilize AdvOF to investigate and explore the impact of adversarial environments on the VLM-based perception module of VLN agents. In particular, AdvOF first precisely aggregates and aligns the victim object positions in both 2D and 3D space, defining and rendering adversarial objects. Then, we collaboratively optimize the adversarial object with regularization between the adversarial and victim object across physical properties and VLM perceptions. Through assigning importance weights to varying views, the optimization is processed stably and multi-viewedly by iterative fusions from local updates and justifications. Our extensive evaluations demonstrate AdvOF can effectively degrade agent performance under adversarial conditions while maintaining minimal interference with normal navigation tasks. This work advances the understanding of service security in VLM-powered navigation systems, providing computational foundations for robust service composition in physical-world deployments.

cs.CR↗

arXiv2025

Attention! Your Vision Language Model Could Be Maliciously Manipulated

Large Vision-Language Models (VLMs) have achieved remarkable success in understanding complex real-world scenarios and supporting data-driven decision-making processes. However, VLMs exhibit significant vulnerability against adversarial examples, either text or image, which can lead to various adversarial outcomes, e.g., jailbreaking, hijacking, and hallucination, etc. In this work, we empirically and theoretically demonstrate that VLMs are particularly susceptible to image-based adversarial examples, where imperceptible perturbations can precisely manipulate each output token. To this end, we propose a novel attack called Vision-language model Manipulation Attack (VMA), which integrates first-order and second-order momentum optimization techniques with a differentiable transformation mechanism to effectively optimize the adversarial perturbation. Notably, VMA can be a double-edged sword: it can be leveraged to implement various attacks, such as jailbreaking, hijacking, privacy breaches, Denial-of-Service, and the generation of sponge examples, etc, while simultaneously enabling the injection of watermarks for copyright protection. Extensive empirical evaluations substantiate the efficacy and generalizability of VMA across diverse scenarios and datasets. Code is available at https://github.com/Trustworthy-AI-Group/VMA.

cs.CV↗

arXiv2024

GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization

Glitch tokens, inputs that trigger unpredictable or anomalous behavior in Large Language Models (LLMs), pose significant challenges to model reliability and safety. Existing detection methods primarily rely on heuristic embedding patterns or statistical anomalies within internal representations, limiting their generalizability across different model architectures and potentially missing anomalies that deviate from observed patterns. We introduce GlitchMiner, an behavior-driven framework designed to identify glitch tokens by maximizing predictive entropy. Leveraging a gradient-guided local search strategy, GlitchMiner efficiently explores the discrete token space without relying on model-specific heuristics or large-batch sampling. Extensive experiments across ten LLMs from five major model families demonstrate that GlitchMiner consistently outperforms existing approaches in detection accuracy and query efficiency, providing a generalizable and scalable solution for effective glitch token discovery. Code is available at [https://github.com/wooozihu/GlitchMiner]

cs.AI↗

arXiv2024

Anisotropic Thermal Conductivity of 3D Printed Graphene Enhanced Thermoplastic Polyurethanes Structure toward Photothermal Conversion

Solar photothermal conversion is one of the most straightforward methods to utilize solar energy. In this manuscript, a novel double-layer structure constructed of graphene enhanced thermoplastic polyurethanes (G-TPU) and neat thermoplastic polyurethanes (N-TPU) was developed via fused deposition modelling (FDM) 3D printing process. The developed G-TPU-N-TPU double-layer structure exhibited anisotropic thermal conductivity that simultaneously satisfied high in-plane (IP) thermal conductivity and low through-plane (TP) thermal conductivity. The top G-TPU layer essentially offered a high IP thermal conductivity of 4.54 W(mK) that lead to overall structure anisotropic thermal conductivity ratio (TCIP-TCTP) of 8. And the low thermal conductivity in the TP direction led to the heat retention effects for thermal storage. Nonetheless, the exceptional photothermal conversion effect of graphene flakes guaranteed the superior photothermal performance that was promising in the photothermal de-icing and infrared labels applications. Finally, the graphene flake enhancement in the mechanical properties of the G-TPU-N-TPU double layer structure was also evaluated that contributed to excellent impact resistance with a puncture energy reaching 12.86 J, and extraordinary wear resistance with a small friction coefficient of 0.1 over 1000 cycles, which ensured the structure suitable for applications at harsh environment.

cond-mat.mtrl-sci↗

arXiv2024

ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

Large-scale pre-trained generative models are taking the world by storm, due to their abilities in generating creative content. Meanwhile, safeguards for these generative models are developed, to protect users' rights and safety, most of which are designed for large language models. Existing methods primarily focus on jailbreak and adversarial attacks, which mainly evaluate the model's safety under malicious prompts. Recent work found that manually crafted safe prompts can unintentionally trigger unsafe generations. To further systematically evaluate the safety risks of text-to-image models, we propose a novel Automatic Red-Teaming framework, ART. Our method leverages both vision language model and large language model to establish a connection between unsafe generations and their prompts, thereby more efficiently identifying the model's vulnerabilities. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. The experiments also validate the effectiveness, adaptability, and great diversity of ART. Additionally, we introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models. Datasets and models can be found in https://github.com/GuanlinLee/ART.

cs.CR↗

arXiv2022

Lower Difficulty and Better Robustness: A Bregman Divergence Perspective for Adversarial Training

In this paper, we investigate on improving the adversarial robustness obtained in adversarial training (AT) via reducing the difficulty of optimization. To better study this problem, we build a novel Bregman divergence perspective for AT, in which AT can be viewed as the sliding process of the training data points on the negative entropy curve. Based on this perspective, we analyze the learning objectives of two typical AT methods, i.e., PGD-AT and TRADES, and we find that the optimization process of TRADES is easier than PGD-AT for that TRADES separates PGD-AT. In addition, we discuss the function of entropy in TRADES, and we find that models with high entropy can be better robustness learners. Inspired by the above findings, we propose two methods, i.e., FAIT and MER, which can both not only reduce the difficulty of optimization under the 10-step PGD adversaries, but also provide better robustness. Our work suggests that reducing the difficulty of optimization under the 10-step PGD adversaries is a promising approach for enhancing the adversarial robustness in AT.

cs.LG↗

arXiv2022

Alleviating Robust Overfitting of Adversarial Training With Consistency Regularization

Adversarial training (AT) has proven to be one of the most effective ways to defend Deep Neural Networks (DNNs) against adversarial attacks. However, the phenomenon of robust overfitting, i.e., the robustness will drop sharply at a certain stage, always exists during AT. It is of great importance to decrease this robust generalization gap in order to obtain a robust model. In this paper, we present an in-depth study towards the robust overfitting from a new angle. We observe that consistency regularization, a popular technique in semi-supervised learning, has a similar goal as AT and can be used to alleviate robust overfitting. We empirically validate this observation, and find a majority of prior solutions have implicit connections to consistency regularization. Motivated by this, we introduce a new AT solution, which integrates the consistency regularization and Mean Teacher (MT) strategy into AT. Specifically, we introduce a teacher model, coming from the average weights of the student models over the training steps. Then we design a consistency loss function to make the prediction distribution of the student models over adversarial examples consistent with that of the teacher model over clean samples. Experiments show that our proposed method can effectively alleviate robust overfitting and improve the robustness of DNN models against common adversarial attacks.

cs.LG↗

arXiv2022

Investigation of plasmonic enhanced solar photothermal effect of Au NR@PVDF micro/nano-film

Gold nanospheres (Au NSs) and gold nanorods (Au NRs) are traditional noble metal plasmonic nanomaterials. Particularly, Au NRs with tunable longitudinal plasmon resonance from visible to the near infrared (NIR) range were suitable for high efficient photothermal applications due to extended light receiving range. In this work, we synthesized Au NRs and Au NSs of similar volume, and subsequently developed them into Au NR/PVDF and Au NS/PVDF nanofilm, both of which exhibited excellent solar photothermal performance evaluated by solar photothermal experiments. We found that Au NR/PVDF nanofilm showed higher solar photothermal performance than Au NS/PVDF nanofilm. Through detailed analysis, such as morphological characterization, optical measurement, and finite element method (FEM) modeling, we found that the plasmonic coupling effects inside the aggregated Au NRs nanoclusters contributed to the spectral blue-shifts and intensified photothermal performance. As compared to Au NS/PVDF nanofilms, Au NR/PVDF nanofilm exhibited higher efficient light-to-heat conversion rate, because of the extended light receiving range and high absorbance, as the result of strong plasmonic interactions inside nanoclusters, which was further validated by monochromatic laser photothermal experiments and FEM simulations. Our work proved that the Au NRs have huge potential for plasmonic solar photothermal applications, and are envisioned for novel plasmonic applications.

physics.optics↗