arXiv subjects

Jian Liang

Publications and source records attributed to Jian Liang.

At least 109 records · Page 6Linked to original sources

arXiv2025

SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion

Recent advances in diffusion models have led to significant progress in audio-driven lip synchronization. However, existing methods typically rely on constrained audio-visual alignment priors or multi-stage learning of intermediate representations to force lip motion synthesis. This leads to complex training pipelines and limited motion naturalness. In this paper, we present SayAnything, a conditional video diffusion framework that directly synthesizes lip movements from audio input while preserving speaker identity. Specifically, we propose three specialized modules including identity preservation module, audio guidance module, and editing control module. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation without requiring additional supervision signals or intermediate representations. Extensive experiments demonstrate that SayAnything generates highly realistic videos with improved lip-teeth coherence, enabling unseen characters to say anything, while effectively generalizing to animated characters.

cs.CV↗

arXiv2025

Protecting Model Adaptation from Trojans in the Unlabeled Data

Model adaptation tackles the distribution shift problem with a pre-trained model instead of raw data, which has become a popular paradigm due to its great privacy protection. Existing methods always assume adapting to a clean target domain, overlooking the security risks of unlabeled samples. This paper for the first time explores the potential trojan attacks on model adaptation launched by well-designed poisoning target data. Concretely, we provide two trigger patterns with two poisoning strategies for different prior knowledge owned by attackers. These attacks achieve a high success rate while maintaining the normal performance on clean samples in the test stage. To defend against such backdoor injection, we propose a plug-and-play method named DiffAdapt, which can be seamlessly integrated with existing adaptation algorithms. Experiments across commonly used benchmarks and adaptation methods demonstrate the effectiveness of DiffAdapt. We hope this work will shed light on the safety of transfer learning with unlabeled data.

cs.CR↗

arXiv2025

Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation

Capitalizing on the complementary advantages of generative and discriminative models has always been a compelling vision in machine learning, backed by a growing body of research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion score to facilitate the test-time adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowledge from a single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of our DUSA in adapting a wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted to dissect the pivotal elements in DUSA. Code is publicly available at https://github.com/BIT-DA/DUSA.

cs.CV↗

arXiv2024

Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation

Unsupervised domain adaptation aims to transfer knowledge from a related, label-rich source domain to an unlabeled target domain, thereby circumventing the high costs associated with manual annotation. Recently, there has been growing interest in source-free domain adaptation, a paradigm in which only a pre-trained model, rather than the labeled source data, is provided to the target domain. Given the potential risk of source data leakage via model inversion attacks, this paper introduces a novel setting called black-box domain adaptation, where the source model is accessible only through an API that provides the predicted label along with the corresponding confidence value for each query. We develop a two-step framework named $\textbf{Pro}$totypical $\textbf{D}$istillation and $\textbf{D}$ebiased tun$\textbf{ing}$ ($\textbf{ProDDing}$). In the first step, ProDDing leverages both the raw predictions from the source model and prototypes derived from the target domain as teachers to distill a customized target model. In the second step, ProDDing keeps fine-tuning the distilled model by penalizing logits that are biased toward certain classes. Empirical results across multiple benchmarks demonstrate that ProDDing outperforms existing black-box domain adaptation methods. Moreover, in the case of hard-label black-box domain adaptation, where only predicted labels are available, ProDDing achieves significant improvements over these methods. Code will be available at \url{https://github.com/tim-learn/ProDDing/}.

cs.LG↗

arXiv2024

Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks

Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks. However, selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial. Existing VLM selection methods focus on the class-name-only setting, relying on a supervised large-scale dataset and large language models, which may not be accessible or feasible during deployment. This paper introduces the problem of \textbf{unsupervised vision-language model selection}, where only unsupervised downstream datasets are available, with no additional information provided. To solve this problem, we propose a method termed Visual-tExtual Graph Alignment (VEGA), to select VLMs without any annotations by measuring the alignment of the VLM between the two modalities on the downstream task. VEGA is motivated by the pretraining paradigm of VLMs, which aligns features with the same semantics from the visual and textual modalities, thereby mapping both modalities into a shared representation space. Specifically, we first construct two graphs on the vision and textual features, respectively. VEGA is then defined as the overall similarity between the visual and textual graphs at both node and edge levels. Extensive experiments across three different benchmarks, covering a variety of application scenarios and downstream datasets, demonstrate that VEGA consistently provides reliable and accurate estimates of VLMs' performance on unlabeled downstream tasks.

cs.CV↗

arXiv2024

Sample Correlation for Fingerprinting Deep Face Recognition

Face recognition has witnessed remarkable advancements in recent years, thanks to the development of deep learning techniques.However, an off-the-shelf face recognition model as a commercial service could be stolen by model stealing attacks, posing great threats to the rights of the model owner.Model fingerprinting, as a model stealing detection method, aims to verify whether a suspect model is stolen from the victim model, gaining more and more attention nowadays.Previous methods always utilize transferable adversarial examples as the model fingerprint, but this method is known to be sensitive to adversarial defense and transfer learning techniques.To address this issue, we consider the pairwise relationship between samples instead and propose a novel yet simple model stealing detection method based on SAmple Correlation (SAC).Specifically, we present SAC-JC that selects JPEG compressed samples as model inputs and calculates the correlation matrix among their model outputs.Extensive results validate that SAC successfully defends against various model stealing attacks in deep face recognition, encompassing face verification and face emotion recognition, exhibiting the highest performance in terms of AUC, p-value and F1 score.Furthermore, we extend our evaluation of SAC-JC to object recognition datasets including Tiny-ImageNet and CIFAR10, which also demonstrates the superior performance of SAC-JC to previous methods.The code will be available at \url{https://github.com/guanjiyang/SAC_JC}.

cs.CV↗

arXiv2024

Towards Compatible Fine-tuning for Vision-Language Model Updates

So far, efficient fine-tuning has become a popular strategy for enhancing the capabilities of foundation models on downstream tasks by learning plug-and-play modules. However, existing methods overlook a crucial issue: if the underlying foundation model is updated, are these plug-and-play modules still effective? In this paper, we first conduct a detailed analysis of various fine-tuning methods on the CLIP in terms of their compatibility with model updates. The study reveals that many high-performing fine-tuning methods fail to be compatible with the upgraded models. To address this, we propose a novel approach, Class-conditioned Context Optimization (ContCoOp), which integrates learnable prompts with class embeddings using an attention layer before inputting them into the text encoder. Consequently, the prompts can dynamically adapt to the changes in embedding space (due to model updates), ensuring continued effectiveness. Extensive experiments over 15 datasets show that our ContCoOp achieves the highest compatibility over the baseline methods, and exhibits robust out-of-distribution generalization.

cs.CV↗

arXiv2024

LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application

Contemporary recommendation systems predominantly rely on ID embedding to capture latent associations among users and items. However, this approach overlooks the wealth of semantic information embedded within textual descriptions of items, leading to suboptimal performance and poor generalizations. Leveraging the capability of large language models to comprehend and reason about textual content presents a promising avenue for advancing recommendation systems. To achieve this, we propose an Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework that synergizes open-world knowledge with collaborative knowledge. We address computational complexity concerns by utilizing pretrained LLMs as item encoders and freezing LLM parameters to avoid catastrophic forgetting and preserve open-world knowledge. To bridge the gap between the open-world and collaborative domains, we design a twin-tower structure supervised by the recommendation task and tailored for practical industrial application. Through experiments on the real large-scale industrial dataset and online A/B tests, we demonstrate the efficacy of our approach in industry application. We also achieve state-of-the-art performance on six Amazon Review datasets to verify the superiority of our method.

cs.IR↗

arXiv2024

Exploring Vacant Classes in Label-Skewed Federated Learning

Label skews, characterized by disparities in local label distribution across clients, pose a significant challenge in federated learning. As minority classes suffer from worse accuracy due to overfitting on local imbalanced data, prior methods often incorporate class-balanced learning techniques during local training. Although these methods improve the mean accuracy across all classes, we observe that vacant classes-referring to categories absent from a client's data distribution-remain poorly recognized. Besides, there is still a gap in the accuracy of local models on minority classes compared to the global model. This paper introduces FedVLS, a novel approach to label-skewed federated learning that integrates both vacant-class distillation and logit suppression simultaneously. Specifically, vacant-class distillation leverages knowledge distillation during local training on each client to retain essential information related to vacant classes from the global model. Moreover, logit suppression directly penalizes network logits for non-label classes, effectively addressing misclassifications in minority classes that may be biased toward majority classes. Extensive experiments validate the efficacy of FedVLS, demonstrating superior performance compared to previous state-of-the-art (SOTA) methods across diverse datasets with varying degrees of label skews. Our code is available at https://github.com/krumpguo/FedVLS.

cs.LG↗

arXiv2024

A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts

Machine learning methods strive to acquire a robust model during the training process that can effectively generalize to test samples, even in the presence of distribution shifts. However, these methods often suffer from performance degradation due to unknown test distributions. Test-time adaptation (TTA), an emerging paradigm, has the potential to adapt a pre-trained model to unlabeled data during testing, before making predictions. Recent progress in this paradigm has highlighted the significant benefits of using unlabeled data to train self-adapted models prior to inference. In this survey, we categorize TTA into several distinct groups based on the form of test data, namely, test-time domain adaptation, test-time batch adaptation, and online test-time adaptation. For each category, we provide a comprehensive taxonomy of advanced algorithms and discuss various learning scenarios. Furthermore, we analyze relevant applications of TTA and discuss open challenges and promising areas for future research. For a comprehensive list of TTA methods, kindly refer to \url{https://github.com/tim-learn/awesome-test-time-adaptation}.

cs.LG↗

arXiv2024

Glass Transition in Monolayers of Rough Colloidal Ellipsoids

Structure-dynamics correlation is one of the major ongoing debates in the glass transition, although a number of structural features have been found connected to the dynamic heterogeneity in different glass-forming colloidal systems. Here using colloidal experiments combined with coarse-grained molecular dynamics simulations, we investigate the glass transition in monolayers of rough colloidal ellipsoids. Compared with smooth colloidal ellipsoids, the surface roughness of ellipsoids is found to significantly change the nature of glass transition. In particular, we find that the surface roughness induced by coating only a few small hemispheres on the ellipsoids can eliminate the existence of orientational glass and the two-step glass transition found in monolayers of smooth ellipsoids. This is due to the surface roughness-induced coupling between the translational and rotational degrees of freedom in colloidal ellipsoids, which also destroys the structure-dynamics correlation found in glass-forming suspensions of colloidal ellipsoids. Our results not only suggest a new way of using surface roughness to manipulate the glass transition in colloidal systems, but also highlight the importance of detailed particle shape on the glass transition and structure-dynamics correlation in suspensions of anisotropic colloids.

cond-mat.soft↗

arXiv2024

Mirror Target YOLO: An Improved YOLOv8 Method with Indirect Vision for Heritage Buildings Fire Detection

Fires can cause severe damage to heritage buildings, making timely fire detection essential. Traditional dense cabling and drilling can harm these structures, so reducing the number of cameras to minimize such impact is challenging. Additionally, avoiding false alarms due to noise sensitivity and preserving the expertise of managers in fire-prone areas is crucial. To address these needs, we propose a fire detection method based on indirect vision, called Mirror Target YOLO (MITA-YOLO). MITA-YOLO integrates indirect vision deployment and an enhanced detection module. It uses mirror angles to achieve indirect views, solving issues with limited visibility in irregular spaces and aligning each indirect view with the target monitoring area. The Target-Mask module is designed to automatically identify and isolate the indirect vision areas in each image, filtering out non-target areas. This enables the model to inherit managers' expertise in assessing fire-risk zones, improving focus and resistance to interference in fire detection.In our experiments, we created an 800-image fire dataset with indirect vision. Results show that MITA-YOLO significantly reduces camera requirements while achieving superior detection performance compared to other mainstream models.

cs.CV↗

arXiv2024

Light Cone Distribution Amplitude for the $Λ$ Baryon from Lattice QCD

We calculate the leading-twist light-cone distribution amplitudes of the light $Λ$ baryon using lattice methods within the framework of large momentum effective theory. Our numerical computations are conducted employing $N_f=2+1$ stout smeared clover fermions and a Symanzik gauge action on a lattice with spacing $a=0.077\;\rm{fm}$, and a pion mass of 303 MeV. To approach the large momentum regime, we simulate the equal-time correlations with the hadron momentum $P^z = \{2.52, 3.02, 3.52\}$ GeV. By investigating the potential analytic characteristics of the baryon quasi-distribution amplitude in coordinate space, we validate these findings through our lattice calculations. After renormalization and extrapolation, we present results for the three-dimensional distribution of momentum fractions for the two light quarks. Based on these findings the paper briefly discusses the phenomenological impact on weak decays of $Λ_b$, and outlines potential systematic uncertainties that can be improved in the future. This work lays the theoretical foundation for accessing baryon LCDAs from lattice QCD.

hep-lat↗

arXiv2024

Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning

Multimodal Large Language Model (MLLM) have demonstrated strong generalization capabilities across diverse distributions and tasks, largely due to extensive pre-training datasets. Fine-tuning MLLM has become a common practice to improve performance on specific downstream tasks. However, during fine-tuning, MLLM often faces the risk of forgetting knowledge acquired during pre-training, which can result in a decline in generalization abilities. To balance the trade-off between generalization and specialization, we propose measuring the parameter importance for both pre-trained and fine-tuning distributions, based on frozen pre-trained weight magnitude and accumulated fine-tuning gradient values. We further apply an importance-aware weight allocation strategy, selectively updating relatively important parameters for downstream tasks. We conduct empirical evaluations on both image captioning and visual question-answering tasks using various MLLM architectures. The comprehensive experimental analysis demonstrates the effectiveness of the proposed solution, highlighting the efficiency of the crucial modules in enhancing downstream specialization performance while mitigating generalization degradation in MLLM Fine-Tuning.

cs.CL↗

arXiv2024

Detecting the flavor content of the vacuum using the Dirac operator spectrum

We compute the overlap Dirac spectrum on three gauge ensembles generated using $2+1$-flavor domain wall fermions. The three ensembles have different lattice spacings and two of them have quark masses tuned to the physical point. The spectral density is determined up to $λ\sim$100 MeV with subpercentage statistical uncertainty. We find that the density is close to a constant below $λ\sim$ 20 MeV as predicted by chiral perturbative theory ($χ$PT), and then increases linearly due to the strange quark mass. By fitting to the next-to-leading order $χ$PT form and using the non-perturbative RI/MOM renormalization, the $\rm SU(2)$ (keeping the strange quark mass at the physical point) and $\rm SU(3)$ chiral condensates at $\overline{\textrm{MS}}$ 2 GeV are determined to be $Σ=(265.4(0.5)(4.2)\ \textrm{MeV})^3$ and $Σ_0=(234.3(0.5)(25.8)\ \textrm{MeV})^3$, respectively. The pion decay constants are also determined to be $F=84.1(1.9)(8.0)$ and $F_0=58.6(0.5)(10.0)$ MeV. The systematic errors are carefully estimated including the effects of fitting ranges and the uncertainty of low-energy constant $L_6$. We also show that one can resolve the sea flavor content of the sea quarks and constrain their masses with {$\sim10\%-20\%$} statistical uncertainties using the Dirac spectral density.

hep-lat↗

arXiv2024

Origin of hadron spin based on Lattice QCD study on the charmed hadrons

We perform the first Lattice calculation about the charmed hadron spin decomposition using overlap fermions on a 2+1 flavor RBC/UKQCD domain-wall gauge configurations at 0.083 fm with 300 MeV pion mass. It is found that the contributions of quark spin to the spin of 1S, 1P charmonia and also proton-like triple heavy quark state are comparable with the expectation of non-relativistic quark model. Such an observation provides evidence that the non-triviality of proton spin decomposition mainly arises from the relativistic effects of the light quark. Conversely, the substantial gluon angular momentum contribution in the spin $(1/2)^+$ state with triple heavy quarks at the charm quark mass, remains significant, highlighting the ongoing importance of the gluon in the realm of charmed baryon physics.

hep-lat↗

arXiv2024

An Enhanced Federated Prototype Learning Method under Domain Shift

Federated Learning (FL) allows collaborative machine learning training without sharing private data. Numerous studies have shown that one significant factor affecting the performance of federated learning models is the heterogeneity of data across different clients, especially when the data is sampled from various domains. A recent paper introduces variance-aware dual-level prototype clustering and uses a novel $α$-sparsity prototype loss, which increases intra-class similarity and reduces inter-class similarity. To ensure that the features converge within specific clusters, we introduce an improved algorithm, Federated Prototype Learning with Convergent Clusters, abbreviated as FedPLCC. To increase inter-class distances, we weight each prototype with the size of the cluster it represents. To reduce intra-class distances, considering that prototypes with larger distances might come from different domains, we select only a certain proportion of prototypes for the loss function calculation. Evaluations on the Digit-5, Office-10, and DomainNet datasets show that our method performs better than existing approaches.

cs.LG↗

arXiv2024

The radiative decay of scalar glueball from lattice QCD

We perform the first lattice QCD study on the radiative decay of the scalar glueball to the vector meson $ϕ$ in the quenched approximation. The calculations are carried out on three gauge ensembles with different lattice spacings, which enable us to do the continuum extrapolation. We first revisit the radiative $J/ψ$ decay into the scalar glueball $G$ and obtain the partial decay width $Γ(J/ψ\to γG)=0.578(86)~\text{keV}$ and the branching fraction $\text{Br}(J/ψ\to γG) = 6.2(9)\times 10^{-3}$. We then extend the similar calculation to the process $G\to γϕ$ and get the partial decay width $Γ(G \to γϕ)= 0.074(47)~\text{keV}$, which implies that the combined branching fraction of $J/ψ\toγG\to γγϕ$ is as small as $\mathcal{O}(10^{-9})$ such that this process is hardly detected by the BESIII experiment even with the large $J/ψ$ sample of $\mathcal{O}(10^{10})$. With the vector meson dominance model, the two-photon decay width of the scalar glueball is estimated to be $Γ(G\toγγ)=0.53(46)~\text{eV}$, which results in a large stickiness $S(G)\sim \mathcal{O}(10^4)$ of the scalar glueball by assuming the stickiness of $f_2(1270)$ to be one.

hep-lat↗