arXiv subjects

Han Zhang

Publications and source records attributed to Han Zhang.

At least 217 records · Page 12Linked to original sources

arXiv2025

Protein Large Language Models: A Comprehensive Survey

Protein-specific large language models (Protein LLMs) are revolutionizing protein science by enabling more efficient protein structure prediction, function annotation, and design. While existing surveys focus on specific aspects or applications, this work provides the first comprehensive overview of Protein LLMs, covering their architectures, training datasets, evaluation metrics, and diverse applications. Through a systematic analysis of over 100 articles, we propose a structured taxonomy of state-of-the-art Protein LLMs, analyze how they leverage large-scale protein sequence data for improved accuracy, and explore their potential in advancing protein engineering and biomedical research. Additionally, we discuss key challenges and future directions, positioning Protein LLMs as essential tools for scientific discovery in protein science. Resources are maintained at https://github.com/Yijia-Xiao/Protein-LLM-Survey.

q-bio.BM↗

arXiv2025

MITracker: Multi-View Integration for Visual Object Tracking

Multi-view object tracking (MVOT) offers promising solutions to challenges such as occlusion and target loss, which are common in traditional single-view tracking. However, progress has been limited by the lack of comprehensive multi-view datasets and effective cross-view integration methods. To overcome these limitations, we compiled a Multi-View object Tracking (MVTrack) dataset of 234K high-quality annotated frames featuring 27 distinct objects across various scenes. In conjunction with this dataset, we introduce a novel MVOT method, Multi-View Integration Tracker (MITracker), to efficiently integrate multi-view object features and provide stable tracking outcomes. MITracker can track any object in video frames of arbitrary length from arbitrary viewpoints. The key advancements of our method over traditional single-view approaches come from two aspects: (1) MITracker transforms 2D image features into a 3D feature volume and compresses it into a bird's eye view (BEV) plane, facilitating inter-view information fusion; (2) we propose an attention mechanism that leverages geometric information from fused 3D feature volume to refine the tracking results at each view. MITracker outperforms existing methods on the MVTrack and GMTD datasets, achieving state-of-the-art performance. The code and the new dataset will be available at https://mii-laboratory.github.io/MITracker/.

cs.CV↗

arXiv2025

DiffKAN-Inpainting: KAN-based Diffusion model for brain tumor inpainting

Brain tumors delay the standard preprocessing workflow for further examination. Brain inpainting offers a viable, although difficult, solution for tumor tissue processing, which is necessary to improve the precision of the diagnosis and treatment. Most conventional U-Net-based generative models, however, often face challenges in capturing the complex, nonlinear latent representations inherent in brain imaging. In order to accomplish high-quality healthy brain tissue reconstruction, this work proposes DiffKAN-Inpainting, an innovative method that blends diffusion models with the Kolmogorov-Arnold Networks architecture. During the denoising process, we introduce the RePaint method and tumor information to generate images with a higher fidelity and smoother margin. Both qualitative and quantitative results demonstrate that as compared to the state-of-the-art methods, our proposed DiffKAN-Inpainting inpaints more detailed and realistic reconstructions on the BraTS dataset. The knowledge gained from ablation study provide insights for future research to balance performance with computing cost.

eess.IV↗

arXiv2025

ArrayBot: Reinforcement Learning for Generalizable Distributed Manipulation through Touch

We present ArrayBot, a distributed manipulation system consisting of a $16 \times 16$ array of vertically sliding pillars integrated with tactile sensors, which can simultaneously support, perceive, and manipulate the tabletop objects. Towards generalizable distributed manipulation, we leverage reinforcement learning (RL) algorithms for the automatic discovery of control policies. In the face of the massively redundant actions, we propose to reshape the action space by considering the spatially local action patch and the low-frequency actions in the frequency domain. With this reshaped action space, we train RL agents that can relocate diverse objects through tactile observations only. Surprisingly, we find that the discovered policy can not only generalize to unseen object shapes in the simulator but also transfer to the physical robot without any domain randomization. Leveraging the deployed policy, we present abundant real-world manipulation tasks, illustrating the vast potential of RL on ArrayBot for distributed manipulation.

cs.RO↗

arXiv2025

DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove

Dexterous hand teleoperation plays a pivotal role in enabling robots to achieve human-level manipulation dexterity. However, current teleoperation systems often rely on expensive equipment and lack multi-modal sensory feedback, restricting human operators' ability to perceive object properties and perform complex manipulation tasks. To address these limitations, we present DOGlove, a low-cost, precise, and haptic force feedback glove system for teleoperation and manipulation. DoGlove can be assembled in hours at a cost under 600 USD. It features a customized joint structure for 21-DoF motion capture, a compact cable-driven torque transmission mechanism for 5-DoF multidirectional force feedback, and a linear resonate actuator for 5-DoF fingertip haptic feedback. Leveraging action and haptic force retargeting, DOGlove enables precise and immersive teleoperation of dexterous robotic hands, achieving high success rates in complex, contact-rich tasks. We further evaluate DOGlove in scenarios without visual feedback, demonstrating the critical role of haptic force feedback in task performance. In addition, we utilize the collected demonstrations to train imitation learning policies, highlighting the potential and effectiveness of DOGlove. DOGlove's hardware and software system will be fully open-sourced at https://do-glove.github.io/.

cs.RO↗

arXiv2025

Towards AI-driven Sign Language Generation with Non-manual Markers

Sign languages are essential for the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems have the potential to support communication by translating from written languages, such as English, into signed videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures, the absence of facial cues and body language, and insufficient visual and motion fidelity. We address these challenges by building on recent advances in LLMs and video generation models to translate English sentences into natural-looking AI ASL signers. The text component of our model extracts information for manual and non-manual components of ASL, which are used to synthesize skeletal pose sequences and corresponding video frames. Our findings from a user study with 30 DHH participants and thorough technical evaluations demonstrate significant progress and identify critical areas necessary to meet user needs.

cs.HC↗

arXiv2025

A Comprehensive Review of Protein Language Models

At the intersection of the rapidly growing biological data landscape and advancements in Natural Language Processing (NLP), protein language models (PLMs) have emerged as a transformative force in modern research. These models have achieved remarkable progress, highlighting the need for timely and comprehensive overviews. However, much of the existing literature focuses narrowly on specific domains, often missing a broader analysis of PLMs. This study provides a systematic review of PLMs from a macro perspective, covering key historical milestones and current mainstream trends. We focus on the models themselves and their evaluation metrics, exploring aspects such as model architectures, positional encoding, scaling laws, and datasets. In the evaluation section, we discuss benchmarks and downstream applications. To further support ongoing research, we introduce relevant mainstream tools. Lastly, we critically examine the key challenges and limitations in this rapidly evolving field.

q-bio.BM↗

arXiv2025

Postselection-Free Cavity-Enhanced Narrow-Band Orbital Angular Momentum Entangled Photon Source

Cavity-enhanced spontaneous parametric down-conversion (SPDC) provides a significant way to produce $\sim$10 MHz narrow-band photon pairs, which matches the bandwidth of photon for quantum memory. However, the output photon pairs from the cavity is not entangled and the postselection is required to create the entanglement outside the cavity, so the direct output of cavity-enhanced narrow-band entangled photon pairs is still an open challenge. Here we propose a solution that realizes the first postselection-free cavity-enhanced narrow-band entangled photon pairs. The entanglement is achieved in degree of freedom (DOF) of orbital angular momentum (OAM) by implementing an OAM-conservation SPDC process in an actively and precisely controlled cavity supporting degenerate high-order OAM modes. The measured linewidth for the two photons is 13.8 MHz and the measured fidelity is 0.969(3) for the directly generated OAM entangled two photons. We deterministically transfer the OAM entanglement to polarization one with almost no loss and obtain polarization entangled two photons with a fidelity of 0.948(2). Moreover, we produce narrow-band OAM-polarization hyperentangled photon pairs with a fidelity of 0.850(2) by establishing polarization entanglement with preservation of OAM entanglement, which is realized by interfering the two photons on a polarizing beam splitter (PBS) and post-selecting the events of one and only one photon in on each of the PBS port. Novel cavity may find applications in cavity-based light-matter interaction. Our results provide an efficient and promising approach to create narrow-band entangled photon sources for memory-based long-distance quantum communication and network.

quant-ph↗

arXiv2025

Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark

Empathetic Response Generation (ERG) is one of the key tasks of the affective computing area, which aims to produce emotionally nuanced and compassionate responses to user's queries. However, existing ERG research is predominantly confined to the singleton text modality, limiting its effectiveness since human emotions are inherently conveyed through multiple modalities. To combat this, we introduce an avatar-based Multimodal ERG (MERG) task, entailing rich text, speech, and facial vision information. We first present a large-scale high-quality benchmark dataset, \textbf{AvaMERG}, which extends traditional text ERG by incorporating authentic human speech audio and dynamic talking-face avatar videos, encompassing a diverse range of avatar profiles and broadly covering various topics of real-world scenarios. Further, we deliberately tailor a system, named \textbf{Empatheia}, for MERG. Built upon a Multimodal Large Language Model (MLLM) with multimodal encoder, speech and avatar generators, Empatheia performs end-to-end MERG, with Chain-of-Empathetic reasoning mechanism integrated for enhanced empathy understanding and reasoning. Finally, we devise a list of empathetic-enhanced tuning strategies, strengthening the capabilities of emotional accuracy and content, avatar-profile consistency across modalities. Experimental results on AvaMERG data demonstrate that Empatheia consistently shows superior performance than baseline methods on both textual ERG and MERG. Overall, this work is expected to pioneer the MERG research by introducing a novel benchmark and an end-to-end model, laying a solid foundation for future advancements in multimodal empathetic response generation.

cs.MM↗

arXiv2025

Roles of the scalar $f_0(500)$ and $f_0(980)$ in the process $D^0\to π^0π^0 \bar{K}^0$

Motivated by the near-threshold enhancement and the dip structure around 1~GeV in the $π^0π^0$ invariant mass distribution of the process $D^0\to π^0π^0\bar{K}^0$ observed by the CLEO Collaboration, we have investigated this process by taking into account the contribution from the $S$-wave pseudoscalar meson-pseudoscalar meson interactions within the chiral unitary approach, and also the one from the intermediate resonance $K^{*}(892)$. Our results are in good agreement with the CLEO measurements, which implies that, the near-threshold enhancement near the $π^0π^0$ threshold is mainly due to the contributions from the scalar meson $f_0(500)$ and the intermediate $K^*$, and the cusp structure around 1~GeV in the $π^0π^0$ invariant mass distribution should be associated with the scalar meson $f_0(980)$.

hep-ph↗

arXiv2025

A phenomenological estimate of rescattering effects in $B_s\to K^{0}\bar{K}^{0}$

The measurements in $b\to s$ penguin-dominated decays are widely recognized as a powerful test for searching for New Physics by studying the deviation from theoretical estimations within the Standard Model. We examine the final-state rescattering effects on the decay $B_s\to K^{*0}\bar{K}^{*0}$ and provide estimations of the branching ratio and longitudinal polarization of $B_s\to K^{*0}\bar{K}^{*0}$, which is consistent with experimental observations. Our conclusion is that both short- and long-distance interactions contribute significantly in this decay. The small longitudinal polarization in $B\to VV$ modes may not be a signal for New Physics.

hep-ph↗

arXiv2025

Addressing Bias in Generative AI: Challenges and Research Opportunities in Information Management

Generative AI technologies, particularly Large Language Models (LLMs), have transformed information management systems but introduced substantial biases that can compromise their effectiveness in informing business decision-making. This challenge presents information management scholars with a unique opportunity to advance the field by identifying and addressing these biases across extensive applications of LLMs. Building on the discussion on bias sources and current methods for detecting and mitigating bias, this paper seeks to identify gaps and opportunities for future research. By incorporating ethical considerations, policy implications, and sociotechnical perspectives, we focus on developing a framework that covers major stakeholders of Generative AI systems, proposing key research questions, and inspiring discussion. Our goal is to provide actionable pathways for researchers to address bias in LLM applications, thereby advancing research in information management that ultimately informs business practices. Our forward-looking framework and research agenda advocate interdisciplinary approaches, innovative methods, dynamic perspectives, and rigorous evaluation to ensure fairness and transparency in Generative AI-driven information systems. We expect this study to serve as a call to action for information management scholars to tackle this critical issue, guiding the improvement of fairness and effectiveness in LLM-based systems for business practice.

cs.CY↗

arXiv2025

Generative AI-enabled Blockage Prediction for Robust Dual-Band mmWave Communication

In mmWave wireless networks, signal blockages present a significant challenge due to the susceptibility to environmental moving obstructions. Recently, the availability of visual data has been leveraged to enhance blockage prediction accuracy in mmWave networks. In this work, we propose a Vision Transformer (ViT)-based approach for visual-aided blockage prediction that intelligently switches between mmWave and Sub-6 GHz frequencies to maximize network throughput and maintain reliable connectivity. Given the computational demands of processing visual data, we implement our solution within a hierarchical fog-cloud computing architecture, where fog nodes collaborate with cloud servers to efficiently manage computational tasks. This structure incorporates a generative AI-based compression technique that significantly reduces the volume of visual data transmitted between fog nodes and cloud centers. Our proposed method is tested with the real-world DeepSense 6G dataset, and according to the simulation results, it achieves a blockage prediction accuracy of 92.78% while reducing bandwidth usage by 70.31%.

eess.SP↗

arXiv2025

Rethinking Early-Fusion Strategies for Improved Multimodal Image Segmentation

RGB and thermal image fusion have great potential to exhibit improved semantic segmentation in low-illumination conditions. Existing methods typically employ a two-branch encoder framework for multimodal feature extraction and design complicated feature fusion strategies to achieve feature extraction and fusion for multimodal semantic segmentation. However, these methods require massive parameter updates and computational effort during the feature extraction and fusion. To address this issue, we propose a novel multimodal fusion network (EFNet) based on an early fusion strategy and a simple but effective feature clustering for training efficient RGB-T semantic segmentation. In addition, we also propose a lightweight and efficient multi-scale feature aggregation decoder based on Euclidean distance. We validate the effectiveness of our method on different datasets and outperform previous state-of-the-art methods with lower parameters and computation.

cs.CV↗

arXiv2025

NLO EW corrections to tau pair production via photon fusion in Pb-Pb ultraperipheral collision

We study the next-to-leading order (NLO) electroweak (EW) corrections to the $γγ\to τ^+ τ^-$ process in Pb-Pb ultraperipheral collision (UPC). We find that the EW correction $δσ_{\mathrm{EW}}$ decreases the total cross section $σ_{\mathrm{NLO}} = σ_{\mathrm{LO}} + δσ_{\mathrm{EW}}$ by -3\% at Pb-Pb center-of-mass energy $\sqrt{s_{NN}}=5.02$ TeV. The weak correction plays significant role whose contribution is about -4 times of that of QED. The CMS and ATLAS collaborations use the reaction $γγ\to τ^+ τ^-$ in Pb-Pb and proton-proton UPC to constrain tau's anomalous magnetic moment $a_τ$. By parameterizing the $γττ$ vertex with two form factors $F_{1,2}$, the cross section can be written as $σ_{a_τ} = σ_{\mathrm{LO}} + δσ_{a_τ}$, where $δσ_{a_τ}$ is proportional to $a_τ$. The impact of NLO EW corrections on $a_τ$ bounds in Pb-Pb UPC is limited, as the current experimental bounds are loose. We also find that various differential distributions of the two ratios $\mathrm{d} σ_{\mathrm{NLO}}/ \mathrm{d} σ_{\mathrm{LO}}$ and $\mathrm{d} σ_{a_τ}/ \mathrm{d} σ_{\mathrm{LO}}$ have different lineshapes. This work is significant to precisely study the interaction of $γττ$ via $γγ\to τ^+ τ^-$ process.

hep-ph↗

arXiv2024

COPR: Continual Human Preference Learning via Optimal Policy Regularization

Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences. Given the evolving nature of human preferences, continual alignment becomes more crucial and practical in comparison to traditional static alignment. Nevertheless, making RLHF compatible with Continual Learning (CL) is challenging due to its complex process. Meanwhile, directly learning new human preferences may lead to Catastrophic Forgetting (CF) of historical preferences, resulting in helpless or harmful outputs. To overcome these challenges, we propose the Continual Optimal Policy Regularization (COPR) method, which draws inspiration from the optimal policy theory. COPR utilizes a sampling distribution as a demonstration and regularization constraints for CL. It adopts the Lagrangian Duality (LD) method to dynamically regularize the current policy based on the historically optimal policy, which prevents CF and avoids over-emphasizing unbalanced objectives. We also provide formal proof for the learnability of COPR. The experimental results show that COPR outperforms strong CL baselines on our proposed benchmark, in terms of reward-based, GPT-4 evaluations and human assessment. Furthermore, we validate the robustness of COPR under various CL settings, including different backbones, replay memory sizes, and learning orders.

cs.LG↗

arXiv2024

Correcting Large Language Model Behavior via Influence Function

Recent advancements in AI alignment techniques have significantly improved the alignment of large language models (LLMs) with static human preferences. However, the dynamic nature of human preferences can render some prior training data outdated or even erroneous, ultimately causing LLMs to deviate from contemporary human preferences and societal norms. Existing methodologies, whether they involve the curation of new data for continual alignment or the manual correction of outdated data for re-alignment, demand costly human resources. To address this challenge, we propose a novel approach, Large Language Model Behavior Correction with Influence Function Recall and Post-Training (LANCET), which requires no human involvement. LANCET consists of two phases: (1) using influence functions to identify the training data that significantly impact undesirable model outputs, and (2) applying an Influence function-driven Bregman Optimization (IBO) technique to adjust the model's behavior based on these influence distributions. Our experiments demonstrate that LANCET effectively and efficiently correct inappropriate behaviors of LLMs. Furthermore, LANCET can outperform methods that rely on collecting human preferences, and it enhances the interpretability of learning human preferences within LLMs.

cs.LG↗

arXiv2024

ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers

Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the inherent in-context generation capabilities of pretrained diffusion transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks with minimal or no architectural modifications. These capabilities are unlocked by concatenating self-attention tokens across multiple input and target images, combined with grouped and masked generation pipelines. Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications. Users can interact with ChatDiT to create interleaved text-image articles, multi-page picture books, edit images, design IP derivatives, or develop character design settings, all through free-form natural language across one or more conversational rounds. At its core, ChatDiT employs a multi-agent system comprising three key components: an Instruction-Parsing agent that interprets user-uploaded images and instructions, a Strategy-Planning agent that devises single-step or multi-step generation actions, and an Execution agent that performs these actions using an in-context toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with diverse instructions and varying numbers of input and target images. Despite its simplicity and training-free approach, ChatDiT surpasses all competitors, including those specifically designed and trained on extensive multi-task datasets. We further identify key limitations of pretrained DiTs in zero-shot adapting to tasks. We release all code, agents, results, and intermediate outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT

cs.CV↗