arXiv subjects

Xun Chen

Publications and source records attributed to Xun Chen.

At least 127 records · Page 7Linked to original sources

arXiv2024

Learning to Compress Prompt in Natural Language Formats

Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.

cs.CL↗

arXiv2024

Machine learning holographic black hole from lattice QCD equation of state

Based on lattice QCD results of equation of state (EOS) and baryon number susceptibility at zero baryon chemical potential, and supplemented by machine learning techniques, we construct the analytic form of the holographic black hole metric in the Einstein-Maxwell-Dilaton (EMD) framework for pure gluon, 2-flavor, and 2+1-flavor systems, respectively. The dilaton potentials solved from Einstein equations are in good agreement with the extended non-conformal DeWolfe-Gubser-Rosen (DGR) type dilaton potentials fixed by lattice QCD EOS, which indicates the robustness of the EMD framework. The predicted critical endpoint (CEP) in the 2+1-flavor system is located at $(T^c$=0.094GeV, $μ^c_B$=0.74GeV), which is close to the results from the realistic Polyakov-Nambu-Jona-Lasinio(PNJL) model, the functional renormalization group, and the holographic model with extended DeWolfe-Gubser-Rosen dilaton potential.

hep-ph↗

arXiv2024

InternLM2 Technical Report

The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.

cs.CL↗

arXiv2024

Role of Coulomb interaction in elastic pion-proton scattering from holography

Differential cross sections of the elastic pion-proton scattering are investigated at very small momentum transfer in a holographic QCD model, considering both the strong and Coulomb interaction in the Regge regime. The strong interaction is described by the Pomeron and Reggeon exchange, and the Coulomb interaction is characterized by the one photon exchange. The two interactions are linked through an interference term and we only need to determine a single adjustable parameter involved in this term. As to the parameters for the strong interaction, we can utilize the values determined in the previous studies. The differential cross sections can be predicted without any additional parameters, and it is shown that our predictions are consistent with the experimental data. We explicitly show the momentum transfer dependence for the interference effect. The energy dependence of the contribution ratios for each component is also discussed.

hep-ph↗

arXiv2024

TrojFSP: Trojan Insertion in Few-shot Prompt Tuning

Prompt tuning is one of the most effective solutions to adapting a fixed pre-trained language model (PLM) for various downstream tasks, especially with only a few input samples. However, the security issues, e.g., Trojan attacks, of prompt tuning on a few data samples are not well-studied. Transferring established data poisoning attacks directly to few-shot prompt tuning presents multiple challenges. One significant issue is the \textit{poisoned imbalance issue}, where non-target class samples are added to the target class, resulting in a greater number of target-class samples compared to non-target class. While this issue is not critical in regular tuning, it significantly hampers the few-shot prompt tuning, making it difficult to simultaneously achieve a high attack success rate (ASR) and maintain clean data accuracy (CDA). Additionally, few-shot prompting is prone to overfitting in terms of both ASR and CDA. In this paper, we introduce \textit{TrojFSP}, a method designed to address the challenges. To solve the poisoned imbalance issue, we develop a \textit{Target-Class Shrink (TC-Shrink)} technique, which aims to equalize the number of poisoning samples. To combat overfitting, we employ a \textit{Selective Token Poisoning} technique to boost attack performance. Furthermore, we introduce a \textit{Trojan-Trigger Attention} objective function to amplify the attention of the poisoned trojan prompt on triggers. Experiments show that our TrojFSP achieves an ASR of over 99\% while maintaining negligible decreases in CDA across various PLMs and datasets.

cs.LG↗

arXiv2024

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorporates three flexible sharding strategies: Full-Replica, Full-Sharding, and Partial-Sharding, and allows each component within the model states (Parameters, Gradients, Optimizer States) to independently choose a sharding strategy as well as the device mesh. We conduct a thorough analysis of communication costs, formulating an optimization problem to discover the optimal sharding strategy. Additionally, AMSP optimizes distributed LLM training by efficiently overlapping communication with computation. Evaluations demonstrate up to 52\% Model FLOPs Utilization (MFU) when training the LLaMA-based model on 1024 GPUs, resulting in a 1.56 times improvement in training throughput compared to newly proposed systems like MiCS and ZeRO++.

cs.DC↗

arXiv2024

The potential of QQQ in the anisotropic background

In this work, we use the AdS/CFT correspondence to study the behavior of a triply heavy baryon within anisotropic backgrounds. Beginning with the total action of the three quarks, we derive the balance equation for the three-quark system and compute the separation distance and potential energy. Our results reveal a consistent decrease in both the separation distance and potential energy for the A configuration and the B configuration as the anisotropy coefficient $a$ increases. This suggests that the presence of an anisotropic background promotes the dissolution of the three-quark system. Additionally, we compare the potential energies of the A and B configurations and observe that the A configuration has a slightly smaller potential energy, suggesting greater stability compared to the B configuration.

hep-ph↗

arXiv2024

Detecting Neutrinos from Supernova Bursts in PandaX-4T

Neutrinos from core-collapse supernovae are essential for the understanding of neutrino physics and stellar evolution. The dual-phase xenon dark matter detectors can provide a way to track explosions of galactic supernovae by detecting neutrinos through coherent elastic neutrino-nucleus scatterings. In this study, a variation of progenitor masses as well as explosion models are assumed to predict the neutrino fluxes and spectra, which result in the number of expected neutrino events ranging from 6.6 to 13.7 at a distance of 10 kpc over a 10-second duration with negligible backgrounds at PandaX-4T. Two specialized triggering alarms for monitoring supernova burst neutrinos are built. The efficiency of detecting supernova explosions at various distances in the Milky Way is estimated. These alarms will be implemented in the real-time supernova monitoring system at PandaX-4T in the near future, providing the astronomical communities with supernova early warnings.

hep-ex↗

arXiv2024

Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies

Automated log analysis is crucial in modern software-intensive systems for facilitating program comprehension throughout software maintenance and engineering life cycles. Existing methods perform tasks such as log parsing and log anomaly detection by providing a single prediction value without interpretation. However, given the increasing volume of system events, the limited interpretability of analysis results hinders analysts' comprehension of program status and their ability to take appropriate actions. Moreover, these methods require substantial in-domain training data, and their performance declines sharply (by up to 62.5%) in online scenarios involving unseen logs from new domains, a common occurrence due to rapid software updates. In this paper, we propose LogPrompt, a novel interpretable log analysis approach for online scenarios. LogPrompt employs large language models (LLMs) to perform online log analysis tasks via a suite of advanced prompt strategies tailored for log tasks, which enhances LLMs' performance by up to 380.7% compared with simple prompts. Experiments on nine publicly available evaluation datasets across two tasks demonstrate that LogPrompt, despite requiring no in-domain training, outperforms existing approaches trained on thousands of logs by up to 55.9%. We also conduct a human evaluation of LogPrompt's interpretability, with six practitioners possessing over 10 years of experience, who highly rated the generated content in terms of usefulness and readability (averagely 4.42/5). LogPrompt also exhibits remarkable compatibility with open-source and smaller-scale LLMs, making it flexible for practical deployment. Code of LogPrompt is available at https://github.com/lunyiliu/LogPrompt.

cs.SE↗

arXiv2024

Holographic Schwinger Effect in Anisotropic Media

According to gauge/gravity correspondence, we study the holographic Schwinger effect within an anisotropic background. Firstly, the separate length of the particle-antiparticle pairs is computed within the context of an anisotropic background which is parameterized by dynamical exponent $ν$. It is found that the maximum separate length $x$ increases with the increase of dynamical exponent $ν$. By analyzing the potential energy, we find that the potential barrier increases with the dynamical exponent $ν$ at a small separate distance. This observation implies that the Schwinger effect within an anisotropic background is comparatively weaker when contrasted with its manifestation in an isotropic background.Finally, we also find that the Schwinger effect in the transverse direction is weakened compared to the parallel direction in the anisotropic background, which is consistent with the top-down model.

hep-th↗

arXiv2024

Doubly-heavy tetraquark at finite temperature in a holographic model

In this paper, we employ gauge/gravity duality to investigate the string breaking and melting of doubly-heavy tetraquark that includes two heavy quarks and two light antiquarks in a holographic model at finite temperature. Firstly, we investigate four configurations of $\rm{QQ\bar{q}\bar{q}}$ in the confined phase and consider different separation distances of the heavy quarks at varying temperatures. At high temperature, $\rm{QQ\bar{q}\bar{q}}$ melts at certain distances and the confined quarks are released. As the temperature continues to increase, some configurations of doubly-heavy tetraquark can not exist. Furthermore, we investigate three decay modes of $\rm{QQ\bar{q}\bar{q}}$ and compare the potential energy of $\rm{QQ\bar{q}\bar{q}}$ with that of $\rm{QQq}$ at finite temperature .

hep-ph↗

arXiv2024

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding

Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization.

cs.DC↗

arXiv2024

A Practical Survey on Emerging Threats from AI-driven Voice Attacks: How Vulnerable are Commercial Voice Control Systems?

The emergence of Artificial Intelligence (AI)-driven audio attacks has revealed new security vulnerabilities in voice control systems. While researchers have introduced a multitude of attack strategies targeting voice control systems (VCS), the continual advancements of VCS have diminished the impact of many such attacks. Recognizing this dynamic landscape, our study endeavors to comprehensively assess the resilience of commercial voice control systems against a spectrum of malicious audio attacks. Through extensive experimentation, we evaluate six prominent attack techniques across a collection of voice control interfaces and devices. Contrary to prevailing narratives, our results suggest that commercial voice control systems exhibit enhanced resistance to existing threats. Particularly, our research highlights the ineffectiveness of white-box attacks in black-box scenarios. Furthermore, the adversaries encounter substantial obstacles in obtaining precise gradient estimations during query-based interactions with commercial systems, such as Apple Siri and Samsung Bixby. Meanwhile, we find that current defense strategies are not completely immune to advanced attacks. Our findings contribute valuable insights for enhancing defense mechanisms in VCS. Through this survey, we aim to raise awareness within the academic community about the security concerns of VCS and advocate for continued research in this crucial area.

cs.CR↗

arXiv2023

Stable Unlearnable Example: Enhancing the Robustness of Unlearnable Examples via Stable Error-Minimizing Noise

The open source of large amounts of image data promotes the development of deep learning techniques. Along with this comes the privacy risk of these open-source image datasets being exploited by unauthorized third parties to train deep learning models for commercial or illegal purposes. To avoid the abuse of public data, a poisoning-based technique, the unlearnable example, is proposed to significantly degrade the generalization performance of models by adding a kind of imperceptible noise to the data. To further enhance its robustness against adversarial training, existing works leverage iterative adversarial training on both the defensive noise and the surrogate model. However, it still remains unknown whether the robustness of unlearnable examples primarily comes from the effect of enhancement in the surrogate model or the defensive noise. Observing that simply removing the adversarial noise on the training process of the defensive noise can improve the performance of robust unlearnable examples, we identify that solely the surrogate model's robustness contributes to the performance. Furthermore, we found a negative correlation exists between the robustness of defensive noise and the protection performance, indicating defensive noise's instability issue. Motivated by this, to further boost the robust unlearnable example, we introduce stable error-minimizing noise (SEM), which trains the defensive noise against random perturbation instead of the time-consuming adversarial perturbation to improve the stability of defensive noise. Through extensive experiments, we demonstrate that SEM achieves a new state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet Subset in terms of both effectiveness and efficiency. The code is available at https://github.com/liuyixin-louis/Stable-Unlearnable-Example.

cs.LG↗

arXiv2023

Search for Dark-Matter-Nucleon Interactions with a Dark Mediator in PandaX-4T

We report results of a search for dark-matter-nucleon interactions via a dark mediator using optimized low-energy data from the PandaX-4T liquid xenon experiment. With the ionization-signal-only data and utilizing the Migdal effect, we set the most stringent limits on the cross section for dark matter masses ranging from 30~$\rm{MeV/c^2}$ to 2~$\rm{GeV/c^2}$. Under the assumption that the dark mediator is a dark photon that decays into scalar dark matter pairs in the early Universe, we rule out significant parameter space of such thermal relic dark-matter model.

hep-ex↗

arXiv2023

Unsupervised Protein-Ligand Binding Energy Prediction via Neural Euler's Rotation Equation

Protein-ligand binding prediction is a fundamental problem in AI-driven drug discovery. Prior work focused on supervised learning methods using a large set of binding affinity data for small molecules, but it is hard to apply the same strategy to other drug classes like antibodies as labelled data is limited. In this paper, we explore unsupervised approaches and reformulate binding energy prediction as a generative modeling task. Specifically, we train an energy-based model on a set of unlabelled protein-ligand complexes using SE(3) denoising score matching and interpret its log-likelihood as binding affinity. Our key contribution is a new equivariant rotation prediction network called Neural Euler's Rotation Equations (NERE) for SE(3) score matching. It predicts a rotation by modeling the force and torque between protein and ligand atoms, where the force is defined as the gradient of an energy function with respect to atom coordinates. We evaluate NERE on protein-ligand and antibody-antigen binding affinity prediction benchmarks. Our model outperforms all unsupervised baselines (physics-based and statistical potentials) and matches supervised learning methods in the antibody case.

q-bio.BM↗

arXiv2023

Optimization of convolutional neural networks for background suppression in the PandaX-III experiment

The tracks recorded by a gaseous detector provide a possibility for charged particle identification. For searching the neutrinoless double beta decay events of 136Xe in the PandaX-III experiment, we optimized the convolutional neural network based on the Monte Carlo simulation data to improve the signal-background discrimination power. EfficientNet is chosen as the baseline model and the optimization is performed by tuning the hyperparameters. In particular, the maximum discrimination power is achieved by optimizing the channel number of the top convolutional layer. In comparison with our previous work, the significance of discrimination has been improved by about 70%.

physics.ins-det↗

arXiv2023

String breaking and running coupling of $\rm Q\bar{Q}$ in a rotating media from holography

Through gravity/gauge duality, the string breaking and running coupling constant of heavy quark-antiquark pair are investigated in the rotating background. For the meson decay mode $\rm Q\bar{Q} \rightarrow Q \bar{q}+\bar{Q} q$, we discuss the string breaking and running coupling in the parallel and transverse case. It is found that the parallel case has a more significant on string breaking and running coupling constant than the transverse case in the confined phase. The string-breaking distance and the maximum value of running coupling will decrease with the increase of angular velocity in the parallel case. Besides, we also investigate the running coupling and screening distance at finite angular velocity in the deconfined phase. It is found that the maximum values of the running coupling and screening distance are decreasing functions of angular velocity. The parallel case has a more significant influence on the running coupling and screening distance than the transverse case in the deconfined phase.

hep-ph↗