arXiv subjects

Peng Sun

Publications and source records attributed to Peng Sun.

At least 145 records · Page 8Linked to original sources

arXiv2024

Extraction of trans-helicity worm-gear distributions and opportunities at the Electron-Ion Collider in China

We present a global analysis of the trans-helicity worm-gear distribution function, $g_{1T}^\perp$, by fitting the longitudinal-transverse double spin asymmetry data of the semi-inclusive deep inelastic scattering. The analysis is performed within the framework of transverse momentum dependent factorization and evolution. It is found that the $u$-quark favors a positive distribution and the $d$-quark favors a negative distribution, which is consistent with previous model calculations and phenomenological extractions. Based on the fit to existing world data, we also study the impact of the proposed electron-ion collider in China and conclude that it can significantly improve the precision of the worm-gear distribution function and hence enhance our understanding of nucleon spin structures.

hep-ph↗

arXiv2024

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R$^3$ overcomes these limitations by learning from correct demonstrations. Specifically, R$^3$ progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R$^3$ establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7B, our method surpasses RL baseline on eight reasoning tasks by $4.1$ points on average. Notebaly, in program-based reasoning on GSM8K, it exceeds the baseline by $4.2$ points across three backbone models, and without any extra data, Codellama-7B + R$^3$ performs comparable to larger models or closed-source models.

cs.AI↗

arXiv2024

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorporates three flexible sharding strategies: Full-Replica, Full-Sharding, and Partial-Sharding, and allows each component within the model states (Parameters, Gradients, Optimizer States) to independently choose a sharding strategy as well as the device mesh. We conduct a thorough analysis of communication costs, formulating an optimization problem to discover the optimal sharding strategy. Additionally, AMSP optimizes distributed LLM training by efficiently overlapping communication with computation. Evaluations demonstrate up to 52\% Model FLOPs Utilization (MFU) when training the LLaMA-based model on 1024 GPUs, resulting in a 1.56 times improvement in training throughput compared to newly proposed systems like MiCS and ZeRO++.

cs.DC↗

arXiv2024

Towards three-dimensional nucleon structures at the Electron-Ion Collider in China: A study of the Sivers function

Transverse momentum-dependent parton distribution functions (TMDs) provide three-dimensional imaging of the nucleon in the momentum space. With its fundamental importance in understanding the spin structure of the nucleon, the precise measurement of TMDs is considered as one of the main physics topics of the proposed Electron-Ion Collider in China (EicC). In this paper, we investigate the impact of future semi-inclusive deep inelastic scattering (SIDIS) data from EicC on the extraction of TMDs. Taking the Sivers function as an example, we revisited the world SIDIS data, which serves as the input for the simulated EicC data. By performing a combined analysis of the world data and the EicC simulated data, we quantitatively demonstrate that the Sivers functions can be precisely determined for various quark flavors, especially for sea quarks, in the region, $x>0.005$, directly covered by the EicC.

hep-ph↗

arXiv2024

RobWE: Robust Watermark Embedding for Personalized Federated Learning Model Ownership Protection

Embedding watermarks into models has been widely used to protect model ownership in federated learning (FL). However, existing methods are inadequate for protecting the ownership of personalized models acquired by clients in personalized FL (PFL). This is due to the aggregation of the global model in PFL, resulting in conflicts over clients' private watermarks. Moreover, malicious clients may tamper with embedded watermarks to facilitate model leakage and evade accountability. This paper presents a robust watermark embedding scheme, named RobWE, to protect the ownership of personalized models in PFL. We first decouple the watermark embedding of personalized models into two parts: head layer embedding and representation layer embedding. The head layer belongs to clients' private part without participating in model aggregation, while the representation layer is the shared part for aggregation. For representation layer embedding, we employ a watermark slice embedding operation, which avoids watermark embedding conflicts. Furthermore, we design a malicious watermark detection scheme enabling the server to verify the correctness of watermarks before aggregating local models. We conduct an exhaustive experimental evaluation of RobWE. The results demonstrate that RobWE significantly outperforms the state-of-the-art watermark embedding schemes in FL in terms of fidelity, reliability, and robustness.

cs.CR↗

arXiv2024

Large Language Models as Agents in Two-Player Games

By formally defining the training processes of large language models (LLMs), which usually encompasses pre-training, supervised fine-tuning, and reinforcement learning with human feedback, within a single and unified machine learning paradigm, we can glean pivotal insights for advancing LLM technologies. This position paper delineates the parallels between the training methods of LLMs and the strategies employed for the development of agents in two-player games, as studied in game theory, reinforcement learning, and multi-agent systems. We propose a re-conceptualization of LLM learning processes in terms of agent learning in language-based games. This framework unveils innovative perspectives on the successes and challenges in LLM development, offering a fresh understanding of addressing alignment issues among other strategic considerations. Furthermore, our two-player game approach sheds light on novel data preparation and machine learning techniques for training LLMs.

cs.CL↗

arXiv2024

Generalized Video Anomaly Event Detection: Systematic Taxonomy and Comparison of Deep Models

Video Anomaly Detection (VAD) serves as a pivotal technology in the intelligent surveillance systems, enabling the temporal or spatial identification of anomalous events within videos. While existing reviews predominantly concentrate on conventional unsupervised methods, they often overlook the emergence of weakly-supervised and fully-unsupervised approaches. To address this gap, this survey extends the conventional scope of VAD beyond unsupervised methods, encompassing a broader spectrum termed Generalized Video Anomaly Event Detection (GVAED). By skillfully incorporating recent advancements rooted in diverse assumptions and learning frameworks, this survey introduces an intuitive taxonomy that seamlessly navigates through unsupervised, weakly-supervised, supervised and fully-unsupervised VAD methodologies, elucidating the distinctions and interconnections within these research trajectories. In addition, this survey facilitates prospective researchers by assembling a compilation of research resources, including public datasets, available codebases, programming tools, and pertinent literature. Furthermore, this survey quantitatively assesses model performance, delves into research challenges and directions, and outlines potential avenues for future exploration.

cs.CV↗

arXiv2024

Hybrid Message Passing-Based Detectors for Uplink Grant-Free NOMA Systems

This paper studies improving the detector performance which considers the activity state (AS) temporal correlation of the user equipments (UEs) in the time domain under the uplink grant-free non-orthogonal multiple access (GF-NOMA) system. The Bernoulli Gaussian-Markov chain (BG-MC) probability model is used for exploiting both the sparsity and slow change characteristic of the AS of the UE. The GAMP Bernoulli Gaussian-Markov chain (GAMP-BG-MC) algorithm is proposed to improve the detector performance, which can utilize the bidirectional message passing between the neighboring time slots to fully exploit the temporally-correlated AS of the UE. Furthermore, the parameters of the BG-MC model can be updated adaptively during the estimation procedure with unknown system statistics. Simulation results show that the proposed algorithm can improve the detection accuracy compared with the existing methods while keeping the same order complexity.

cs.IT↗

arXiv2024

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding

Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization.

cs.DC↗

arXiv2024

BiSinger: Bilingual Singing Voice Synthesis

Although Singing Voice Synthesis (SVS) has made great strides with Text-to-Speech (TTS) techniques, multilingual singing voice modeling remains relatively unexplored. This paper presents BiSinger, a bilingual pop SVS system for English and Chinese Mandarin. Current systems require separate models per language and cannot accurately represent both Chinese and English, hindering code-switch SVS. To address this gap, we design a shared representation between Chinese and English singing voices, achieved by using the CMU dictionary with mapping rules. We fuse monolingual singing datasets with open-source singing voice conversion techniques to generate bilingual singing voices while also exploring the potential use of bilingual speech data. Experiments affirm that our language-independent representation and incorporation of related datasets enable a single model with enhanced performance in English and code-switch SVS while maintaining Chinese song performance. Audio samples are available at https://bisinger-svs.github.io.

eess.AS↗

arXiv2024

Quark masses and low energy constants in the continuum from the tadpole improved clover ensembles

We present the light-flavor quark masses and low energy constants using the 2+1 flavor full-QCD ensembles with stout smeared clover fermion action and Symanzik gauge actions. Both the fermion and gauge actions are tadpole improved self-consistently. The simulations are performed on 11 ensembles at 3 lattice spacings $a\in[0.05,0.11]$ fm, 4 spatial sizes $L\in[2.5, 5.1]$ fm, 7 pion masses $m_π\in[135,350]$ MeV, and several values of the strange quark mass. The quark mass is defined through the partially conserved axial current (PCAC) relation and renormalized to $\overline{\mathrm{MS}}$ 2 GeV through the intermediate regularization independent momentum subtraction (RI/MOM) scheme. The systematic uncertainty of using the symmetric momentum subtraction (SMOM) scheme is also included. Eventually, we predict $m_u=2.45(22)(20)$ MeV, $m_d=4.74(11)(09)$ MeV, and $m_s=98.8(2.9)(4.7)$ MeV with the systematic uncertainties from lattice spacing determination, continuum extrapolation and renormalization constant included. We also obtain the chiral condensate $Σ^{1/3}=268.6(3.6)(0.7)$ MeV and the pion decay constant $F=86.6(7)(1.4) $ MeV in the $N_f=2$ chiral limit, and the next-to-leading order low energy constants $\ell_3=2.43(54)(05)$ and $\ell_4=4.322(75)(96)$.

hep-lat↗

arXiv2023

Automatic differentiation accelerated shape optimization approaches to photonic inverse design on rectilinear simulation grids

Shape optimization approaches to inverse design offer low-dimensional, physically-guided parameterizations of structures by representing them as combinations of shape primitives. However, on discretized rectilinear simulation grids, computing the gradient of a user objective via the adjoint variables method requires a sum reduction of the forward/adjoint field solutions and the Jacobian of the simulation material distribution with respect to the structural shape parameters. These shape parameters often perturb large or global parts of the simulation grid resulting in many non-zero Jacobian entries, which are typically computed by finite-difference in practice. Consequently, the gradient calculation can be non-trivial. In this work we propose to accelerate the gradient calculation by invoking automatic differentiation (AutoDiff) in instantiations of structural material distributions. In doing so, we develop extensible differentiable mappings from shape parameters to shape primitives and differentiable effective logic operations (denoted AutoDiffGeo). These AutoDiffGeo definitions may introduce some additional discretization error into the field solutions because they relax notions of sub-pixel smoothing along shape boundaries. However, we show that some mappings (e.g. simple cuboids) can achieve zero error with respect to volumetric averaging strategies. We demonstrate AutoDiff enhanced shape optimization using three integrated photonic examples: a multi-etch blazed grating coupler, a non-adiabatic waveguide transition taper, and a polarization-splitting grating coupler. We find accelerations of the gradient calculation by AutoDiff relative to finite-difference often exceed 50x, resulting in total wall time accelerations of 4x or more on the same hardware with little or no compromise to final device performance. Our code is available open source at https://github.com/smhooten/emopt

cs.CE↗

arXiv2023

TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch

TorchAudio is an open-source audio and speech processing library built for PyTorch. It aims to accelerate the research and development of audio and speech technologies by providing well-designed, easy-to-use, and performant PyTorch components. Its contributors routinely engage with users to understand their needs and fulfill them by developing impactful features. Here, we survey TorchAudio's development principles and contents and highlight key features we include in its latest version (2.1): self-supervised learning pre-trained pipelines and training recipes, high-performance CTC decoders, speech recognition models and training recipes, advanced media I/O capabilities, and tools for performing forced alignment, multi-channel speech enhancement, and reference-less speech assessment. For a selection of these features, through empirical studies, we demonstrate their efficacy and show that they achieve competitive or state-of-the-art performance.

eess.AS↗

arXiv2023

Role of sea quarks in the nucleon transverse spin

We present a phenomenological extraction of transversity distribution functions and Collins fragmentation functions by simultaneously fitting to semi-inclusive deep inelastic scattering and electron-positron annihilation data. The analysis is performed within the transverse momentum dependent factorization formalism, and sea quark transversity distributions are taken into account for the first time. We find the $\bar u$ quark favors a negative transversity distribution while that of the $\bar d$ quark is consistent with zero according to the current accuracy. In addition, based on a combined analysis of world data and simulated data, we quantitatively demonstrate the impact of the proposed Electron-ion Collider in China on precise determinations of the transversity distributions, especially for sea quarks, and the Collins fragmentation functions.

hep-ph↗

arXiv2023

Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception

Multi-agent collaborative perception as a potential application for vehicle-to-everything communication could significantly improve the perception performance of autonomous vehicles over single-agent perception. However, several challenges remain in achieving pragmatic information sharing in this emerging research. In this paper, we propose SCOPE, a novel collaborative perception framework that aggregates the spatio-temporal awareness characteristics across on-road agents in an end-to-end manner. Specifically, SCOPE has three distinct strengths: i) it considers effective semantic cues of the temporal context to enhance current representations of the target agent; ii) it aggregates perceptually critical spatial information from heterogeneous agents and overcomes localization errors via multi-scale feature interactions; iii) it integrates multi-source representations of the target agent based on their complementary contributions by an adaptive fusion paradigm. To thoroughly evaluate SCOPE, we consider both real-world and simulated scenarios of collaborative 3D object detection tasks on three datasets. Extensive experiments demonstrate the superiority of our approach and the necessity of the proposed components.

cs.CV↗

arXiv2023

Hidden-charm Hexaquarks from Lattice QCD

We present a lattice QCD study of hidden-charm hexaquarks with quark content $usc\bar{d}\bar{s}\bar{c}$ based on four ensembles of gauge configurations generated by CLQCD Collaboration with pion mass in the range of 220-300MeV. Four operators with quantum numbers $0^{++}, 0^{-+}, 1^{++}$ and $1^{--}$ respectively are constructed to interpolate the hexaquarks. After validating the spectrum and the dispersion relation for ordinary hadrons, we calculate the masses of the hexaquarks and extrapolate the results to the physical pion mass and the continuum limit. We find that the masses of the four hexaquarks are all below the $Ξ_c \bar Ξ_c$ threshold, while the $0^{-+}$ hexaquark lies around the $η_c K^+K^-$ threshold. These results will be helpful for experimental searches in future and for a deep understanding of the nature of multiquark states.

hep-lat↗

arXiv2023

ST-GIN: An Uncertainty Quantification Approach in Traffic Data Imputation with Spatio-temporal Graph Attention and Bidirectional Recurrent United Neural Networks

Traffic data serves as a fundamental component in both research and applications within intelligent transportation systems. However, real-world transportation data, collected from loop detectors or similar sources, often contains missing values (MVs), which can adversely impact associated applications and research. Instead of discarding this incomplete data, researchers have sought to recover these missing values through numerical statistics, tensor decomposition, and deep learning techniques. In this paper, we propose an innovative deep learning approach for imputing missing data. A graph attention architecture is employed to capture the spatial correlations present in traffic data, while a bidirectional neural network is utilized to learn temporal information. Experimental results indicate that our proposed method outperforms all other benchmark techniques, thus demonstrating its effectiveness.

cs.LG↗

arXiv2023

SST: A Simplified Swin Transformer-based Model for Taxi Destination Prediction based on Existing Trajectory

Accurately predicting the destination of taxi trajectories can have various benefits for intelligent location-based services. One potential method to accomplish this prediction is by converting the taxi trajectory into a two-dimensional grid and using computer vision techniques. While the Swin Transformer is an innovative computer vision architecture with demonstrated success in vision downstream tasks, it is not commonly used to solve real-world trajectory problems. In this paper, we propose a simplified Swin Transformer (SST) structure that does not use the shifted window idea in the traditional Swin Transformer, as trajectory data is consecutive in nature. Our comprehensive experiments, based on real trajectory data, demonstrate that SST can achieve higher accuracy compared to state-of-the-art methods.

cs.CV↗