arXiv subjects

Jian Li

Publications and source records attributed to Jian Li.

At least 271 records · Page 15Linked to original sources

arXiv2024

Look Before You Leap: Towards Decision-Aware and Generalizable Tool-Usage for Large Language Models

Tool-augmented large language models (LLMs) are attracting widespread attention when accessing up-to-date knowledge and alleviating hallucination issues. Nowadays, advanced closed-source LLMs (e.g., ChatGPT) have demonstrated surprising tool-usage capabilities through prompting and in-context learning techniques. To empower the capabilities of open-source LLMs (e.g., LLaMA) in manipulating tools, current efforts focus on either template-driven or token-triggered tool-usage. However, the former hampers LLMs' flexibility to address diverse user's queries due to constrained tool interactions, while the latter limits the generalizability when engaging with new tools, since tool-usage learning is based on task- and tool-specific datasets. To alleviate these concerns, in this paper, we propose a decision-aware and generalizable tool-usage framework (DEER). Specifically, we first construct the tool-usage samples with multiple decision branches via an automatic generation pipeline, thereby inspiring the decision-making awareness of LLMs under diverse scenarios. Meanwhile, we propose a novel tool sampling strategy to enhance the generalizability of LLMs over unseen tools. Extensive experiments demonstrate that our proposed DEER is effective and significantly outperforms baselines across various datasets.

cs.CL↗

arXiv2024

MooER: LLM-based Speech Recognition and Translation Models from Moore Threads

In this paper, we present MooER, a LLM-based large-scale automatic speech recognition (ASR) / automatic speech translation (AST) model of Moore Threads. A 5000h pseudo labeled dataset containing open source and self collected speech data is used for training. We achieve performance comparable to other open source models trained with up to hundreds of thousands of hours of labeled speech data. Meanwhile, experiments conducted on Covost2 Zh2en testset suggest that our model outperforms other open source Speech LLMs. A BLEU score of 25.2 can be obtained. The main contributions of this paper are summarized as follows. First, this paper presents a training strategy for encoders and LLMs on speech related tasks (including ASR and AST) using a small size of pseudo labeled data without any extra manual annotation and selection. Second, we release our ASR and AST models and plan to open-source our training code and strategy in the near future. Moreover, a model trained on 8wh scale training data is planned to be released later on.

cs.CL↗

arXiv2024

Distilling Mathematical Reasoning Capabilities into Small Language Models

This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought (CoT), Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for fine-tuning. Our experimental performance demonstrates that EoTD significantly boosts the reasoning abilities of SLMs, while ETD enables these models to achieve state-of-the-art reasoning performance.

cs.CL↗

arXiv2024

Towards Reliable Advertising Image Generation Using Human Feedback

In the e-commerce realm, compelling advertising images are pivotal for attracting customer attention. While generative models automate image generation, they often produce substandard images that may mislead customers and require significant labor costs to inspect. This paper delves into increasing the rate of available generated images. We first introduce a multi-modal Reliable Feedback Network (RFNet) to automatically inspect the generated images. Combining the RFNet into a recurrent process, Recurrent Generation, results in a higher number of available advertising images. To further enhance production efficiency, we fine-tune diffusion models with an innovative Consistent Condition regularization utilizing the feedback from RFNet (RFFT). This results in a remarkable increase in the available rate of generated images, reducing the number of attempts in Recurrent Generation, and providing a highly efficient production process without sacrificing visual appeal. We also construct a Reliable Feedback 1 Million (RF1M) dataset which comprises over one million generated advertising images annotated by human, which helps to train RFNet to accurately assess the availability of generated images and faithfully reflect the human feedback. Generally speaking, our approach offers a reliable solution for advertising image generation.

cs.CV↗

arXiv2024

A Survey on Model Compression for Large Language Models

Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.

cs.CL↗

arXiv2024

A Phase-resolved View of "Heartbeat"-like variability in IGR J17091-3624 During the 2022 Outburst

IGR J17091-3624, in addition to GRS 1915+105, is the only black hole X-ray binary that displays ``heartbeat"-like variability characterized by structured flares with high amplitudes. In this study, we conduct a detailed phase-resolved analysis of the recently identified ``heartbeat"-like Class X variability in IGR J17091-3624 during its 2022 outburst, utilizing data from NICER and NuSTAR observations. A shortage in the high-energy (>20 keV) X-ray flux is detected at peak phases of the soft X-ray flare at a ~15 sigma confidence level from the phase-folded light curves. Furthermore, our phase-resolved spectral analysis reveals variations in the spectral shape, particularly showing significant synchronous variations in the disk temperature and flux with the count rate. These findings imply that the flare is primarily driven by instabilities within the accretion disk, consistent with previous studies on the well-known Class rho variability in GRS 1915+105. However, we also observe a positive correlation between the disk temperature and flux over the flare cycle, which differs from a loop relation between the two parameters found in the Class rho variability. This could suggest differences in underlying physical processes between the two variability classes. Variations in the Componization component during flares are also observed: the electron temperature and covering fraction show anti-correlations with the disk flux, revealing potential interactions between the accretion disk and the corona during these flares.

astro-ph.HE↗

arXiv2024

A high-order, high-efficiency adaptive time filter algorithm for shale reservoir model based on coupled fluid flow with porous media flow

In this paper, a third-order time adaptive algorithm with less computation, low complexity is provided for shale reservoir model based on coupled fluid flow with porous media flow. The algorithm combines the three-step linear time filters method for simple post-processing and the second-order backward differential formula (BDF2), is third-order accurate and provides, at no extra computational complexity. At the same time, the time filter method can also be used to damp non-physical oscillations inherent in the BDF2 method, ensuring stability. We proves the variable time stepsize second-order backward differential formula plus time filter (BDF2-TF) algorithm's stability and the convergence properties of the fluid velocity u and hydraulic head $ϕ$ in the $L^2$ norm with an order of $O(k_{n+1}^3 + h^3)$. In the experiments, the adaptive algorithm automatically adjusts the time step in response to the varying characteristics of different models, ensuring that errors are maintained within acceptable limits. This algorithm addresses the issue that high-order algorithms may select inappropriate time steps, resulting in instability or reduced precision of the numerical solution, thereby enhancing calculation accuracy and efficiency. We perform three-dimensional numerical experiments to verify the BDF2-TF algorithm's effectiveness, stability, and third-order convergence. Simultaneously, a simplified model is employed to simulate the process of shale oil extraction from reservoirs, further demonstrating the algorithm's practical applicability.

math.NA↗

arXiv2024

Nonlinear magneto-optical response across van Hove singularity in a non-centrosymmetric magnetic Weyl semimetal

We investigate the nonlinear magneto-optical response in non-centrosymmetric magnetic Weyl semimetals featuring a quadratic tilt, focusing particularly on the influence of the van Hove singularity (VHS). In the absence of a magnetic field, the second-order nonlinear Drude conductivity components exhibit inflection or dip behavior across the VHS. In contrast, the second-order nonlinear anomalous Hall conductivity, primarily governed by the Berry curvature dipole, manifests a subtle plateau-like structure. As the tilt strength increases, the VHS energy escalates, thereby amplifying the VHS-induced characteristics within these second-order conductivity components. However, in the presence of a magnetic field, we show that the resultant magnetic moment suppresses nonlinear electron transport while enhancing nonlinear hole transport. %both suppresses and notably enhances nonlinear magnetic-optical transport in the electron and hole regions, respectively. This effect serves to mitigate the impact of the VHS, resulting specifically in an asymmetric peak or a kinked-like structure in the magnetic field-induced contribution to the second-order nonlinear conductivity near the Weyl nodes. These findings provide new insights into the intricate interplay among the VHS, Berry curvature, and magnetic moment in nonlinear magneto-optical transport through non-centrosymmetric magnetic Weyl semimetals.

cond-mat.mes-hall↗

arXiv2024

Phase-resolved Spectroscopy of Low-frequency Quasi-periodic Oscillations from the Newly Discovered Black Hole X-ray Binary Swift J1727.8-1613

Low-frequency quasi-periodic oscillations (LFQPOs) are commonly observed in X-ray light curves of black hole X-ray binaries (BHXRBs); however, their origin remains a topic of debate. In order to thoroughly investigate variations in spectral properties on the QPO timescale, we utilized the Hilbert-Huang transform technique to conduct phase-resolved spectroscopy across a broad energy band for LFQPOs in the newly discovered BHXRB Swift J1727.8-1613. This is achieved through quasi-simultaneous observations from Neutron star Interior Composition ExploreR (NICER), Nuclear Spectroscopic Telescope ARray (NuSTAR), and Hard X-ray Modulation Telescope (Insight-HXMT). Our analysis reveals that both the non-thermal and disk-blackbody components exhibit variations on the QPO timescale, with the former dominating the QPO variability. For the spectral parameters, we observe modulation of the disk temperature, spectral indices, and reflection fraction with the QPO phase with high statistical significance (>5σ). Notably, the variation in the disk temperature is found to precede the variations in the non-thermal and disk fluxes by ~0.4-0.5 QPO cycles. We suggest that these findings offer further evidence that the type-C QPO variability is a result of geometric effects of the accretion flow.

astro-ph.HE↗

arXiv2024

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-bench, GSM8K, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance. Code is available at https://github.com/Outsider565/LoRA-GA.

cs.LG↗

arXiv2024

Trade When Opportunity Comes: Price Movement Forecasting via Locality-Aware Attention and Iterative Refinement Labeling

Price movement forecasting, aimed at predicting financial asset trends based on current market information, has achieved promising advancements through machine learning (ML) methods. Most existing ML methods, however, struggle with the extremely low signal-to-noise ratio and stochastic nature of financial data, often mistaking noises for real trading signals without careful selection of potentially profitable samples. To address this issue, we propose LARA, a novel price movement forecasting framework with two main components: Locality-Aware Attention (LA-Attention) and Iterative Refinement Labeling (RA-Labeling). (1) LA-Attention, enhanced by metric learning techniques, automatically extracts the potentially profitable samples through masked attention scheme and task-specific distance metrics. (2) RA-Labeling further iteratively refines the noisy labels of potentially profitable samples, and combines the learned predictors robust to the unseen and noisy samples. In a set of experiments on three real-world financial markets: stocks, cryptocurrencies, and ETFs, LARA significantly outperforms several machine learning based methods on the Qlib quantitative investment platform. Extensive ablation studies confirm LARA's superior ability in capturing more reliable trading opportunities.

cs.LG↗

arXiv2024

Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates

With the increasing demand for large-scale training of machine learning models, fully decentralized optimization methods have recently been advocated as alternatives to the popular parameter server framework. In this paradigm, each worker maintains a local estimate of the optimal parameter vector, and iteratively updates it by waiting and averaging all estimates obtained from its neighbors, and then corrects it on the basis of its local dataset. However, the synchronization phase is sensitive to stragglers. An efficient way to mitigate this effect is to consider asynchronous updates, where each worker computes stochastic gradients and communicates with other workers at its own pace. Unfortunately, fully asynchronous updates suffer from staleness of stragglers' parameters. To address these limitations, we propose a fully decentralized algorithm DSGD-AAU with adaptive asynchronous updates via adaptively determining the number of neighbor workers for each worker to communicate with. We show that DSGD-AAU achieves a linear speedup for convergence and demonstrate its effectiveness via extensive experiments.

cs.LG↗

arXiv2024

Trichotomy for the orbits of a hypercyclic operator on a Banach space

We obtain a trichotomy for the orbits of a hypercyclic operator $T$ on a separable Banach space $X$: (1) every vector is mean asymptotic to zero; (2) generic vectors are absolutely mean irregular; (3) every hypercyclic vector is mean divergent to infinity. Examples of weighted backward shifts on $\ell^p$ show that all three cases can happen.

math.FA↗

arXiv2024

Glass formation in mechanically interlocked ring polymers: the role of induced chain stiffness

Polymer-related materials exhibit rich glassy behaviors at different length scales due to their various molecular structures and topological constraints. Recent studies have identified transient interpenetration of the long-chain rings contributing to dynamic arrest on the center-of-mass level. Interpenetration of rings is proposed as an approach to facilitate glass formation in polymer melts. In this work, inspired by recent advances in the synthesis of mechanically interlocked polymers, we investigate glass transition on the nanometer-scale segments influenced by permanent interpenetration of rings using molecular dynamics simulations. We find that decreasing chain length in the mechanically interlocked system is equivalent to inducing an effective chain stiffness on the sub-rings. The induced stiffness provides a unified explanation for these unique structural features and transient dynamic arrest in the system of interlocked rings with rather short chains. Further, a crossover is observed in the scaling relation between localization and glassy depth upon cooling. Our work reveals a dynamic transition from weak to strong caging at the crossover temperature. According to the localization model, we demonstrate that the chain stiffness increases the critical temperature and oscillation distance, therefore leads to more fragile dynamics and deeper glassy state. These findings are consistent with the predictions of molecular simulations and theories for polymers with real local stiffness. Our work deepens the understanding of the role of induced stiffness on glass transition, and opens up a new direction to design rich glass materials by manipulating stiffness through mechanical bonds.

cond-mat.soft↗

arXiv2024

Evaluation of Bias Towards Medical Professionals in Large Language Models

This study evaluates whether large language models (LLMs) exhibit biases towards medical professionals. Fictitious candidate resumes were created to control for identity factors while maintaining consistent qualifications. Three LLMs (GPT-4, Claude-3-haiku, and Mistral-Large) were tested using a standardized prompt to evaluate resumes for specific residency programs. Explicit bias was tested by changing gender and race information, while implicit bias was tested by changing names while hiding race and gender. Physician data from the Association of American Medical Colleges was used to compare with real-world demographics. 900,000 resumes were evaluated. All LLMs exhibited significant gender and racial biases across medical specialties. Gender preferences varied, favoring male candidates in surgery and orthopedics, while preferring females in dermatology, family medicine, obstetrics and gynecology, pediatrics, and psychiatry. Claude-3 and Mistral-Large generally favored Asian candidates, while GPT-4 preferred Black and Hispanic candidates in several specialties. Tests revealed strong preferences towards Hispanic females and Asian males in various specialties. Compared to real-world data, LLMs consistently chose higher proportions of female and underrepresented racial candidates than their actual representation in the medical workforce. GPT-4, Claude-3, and Mistral-Large showed significant gender and racial biases when evaluating medical professionals for residency selection. These findings highlight the potential for LLMs to perpetuate biases and compromise healthcare workforce diversity if used without proper bias mitigation strategies.

cs.CY↗

arXiv2024

Nuclear mass predictions based on convolutional neural network

A convolutional neural network (CNN) is employed to investigate nuclear mass. By introducing the masses of neighboring nuclei and the paring effects at the input layer of the network, local features of the target nucleus are extracted to predict its mass. Then, through learning the differences between the experimental nuclear masses and the predicted nuclear masses by the WS4 model, a new global-local model (CNN-WS4) is developed, which incorporates both the global nuclear mass model and local features. Due to the incorporation of local features, the CNN-WS4 model achieves high accuracy on the training set. When extrapolating for newly emerged nuclei, the CNN-WS4 also exhibits appreciable stability, thereby demonstrating its robustness.

nucl-th↗

arXiv2024

PCIE_EgoHandPose Solution for EgoExo4D Hand Pose Challenge

This report presents our team's 'PCIE_EgoHandPose' solution for the EgoExo4D Hand Pose Challenge at CVPR2024. The main goal of the challenge is to accurately estimate hand poses, which involve 21 3D joints, using an RGB egocentric video image provided for the task. This task is particularly challenging due to the subtle movements and occlusions. To handle the complexity of the task, we propose the Hand Pose Vision Transformer (HP-ViT). The HP-ViT comprises a ViT backbone and transformer head to estimate joint positions in 3D, utilizing MPJPE and RLE loss function. Our approach achieved the 1st position in the Hand Pose challenge with 25.51 MPJPE and 8.49 PA-MPJPE. Code is available at https://github.com/KanokphanL/PCIE_EgoHandPose

cs.CV↗

arXiv2024

Tunable non-Hermitian skin effect via gain and loss

We investigate theoretically tunable non-Hermitian skin effect in systems with gain and loss, and find that bipolar (quadripolar) non-Hermitian skin effect characterized by topological invariants in one (two)-dimensional system. We also find the partial non-Hermitian skin effect with the coexistence of localized states and extended states. Both types of the non-Hermitian skin effect have not yet been predicted together in a single system. A feasible experimental scheme of our model is proposed to realize in electric circuits. Our investigation unveils a new type of non-Hermitian skin effect and enhance the tunability of the non-Hermitian systems by gain and loss other than the conventional non-reciprocal hopping.

physics.app-ph↗