arXiv subjects

Deyi Zhang

Publications and source records attributed to Deyi Zhang.

4 recordsLinked to original sources

arXiv2026

Decoding RR Lyrae light curves with deep learning for accurate absolute magnitude estimation

RR Lyrae stars are essential standard candles for distance measurements in the Milky Way and nearby galaxies. Traditional estimates rely on the Period--Absolute Magnitude--Metallicity relation but are limited by uncertainties in metallicity determinations. We present a deep learning approach that directly predicts absolute magnitudes from RRab and RRc light curves, eliminating the need for metallicity estimates. Our model achieves validation precisions of 0.053 mag and 0.036 mag (approximately 2.5% and 1.7% in distance) for individual RRab and RRc stars, respectively. Tests on globular clusters yield typical distance precisions of 1.0% for RRab and 1.7% for RRc stars. For the benchmark systems, combining the RRab- and RRc-based models yields distance moduli of 18.498 $\pm$ 0.001$_{\text{stat}}$ $\pm$ 0.018$_{\text{sys}}$ mag for the Large Magellanic Cloud and 19.564 $\pm$ 0.003$_{\text{stat}}$ $\pm$ 0.019$_{\text{sys}}$ mag for the Sculptor dwarf spheroidal galaxy. These measurements are in excellent agreement with previous results, achieve a distance precision of approximately 1%, and represent a 1.8-fold improvement over traditional RR Lyrae calibration relations. Our approach showcases the ability of AI to directly extract key physical parameters from complex, information-rich light curves, resolve degeneracies, and scale to broader applications.

astro-ph.GA↗

arXiv2025

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

The success of current Large-Language Models (LLMs) hinges on extensive training data that is collected and stored centrally, called Centralized Learning (CL). However, such a collection manner poses a privacy threat, and one potential solution is Federated Learning (FL), which transfers gradients, not raw data, among clients. Unlike traditional networks, FL for LLMs incurs significant communication costs due to their tremendous parameters. This study introduces an innovative approach to compress gradients to improve communication efficiency during LLM FL, formulating the new FL pipeline named CG-FedLLM. This approach integrates an encoder on the client side to acquire the compressed gradient features and a decoder on the server side to reconstruct the gradients. We also developed a novel training strategy that comprises Temporal-ensemble Gradient-Aware Pre-training (TGAP) to identify characteristic gradients of the target model and Federated AutoEncoder-Involved Fine-tuning (FAF) to compress gradients adaptively. Extensive experiments confirm that our approach reduces communication costs and improves performance (e.g., average 3 points increment compared with traditional CL- and FL-based fine-tuning with LlaMA on a well-recognized benchmark, C-Eval). This improvement is because our encoder-decoder, trained via TGAP and FAF, can filter gradients while selectively preserving critical features. Furthermore, we present a series of experimental analyses focusing on the signal-to-noise ratio, compression rate, and robustness within this privacy-centric framework, providing insight into developing more efficient and secure LLMs.

cs.LG↗

arXiv2024

DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately

The emergence of the Large Language Model (LLM) has shown their superiority in a wide range of disciplines, including language understanding and translation, relational logic reasoning, and even partial differential equations solving. The transformer is the pervasive backbone architecture for the foundation model construction. It is vital to research how to adjust the Transformer architecture to achieve an end-to-end privacy guarantee in LLM fine-tuning. In this paper, we investigate three potential information leakage during a federated fine-tuning procedure for LLM (FedLLM). Based on the potential information leakage, we provide an end-to-end privacy guarantee solution for FedLLM by inserting two-stage randomness. The first stage is to train a gradient auto-encoder with a Gaussian random prior based on the statistical information of the gradients generated by local clients. The second stage is to fine-tune the overall LLM with a differential privacy guarantee by adopting appropriate Gaussian noises. We show the efficiency and accuracy gains of our proposed method with several foundation models and two popular evaluation benchmarks. Furthermore, we present a comprehensive privacy analysis with Gaussian Differential Privacy (GDP) and Renyi Differential Privacy (RDP).

cs.LG↗

arXiv2024

Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning

The development of Large Language Models (LLMs) has significantly advanced various AI applications in commercial and scientific research fields, such as scientific literature summarization, writing assistance, and knowledge graph construction. However, a significant challenge is the high risk of hallucination during LLM inference, which can lead to security concerns like factual inaccuracies, inconsistent information, and fabricated content. To tackle this issue, it is essential to develop effective methods for reducing hallucination while maintaining the original capabilities of the LLM. This paper introduces a novel approach called Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination. This method modifies the representation layers of pre-trained LLMs by using contrastive `positive' and `negative' models, trained on data with and without hallucinations. By leveraging the differences between these two models, we create a more straightforward pathway to eliminate hallucinations, and the iterative nature of contrastive learning further enhances performance. Experimental validation on four pre-trained foundation LLMs (LLaMA2, Alpaca, LLaMA3, and Qwen) finetuning with a specially designed dataset shows that our approach achieves an average improvement of 10.1 points on the TruthfulQA benchmark. Comprehensive experiments demonstrate the effectiveness of Iter-AHMCL in reducing hallucination while maintaining the general capabilities of LLMs.

cs.CL↗