arXiv subjects

Bo Yuan

Publications and source records attributed to Bo Yuan.

At least 73 records · Page 4Linked to original sources

arXiv2025

EcoSpa: Efficient Transformer Training with Coupled Sparsity

Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges. While sparse training offers efficiency gains, existing methods fail to preserve critical structural relationships between weight matrices that interact multiplicatively in attention and feed-forward layers. This oversight leads to performance degradation at high sparsity levels. We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs, preserving their interaction patterns through aligned row/column removal. EcoSpa introduces a new granularity for calibrating structural component importance and performs coupled estimation and sparsification across both pre-training and fine-tuning scenarios. Evaluations demonstrate substantial improvements: EcoSpa enables efficient training of LLaMA-1B with 50\% memory reduction and 21\% faster training, achieves $2.2\times$ model compression on GPT-2-Medium with $2.4$ lower perplexity, and delivers $1.6\times$ inference speedup. The approach uses standard PyTorch operations, requiring no custom hardware or kernels, making efficient transformer training accessible on commodity hardware.

cs.LG↗

arXiv2025

Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning

Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.

cs.CL↗

arXiv2025

Resonant inelastic X-ray scattering investigation of Hund's and spin-orbit coupling in $5d^2$ double perovskites

B site ordered $5d^2$ double perovskites ($\mathrm{A_2BB'O_6,\ B'}=5d^2)$ display a remarkable range of physical properties upon variation of the chosen B and $\mathrm{B'}$ site ions. This sensitivity to chemical substitution reflects the delicate balance and profound impact of strong electronic correlation and spin-orbit coupling in such systems. We present rhenium $L_2$ and $L_3$ resonant inelastic X-ray scattering (RIXS) measurements of two such physically dissimilar materials, Mott-insulating $\mathrm{Ba_2YReO_6}$ and semiconducting $\mathrm{Sr_2CrReO_6}$. Despite these differences, our RIXS results reveal similar energy scales of Hund's ($J_H$) and spin-orbit coupling ($ζ$) in the two materials, with both systems firmly in the intermediate Hund's coupling regime where $J_H/ζ\sim 1$. However, there are clear differences in their RIXS spectra. The conductive character of $\mathrm{Sr_2CrReO_6}$ broadens and obfuscates the atomic transitions within an electron-hole continuum, while the insulating character of $\mathrm{Ba_2YReO_6}$ results in sharp atomic excitations. This contrast in their RIXS spectra despite their similar energy scales reflects a difference in the itinerancy-promoting hopping integral and illustrates the impact of the local crystal environment in double perovskites. Finally, $L_2$ and $L_3$ edge analyses of the atomic excitations in $\mathrm{Ba_2YReO_6}$ reveal that the ordering of the low lying excited states is inverted compared to previous reports, such that the appropriate energy scales of Hund's and spin-orbit coupling are significantly modified. We present exact diagonalization calculations of the RIXS spectra at both edges which show good agreement with our results for new energy scales of $ζ=0.290(5)$ eV and $J_H=0.38(2)$ eV ($J_H/ζ=1.30(5)$).

cond-mat.str-el↗

arXiv2025

Expanding Horizons of Level Diversity via Multi-objective Evolutionary Learning

In recent years, the generation of diverse game levels has gained increasing interest, contributing to a richer and more engaging gaming experience. A number of level diversity metrics have been proposed in literature, which are naturally multi-dimensional, leading to conflicted, complementary, or both relationships among these dimensions. However, existing level generation approaches often fail to comprehensively assess diversity across those dimensions. This paper aims to expand horizons of level diversity by considering multi-dimensional diversity when training generative models. We formulate the model training as a multi-objective learning problem, where each diversity metric is treated as a distinct objective. Furthermore, a multi-objective evolutionary learning framework that optimises multiple diversity metrics simultaneously throughout the model training process is proposed. Our case study on the commonly used benchmark Super Mario Bros. demonstrates that our proposed framework can enhance multi-dimensional diversity and identify a Pareto front of generative models, which provides a range of tradeoffs among playability and two representative diversity metrics, including a content-based one and a player-centered one. Such capability enables decision-makers to make informed choices when selecting generators accommodating a variety of scenarios and the diverse needs of players and designers.

cs.LG↗

arXiv2025

Generative AI as a Tool for Enhancing Reflective Learning in Students

Reflection is widely recognized as a cornerstone of student development, fostering critical thinking, self-regulation, and deep conceptual understanding. Traditionally, reflective skills have been cultivated through structured feedback, mentorship, and guided self-assessment. However, these approaches often face challenges such as limited scalability, difficulties in delivering individualized feedback, and a shortage of instructors proficient in facilitating meaningful reflection. This study pioneers the use of generative AI, specifically large language models (LLMs), as an innovative solution to these limitations. By leveraging the capacity of LLMs to deliver personalized, context-sensitive feedback at scale, this research investigates their potential to serve as effective facilitators of reflective exercises, sustaining deep engagement and promoting critical thinking. Through in-depth analyses of prompt engineering strategies and simulated multi-turn dialogues grounded in a project-based learning (PBL) context, the study demonstrates that, with pedagogically aligned prompts, LLMs can serve as accessible and adaptive tools for scalable reflective guidance. Furthermore, LLM-assisted evaluation is employed to objectively assess the performance of both tutors and students across multiple dimensions of reflective learning. The findings contribute to the evolving understanding of AI's role in reflective pedagogy and point to new opportunities for advancing AI-driven intelligent tutoring systems.

cs.HC↗

arXiv2025

Ethical Considerations of Large Language Models in Game Playing

Large language models (LLMs) have demonstrated tremendous potential in game playing, while little attention has been paid to their ethical implications in those contexts. This work investigates and analyses the ethical considerations of applying LLMs in game playing, using Werewolf, also known as Mafia, as a case study. Gender bias, which affects game fairness and player experience, has been observed from the behaviour of LLMs. Some roles, such as the Guard and Werewolf, are more sensitive than others to gender information, presented as a higher degree of behavioural change. We further examine scenarios in which gender information is implicitly conveyed through names, revealing that LLMs still exhibit discriminatory tendencies even in the absence of explicit gender labels. This research showcases the importance of developing fair and ethical LLMs. Beyond our research findings, we discuss the challenges and opportunities that lie ahead in this field, emphasising the need for diving deeper into the ethical implications of LLMs in gaming and other interactive domains.

cs.CL↗

arXiv2025

The nonisospectral integrable hierarchies associated with Lie algebra $\mathfrak{sp}(6)$

In this paper, we consider nonisospectral problems of two distinct dimensions on the loop algebra of the symplectic Lie algebra $\mathfrak{sp}(6)$, and construct two integrable systems. Furthermore, we derive their Hamiltonian structures using the Tu scheme. Additionally, we construct an integrable hierarchy on the generalized Lie algebra $\mathfrak{Gsp}(6)$ and establish its Hamiltonian structure as well.

nlin.SI↗

arXiv2025

CodeEdu: A Multi-Agent Collaborative Platform for Personalized Coding Education

Large Language Models (LLMs) have demonstrated considerable potential in improving coding education by providing support for code writing, explanation, and debugging. However, existing LLM-based approaches generally fail to assess students' abilities, design learning plans, provide personalized material aligned with individual learning goals, and enable interactive learning. Current work mostly uses single LLM agents, which limits their ability to understand complex code repositories and schedule step-by-step tutoring. Recent research has shown that multi-agent LLMs can collaborate to solve complicated problems in various domains like software engineering, but their potential in the field of education remains unexplored. In this work, we introduce CodeEdu, an innovative multi-agent collaborative platform that combines LLMs with tool use to provide proactive and personalized education in coding. Unlike static pipelines, CodeEdu dynamically allocates agents and tasks to meet student needs. Various agents in CodeEdu undertake certain functions specifically, including task planning, personalized material generation, real-time QA, step-by-step tutoring, code execution, debugging, and learning report generation, facilitated with extensive external tools to improve task efficiency. Automated evaluations reveal that CodeEdu substantially enhances students' coding performance.

cs.MA↗

arXiv2025

Soliton hierarchies associated with Lie algebra sp(6)

In this paper, by selecting appropriate spectral matrices within the loop algebra of symplectic Lie algebra sp(6), we construct two distinct classes of integrable soliton hierarchies. Then, by employing the Tu scheme and trace identity, we derive the Hamiltonian structures of the aforementioned two classes of integrable systems. From these two classes of integrable soliton hierarchies, we select one particular hierarchy and employ the Kronecker product to construct an integrable coupling system.

nlin.SI↗

arXiv2025

When Large Language Models Meet Vector Databases: A Survey

This survey explores the synergistic potential of Large Language Models (LLMs) and Vector Databases (VecDBs), a burgeoning but rapidly evolving research area. With the proliferation of LLMs comes a host of challenges, including hallucinations, outdated knowledge, prohibitive commercial application costs, and memory issues. VecDBs emerge as a compelling solution to these issues by offering an efficient means to store, retrieve, and manage the high-dimensional vector representations intrinsic to LLM operations. Through this nuanced review, we delineate the foundational principles of LLMs and VecDBs and critically analyze their integration's impact on enhancing LLM functionalities. This discourse extends into a discussion on the speculative future developments in this domain, aiming to catalyze further research into optimizing the confluence of LLMs and VecDBs for advanced data handling and knowledge extraction capabilities.

cs.DB↗

arXiv2025

Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning

Dynamic material handling (DMH) involves the assignment of dynamically arriving material transporting tasks to suitable vehicles in real time for minimising makespan and tardiness. In real-world scenarios, historical task records are usually available, which enables the training of a decision policy on multiple instances consisting of historical records. Recently, reinforcement learning has been applied to solve DMH. Due to the occurrence of dynamic events such as new tasks, adaptability is highly required. Solving DMH is challenging since constraints including task delay should be satisfied. A feedback is received only when all tasks are served, which leads to sparse reward. Besides, making the best use of limited computational resources and historical records for training a robust policy is crucial. The time allocated to different problem instances would highly impact the learning process. To tackle those challenges, this paper proposes a novel adaptive constrained evolutionary reinforcement learning (ACERL) approach, which maintains a population of actors for diverse exploration. ACERL accesses each actor for tackling sparse rewards and constraint violation to restrict the behaviour of the policy. Moreover, ACERL adaptively selects the most beneficial training instances for improving the policy. Extensive experiments on eight training and eight unseen test instances demonstrate the outstanding performance of ACERL compared with several state-of-the-art algorithms. Policies trained by ACERL can schedule the vehicles while fully satisfying the constraints. Additional experiments on 40 unseen noised instances show the robust performance of ACERL. Cross-validation further presents the overall effectiveness of ACREL. Besides, a rigorous ablation study highlights the coordination and benefits of each ingredient of ACERL.

cs.NE↗

arXiv2025

Anomalous properties of spark plasma sintered boron nitride solids

Hexagonal boron nitride (h-BN) is a brittle ceramic with a layered structure, however, recent experiments have suggested that inter-layer structural engineering could be key to new structural and functional properties. Here we report the scalable bulk synthesis of high-density crystalline h-BN solids, by using high-temperature spark plasma sintering (SPS) of h-BN powders, which show high values of mechanical strength, ductility, dielectric constant, thermal conductivity, and exceptional neutron radiation shielding capability. Through exhaustive characterizations we reveal that SPS induces non-basal plane crystallinity, twisting of layers, and facilitates inter-grain fusion with a high degree of in-plane alignment across macroscale dimensions, resulting in near-theoretical density and improved properties. Our findings highlight the importance of material design, via new approaches such as layer twisting and interlayer interconnections, to create novel ceramics with properties that could go beyond their intrinsic limits.

cond-mat.mtrl-sci↗

arXiv2025

EdgeOL: Efficient in-situ Online Learning on Edge Devices

Emerging applications, such as robot-assisted eldercare and object recognition, generally employ deep learning neural networks (DNNs) and naturally require: i) handling streaming-in inference requests and ii) adapting to possible deployment scenario changes. Online model fine-tuning is widely adopted to satisfy these needs. However, an inappropriate fine-tuning scheme could involve significant energy consumption, making it challenging to deploy on edge devices. In this paper, we propose EdgeOL, an edge online learning framework that optimizes inference accuracy, fine-tuning execution time, and energy efficiency through both inter-tuning and intra-tuning optimizations. Experimental results show that, on average, EdgeOL reduces overall fine-tuning execution time by 64%, energy consumption by 52%, and improves average inference accuracy by 1.75% over the immediate online learning strategy

cs.LG↗

arXiv2025

Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance

Learning from noisy labels (LNL) is a challenge that arises in many real-world scenarios where collected training data can contain incorrect or corrupted labels. Most existing solutions identify noisy labels and adopt active learning to query human experts on them for denoising. In the era of large language models (LLMs), although we can reduce the human effort to improve these methods, their performances are still subject to accurately separating the clean and noisy samples from noisy data. In this paper, we propose an innovative collaborative learning framework NoiseAL based on active learning to combine LLMs and small models (SMs) for learning from noisy labels. During collaborative training, we first adopt two SMs to form a co-prediction network and propose a dynamic-enhanced threshold strategy to divide the noisy data into different subsets, then select the clean and noisy samples from these subsets to feed the active annotator LLMs to rectify noisy samples. Finally, we employ different optimization objectives to conquer subsets with different degrees of label noises. Extensive experiments on synthetic and real-world noise datasets further demonstrate the superiority of our framework over state-of-the-art baselines.

cs.LG↗

arXiv2025

TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model

Vision-Language Models (VLMs) demand substantial computational resources during inference, largely due to the extensive visual input tokens for representing visual information. Previous studies have noted that visual tokens tend to receive less attention than text tokens, suggesting their lower importance during inference and potential for pruning. However, their methods encounter several challenges: reliance on greedy heuristic criteria for token importance and incompatibility with FlashAttention and KV cache. To address these issues, we introduce \textbf{TopV}, a compatible \textbf{TO}ken \textbf{P}runing with inference Time Optimization for fast and low-memory \textbf{V}LM, achieving efficient pruning without additional training or fine-tuning. Instead of relying on attention scores, we formulate token pruning as an optimization problem, accurately identifying important visual tokens while remaining compatible with FlashAttention. Additionally, since we only perform this pruning once during the prefilling stage, it effectively reduces KV cache size. Our optimization framework incorporates a visual-aware cost function considering factors such as Feature Similarity, Relative Spatial Distance, and Absolute Central Distance, to measure the importance of each source visual token, enabling effective pruning of low-importance tokens. Extensive experiments demonstrate that our method outperforms previous token pruning methods, validating the effectiveness and efficiency of our approach.

cs.CV↗

arXiv2025

COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection

Training large-scale neural networks in vision, and multimodal domains demands substantial memory resources, primarily due to the storage of optimizer states. While LoRA, a popular parameter-efficient method, reduces memory usage, it often suffers from suboptimal performance due to the constraints of low-rank updates. Low-rank gradient projection methods (e.g., GaLore, Flora) reduce optimizer memory by projecting gradients and moment estimates into low-rank spaces via singular value decomposition or random projection. However, they fail to account for inter-projection correlation, causing performance degradation, and their projection strategies often incur high computational costs. In this paper, we present COAP (Correlation-Aware Gradient Projection), a memory-efficient method that minimizes computational overhead while maintaining training performance. Evaluated across various vision, language, and multimodal tasks, COAP outperforms existing methods in both training speed and model performance. For LLaMA-1B, it reduces optimizer memory by 61% with only 2% additional time cost, achieving the same PPL as AdamW. With 8-bit quantization, COAP cuts optimizer memory by 81% and achieves 4x speedup over GaLore for LLaVA-v1.5-7B fine-tuning, while delivering higher accuracy.

cs.LG↗

arXiv2025

Label Distribution Learning-Enhanced Dual-KNN for Text Classification

Many text classification methods usually introduce external information (e.g., label descriptions and knowledge bases) to improve the classification performance. Compared to external information, some internal information generated by the model itself during training, like text embeddings and predicted label probability distributions, are exploited poorly when predicting the outcomes of some texts. In this paper, we focus on leveraging this internal information, proposing a dual $k$ nearest neighbor (D$k$NN) framework with two $k$NN modules, to retrieve several neighbors from the training set and augment the distribution of labels. For the $k$NN module, it is easily confused and may cause incorrect predictions when retrieving some nearest neighbors from noisy datasets (datasets with labeling errors) or similar datasets (datasets with similar labels). To address this issue, we also introduce a label distribution learning module that can learn label similarity, and generate a better label distribution to help models distinguish texts more effectively. This module eases model overfitting and improves final classification performance, hence enhancing the quality of the retrieved neighbors by $k$NN modules during inference. Extensive experiments on the benchmark datasets verify the effectiveness of our method.

cs.CL↗

arXiv2025

HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading

Transformer-based large language models (LLMs) demonstrate impressive performance in long context generation. Extending the context length has disproportionately shifted the memory footprint of LLMs during inference to the key-value cache (KV cache). In this paper, we propose HEADINFER, which offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU. HEADINFER employs a fine-grained, head-wise offloading strategy, maintaining only selective attention heads KV cache on the GPU while computing attention output dynamically. Through roofline analysis, we demonstrate that HEADINFER maintains computational efficiency while significantly reducing memory footprint. We evaluate HEADINFER on the Llama-3-8B model with a 1-million-token sequence, reducing the GPU memory footprint of the KV cache from 128 GB to 1 GB and the total GPU memory usage from 207 GB to 17 GB, achieving a 92% reduction compared to BF16 baseline inference. Notably, HEADINFER enables 4-million-token inference with an 8B model on a single consumer GPU with 24GB memory (e.g., NVIDIA RTX 4090) without approximation methods.

cs.LG↗