arXiv subjects

Zhiyuan Li

Publications and source records attributed to Zhiyuan Li.

At least 127 records · Page 7Linked to original sources

arXiv2025

Provable Benefits of Sinusoidal Activation for Modular Addition

This paper studies the role of activation functions in learning modular addition with two-layer neural networks. We first establish a sharp expressivity gap: sine MLPs admit width-$2$ exact realizations for any fixed length $m$ and, with bias, width-$2$ exact realizations uniformly over all lengths. In contrast, the width of ReLU networks must scale linearly with $m$ to interpolate, and they cannot simultaneously fit two lengths with different residues modulo $p$. We then provide a novel Natarajan-dimension generalization bound for sine networks, yielding nearly optimal sample complexity $\widetilde{\mathcal{O}}(p)$ for ERM over constant-width sine networks. We also derive width-independent, margin-based generalization for sine networks in the overparametrized regime and validate it. Empirically, sine networks generalize consistently better than ReLU networks across regimes and exhibit strong length extrapolation.

cs.LG↗

arXiv2025

Improving Robotic Manipulation Robustness via NICE Scene Surgery

Learning robust visuomotor policies for robotic manipulation remains a challenge in real-world settings, where visual distractors can significantly degrade performance and safety. In this work, we propose an effective and scalable framework, Naturalistic Inpainting for Context Enhancement (NICE). Our method minimizes out-of-distribution (OOD) gap in imitation learning by increasing visual diversity through construction of new experiences using existing demonstrations. By utilizing image generative frameworks and large language models, NICE performs three editing operations, object replacement, restyling, and removal of distracting (non-target) objects. These changes preserve spatial relationships without obstructing target objects and maintain action-label consistency. Unlike previous approaches, NICE requires no additional robot data collection, simulator access, or custom model training, making it readily applicable to existing robotic datasets. Using real-world scenes, we showcase the capability of our framework in producing photo-realistic scene enhancement. For downstream tasks, we use NICE data to finetune a vision-language model (VLM) for spatial affordance prediction and a vision-language-action (VLA) policy for object manipulation. Our evaluations show that NICE successfully minimizes OOD gaps, resulting in over 20% improvement in accuracy for affordance prediction in highly cluttered scenes. For manipulation tasks, success rate increases on average by 11% when testing in environments populated with distractors in different quantities. Furthermore, we show that our method improves visual robustness, lowering target confusion by 6%, and enhances safety by reducing collision rate by 7%.

cs.RO↗

arXiv2025

Distracted Robot: How Visual Clutter Undermine Robotic Manipulation

In this work, we propose an evaluation protocol for examining the performance of robotic manipulation policies in cluttered scenes. Contrary to prior works, we approach evaluation from a psychophysical perspective, therefore we use a unified measure of clutter that accounts for environmental factors as well as the distractors quantity, characteristics, and arrangement. Using this measure, we systematically construct evaluation scenarios in both hyper-realistic simulation and real-world and conduct extensive experimentation on manipulation policies, in particular vision-language-action (VLA) models. Our experiments highlight the significant impact of scene clutter, lowering the performance of the policies, by as much as 34% and show that despite achieving similar average performance across the tasks, different VLA policies have unique vulnerabilities and a relatively low agreement on success scenarios. We further show that our clutter measure is an effective indicator of performance degradation and analyze the impact of distractors in terms of their quantity and occluding influence. At the end, we show that finetuning on enhanced data, although effective, does not equally remedy all negative impacts of clutter on performance.

cs.RO↗

arXiv2025

On-chip electro-optically tunable narrow linewidth Brillouin microlasers implemented in thin film lithium niobate

On-chip narrow linewidth microlasers with real-time wavelength tunability are highly desirable for various applications including precision metrology, quantum technology, and coherent information processing. Realizing such laser remains a challenge despite significant advances made by various groups in recent years [Nat. Commun. 13, 5344 (2022); Nature 615, 411 (2023); Appl. Phys. Lett. 124, 131101 (2024); Nat. Photonics 13, 60 (2019)]. In this work, we overcome these hurdles and demonstrate on-chip electro-optically tunable Brillouin microlasers in compact lithium niobate on insulator (LNOI) microdisks with diameters of 31.5 um and 117.0 um by using cross-polarized SBS arrangement. A quasi-continuum band of bound shear mechanical modes inside the suspended microdisk are revealed for the first time, allowing feasible phase matching of stimulated Brillouin lasing (SBL). We achieve efficient cross-polarized optomechanical coupling and SBL via the significant photoelastic tensors of lithium niobate (e.g., p41=-1.51). This approach yields a 118 Hz intrinsic linewidth and a comparatively low threshold power of 3.15 mW. A real-time electro-optic tuning of the cross-polarized Brillouin scheme with a tuning efficiency of ~93.1 kHz/V is also achieved, further showcasing potential of LNOI platform for next-generation tunable photonic systems.

physics.optics↗

arXiv2025

A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent

Adaptive optimizers can reduce to normalized steepest descent (NSD) when only adapting to the current gradient, suggesting a close connection between the two algorithmic families. A key distinction between their analyses, however, lies in the geometries, e.g., smoothness notions, they rely on. In the convex setting, adaptive optimizers are governed by a stronger adaptive smoothness condition, while NSD relies on the standard notion of smoothness. We extend the theory of adaptive smoothness to the nonconvex setting and show that it precisely characterizes the convergence of adaptive optimizers. Moreover, we establish that adaptive smoothness enables acceleration of adaptive optimizers with Nesterov momentum in the convex setting, a guarantee unattainable under standard smoothness for certain non-Euclidean geometry. We further develop an analogous comparison for stochastic optimization by introducing adaptive gradient variance, which parallels adaptive smoothness and leads to dimension-free convergence guarantees that cannot be achieved under standard gradient variance for certain non-Euclidean geometry.

cs.LG↗

arXiv2025

MuISQA: Multi-Intent Retrieval-Augmented Generation for Scientific Question Answering

Complex scientific questions often entail multiple intents, such as identifying gene mutations and linking them to related diseases. These tasks require evidence from diverse sources and multi-hop reasoning, while conventional retrieval-augmented generation (RAG) systems are usually single-intent oriented, leading to incomplete evidence coverage. To assess this limitation, we introduce the Multi-Intent Scientific Question Answering (MuISQA) benchmark, which is designed to evaluate RAG systems on heterogeneous evidence coverage across sub-questions. In addition, we propose an intent-aware retrieval framework that leverages large language models (LLMs) to hypothesize potential answers, decompose them into intent-specific queries, and retrieve supporting passages for each underlying intent. The retrieved fragments are then aggregated and re-ranked via Reciprocal Rank Fusion (RRF) to balance coverage across diverse intents while reducing redundancy. Experiments on both MuISQA benchmark and other general RAG datasets demonstrate that our method consistently outperforms conventional approaches, particularly in retrieval accuracy and evidence coverage.

cs.AI↗

arXiv2025

On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond

Diffusion language models have recently emerged as a competitive alternative to autoregressive language models. Beyond next-token generation, they are more efficient and flexible by enabling parallel and any-order token generation. However, despite empirical successes, their computational power and fundamental limitations remain poorly understood. In this paper, we formally study whether non-autoregressive generation in Masked Diffusion Models (MDM) enables solving problems beyond the reach of Auto-Regressive Models (ARM). Our results show that MDM with sufficiently large context length is computationally universal with decoding steps matching the optimal parallel time complexity in PRAM. However, when controlling for other factors, MDM's flexibility to generate in any-order does not expand what ARM can already solve. To address this, we propose a new form of generation called any-process generation, which extends MDM with capabilities to remask, insert and delete tokens, allowing self-correction, length-variable editing, and adaptive parallelism. Theoretically and empirically, we demonstrate these capabilities enable scalability to significantly harder reasoning problems that are otherwise intractable for ARM and vanilla MDM. Additionally, they prove essential for generation tasks where objects naturally evolve through non-sequential processes, crucial for extending current LLMs beyond natural language to domains such as coding and science.

cs.LG↗

arXiv2025

Weak-to-Strong Generalization Even in Random Feature Networks, Provably

Weak-to-Strong Generalization (Burns et al., 2024) is the phenomenon whereby a strong student, say GPT-4, learns a task from a weak teacher, say GPT-2, and ends up significantly outperforming the teacher. We show that this phenomenon does not require a strong learner like GPT-4. We consider student and teacher that are random feature models, described by two-layer networks with a random and fixed bottom layer and a trained top layer. A "weak" teacher, with a small number of units (i.e. random features), is trained on the population, and a "strong" student, with a much larger number of units (i.e. random features), is trained only on labels generated by the weak teacher. We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the teacher. We also explain how such weak-to-strong generalization is enabled by early stopping. Importantly, we also show the quantitative limits of weak-to-strong generalization in this model.

cs.LG↗

arXiv2025

Optimal Control For Anti-Abeta Treatment in Alzheimer's Disease using a Reaction-Diffusion Model

Alzheimer's disease (AD) is a progressive neurodegenerative disorder that severely impairs survival and quality of life. While anti-amyloid beta (Abeta) therapies can slow disease progression, their efficacy depends on personalized dosing that maximizes benefits and minimizes risks such as amyloid related imaging abnormalities (ARIA). Mathematical modeling offers a powerful tool for understanding AD dynamics and optimizing treatment, yet most models focus solely on temporal behavior, overlooking spatial heterogeneity within the brain. In this study, we propose a spatially explicit reaction-diffusion model to describe Abeta plaque dynamics. We formulate an optimal control problem to minimize plaque concentration while balancing therapeutic efficacy and treatment risk. Under reasonable assumptions, we establish well-posedness and uniqueness of the optimal solution. A Finite Element Method (FEM) based numerical framework is developed to compute personalized treatment strategies. Our model is calibrated using longitudinal Abeta positron emission tomography (PET) data from the Alzheimer's Disease Neuroimaging Initiative (ADNI), enabling estimation of patient-specific parameters such as growth rate and effective diffusivity. Results show that optimized treatment strategies consistently outperform constant dosing regimens across patient groups, achieving substantial reductions in cumulative amyloid burden while minimizing side effects. This integrated, data-driven framework advances personalized, spatially informed therapeutic optimization for Alzheimer's disease.

math.OC↗

arXiv2025

Shift is Good: Mismatched Data Mixing Improves Test Performance

We consider training and testing on mixture distributions with different training and test proportions. We show that in many settings, and in some sense generically, distribution shift can be beneficial, and test performance can improve due to mismatched training proportions, even if the components are unrelated and with no transfer between components. In a variety of scenarios, we identify the optimal training proportions and the extent to which such distribution shift can be beneficial. We show how the same analysis applies also to a compositional setting with differing distribution of component "skills'' at training and test.

cs.LG↗

arXiv2025

Kimi Linear: An Expressive, Efficient Attention Architecture

We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.

cs.CL↗

arXiv2025

Radio signatures of AGN-wind-driven shocks in elliptical galaxies: From simulations to observations

We investigate the synchrotron emission signatures of shocks driven by active galactic nucleus (AGN) wind in elliptical galaxies based on our two-dimensional axisymmetric hydrodynamic MACER numerical simulations. Using these simulation data, we calculate the synchrotron radiation produced by nonthermal electrons accelerated at shocks, adopting reasonable assumptions for the magnetic field and relativistic electron distribution (derived from diffusive shock acceleration theory), and predict the resulting observational signatures. In our fiducial model, shocks driven by AGN winds produce synchrotron emission with luminosities of approximately $10^{29}\,\mathrm{erg\,s^{-1}\,Hz^{-1}}$ in the radio band (0.5-5 GHz), with spectral indices of $α\approx -0.4$ to $-0.6$ during the strongest shock phases, gradually steepening to about $-0.8$ to $-1.4$ as the electron population ages. Spatially, the emission is initially concentrated in regions of strong shocks, later expanding into more extended, diffuse structures. We also apply our model to the dwarf elliptical galaxy Messier 32 (M32), and find remarkable consistency between our simulated emission and the observed nuclear radio source, suggesting that this radio component likely originates from hot-wind-driven shocks. Our results indicate that AGN winds not only influence galaxy gas dynamics through mechanical energy input but also yield direct observational evidence via nonthermal radiation. With the advent of next-generation radio facilities such as the FAST Core Array, SKA, and ngVLA, these emission signatures serve as important probes for detecting and characterizing AGN feedback.

astro-ph.GA↗

arXiv2025

Projective Equivalence of Smooth Hypersurfaces via Cyclic Covers

In this paper, we prove that for any smooth hypersurface $Y$ of degree $d$ in $\mathbb{P}^{n+1}_k$, the cyclic $d$-fold cover $\widetilde{Y} \to \mathbb{P}^{n+1}_k$ branched along $Y$ completely characterizes $Y$ up to projective equivalence. This solves a question asked by Huybrechts in [Huy23, §1.5.6].

math.AG↗

arXiv2025

Inverse source problem with a posteriori interior measurements for space-time fractional diffusion equations

This paper investigates an inverse source problem for space-time fractional diffusion equations from a posteriori interior measurements. The uniqueness result is established by the memory effect of fractional derivatives and the unique continuation property. For the numerical reconstruction, the inverse problem is reformulated as an optimization problem with the Tikhonov regularization. We use the Levenberg-Marquardt method to identity the unknown source from noisy measurements. Finally, we give some numerical examples to illustrate the efficiency and accuracy of the proposed algorithm.

math.NA↗

arXiv2025

ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts

Mixture-of-Experts (MoE) architectures have emerged as a promising approach to scale Large Language Models (LLMs). MoE boosts the efficiency by activating a subset of experts per token. Recent works show that fine-grained experts substantially enriches the combinatorial flexibility of active experts and enhances model expressiveness. However, such a design is fundamentally limited by the layer-local routing mechanism: each layer is restricted to its own expert pool. This requires a careful trade-off between expert dimensionality and routing diversity given fixed parameter budgets. We describe ReXMoE, a novel MoE architecture that improves routing beyond the existing layer-local approaches by allowing routers to reuse experts across adjacent layers. ReXMoE decouples expert dimensionality from per-layer budgets, enabling richer expert combinations without sacrificing individual expert capacity or inflating overall parameters. To this end, we propose a new progressive scaling routing (PSR) strategy to gradually increase the candidate expert pool during training. As a result, ReXMoE improves both language modeling and downstream task performance. Extensive experiments on models ranging from 0.5B to 7B parameters across different architectures demonstrate that ReXMoE consistently improves performance under fixed architectural dimensions, confirming ReXMoE as new design paradigm for parameter-efficient and scalable MoE-based LLMs.

cs.CL↗

arXiv2025

Sharp decay estimates and numerical analysis for weakly coupled systems of two subdiffusion equations

This paper investigates the initial-boundary value problem for weakly coupled systems of time-fractional subdiffusion equations with spatially and temporally varying coupling coefficients. By combining the energy method with the coercivity of fractional derivatives, we convert the original partial differential equations into a coupled ordinary differential system. Through Laplace transform and maximum principle arguments, we reveal a dichotomy in decay behavior: When the highest fractional order is less than one, solutions exhibit sublinear decay, whereas systems with the highest order equal to one demonstrate a distinct superlinear decay pattern. This phenomenon fundamentally distinguishes coupled systems from single fractional diffusion equations, where such accelerated superlinear decay never occurs. Numerical experiments employing finite difference methods and implicit discretization schemes validate the theoretical findings.

math.AP↗

arXiv2025

Optimistic Multi-Agent Policy Gradient

*Relative overgeneralization* (RO) occurs in cooperative multi-agent learning tasks when agents converge towards a suboptimal joint policy due to overfitting to suboptimal behavior of other agents. No methods have been proposed for addressing RO in multi-agent policy gradient (MAPG) methods although these methods produce state-of-the-art results. To address this gap, we propose a general, yet simple, framework to enable optimistic updates in MAPG methods that alleviate the RO problem. Our approach involves clipping the advantage to eliminate negative values, thereby facilitating optimistic updates in MAPG. The optimism prevents individual agents from quickly converging to a local optimum. Additionally, we provide a formal analysis to show that the proposed method retains optimality at a fixed point. In extensive evaluations on a diverse set of tasks including the *Multi-agent MuJoCo* and *Overcooked* benchmarks, our method outperforms strong baselines on 13 out of 19 tested tasks and matches the performance on the rest.

cs.LG↗

arXiv2025

Generate Logical Equivalence Questions

Academic dishonesty is met with zero tolerance in higher education, yet plagiarism has become increasingly prevalent in the era of online teaching and learning. Automatic Question Generation (AQG) presents a potential solution to mitigate copying by creating unique questions for each student. Additionally, AQG can provide a vast array of practice questions. Our AQG focuses on generating logical equivalence questions for Discrete Mathematics, a foundational course for first-year computer science students. A literature review reveals that existing AQGs for this type of question generate all propositions that meet user-defined constraints, resulting in inefficiencies and a lack of uniform question difficulty. To address this, we propose a new approach that defines logical equivalence questions using a formal language, translates this language into two sets of generation rules, and develops a linear-time algorithm for question generation. We evaluated our AQG through two experiments. The first involved a group of students completing questions generated by our system. Statistical analysis shows that the accuracy of these questions is comparable to that of textbook questions. The second experiment assessed the number of steps required to solve our generated questions, textbook questions, and those generated by multiple large language models. The results indicated that the difficulty of our questions was similar to that of textbook questions, confirming the quality of our AQG.

cs.CL↗