arXiv subjects

Steven Li

Publications and source records attributed to Steven Li.

14 recordsLinked to original sources

arXiv2026

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

cs.LG↗

arXiv2026

JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogates

Quantization-aware training (QAT) is widely deployed but typically relies on the Straight-Through Estimator (STE), which passes gradients through non-differentiable quantizers by fiat. This often makes training brittle near bin boundaries and weakly aligned with the actual behavior of the low-precision model. We introduce JacQuant, a QAT framework that learns a lightweight surrogate of the model's local sensitivity to parameter changes and uses it to stabilize and accelerate training within standard variance-reduced optimizers. The surrogate is inexpensive (diagonal or block-diagonal), data-driven, and compatible with common weight and activation quantizers. On code-preserving training phases, we prove convergence for non-convex objectives and obtain linear rates under a PL condition, and we relate the learned sensitivity to end-to-end output fidelity via a simple calibration argument. Across LLM benchmarks at $\leq 2$ bits, JacQuant consistently reaches higher accuracy than STE-based QAT, and the runtime analyses on various models show that the added cost remains negligible under practical group sizes. The method is drop-in and requires no changes to the forward quantizers; our empirical claims are scoped to ultra-low-bit LLM QAT.

cs.LG↗

arXiv2026

WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points

Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is its slow convergence and early performance plateau, particularly below 4-bit-widths. While this problem has been observed in prior work, its precise cause remains unclear. In this paper, we analyze the convergence of QAT by estimating the spectrum of the loss-surface Hessians. We find that the weights converge to flat regions around saddle points, where a large fraction of the Hessian eigenvalues are both positive and negative. During training, an increasing fraction of Hessian eigenvalues concentrates around zero, whose magnitude decreases. At lower bit-widths, the magnitude of eigenvalues in the Hessian spectrum is significantly smaller. To mitigate these issues, we propose an algorithm called WinQ to accelerate QAT, which involves: (1) periodically resetting weights to the linear interpolation of full-precision and quantized weights, reducing the distance to the quantization grid and increasing eigenvalue magnitude, and (2) computing gradients of noise-injected weights to regularize the Hessian. Extensive experiments show that WinQ accelerates QAT by up to 4 times across various quantization methods and models. Under the same training cost, WinQ improves state-of-the-art sub-4-bit quantization by up to 8.8%. These results are consistent across 16 settings with different language models, quantization methods, and bit widths.

cs.LG↗

arXiv2026

Rethinking Model Efficiency: Multi-Agent Inference with Large Models

Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. However, different models may require vastly different numbers of output tokens to achieve comparable performance. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs on simulated data. The experiment shows that a large model with fewer output tokens can be more efficient than a small model with a long output sequence. The empirical study on diverse real-world benchmarks confirms the observation that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. The comparison on benchmark tasks demonstrates that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.

cs.CV↗

arXiv2026

Minimum Energy Cruise of All-Electric Aircraft with Applications to Advanced Air Mobility

Electrified propulsion is expected to play an important role in the sustainable development of Advanced Air Mobility (AAM). However, the limited energy density of batteries motivates the need to minimize energy consumption during flight. This paper studies the minimum total energy problem for an all-electric aircraft in steady cruise flight. The problem is formulated as an optimal control problem in which the cruise airspeed and final cruise time are optimization variables. The battery supply voltage is modeled as an affine function of the battery charge. Pontryagin's Minimum Principle is used to derive the necessary and sufficient conditions for optimality, from which closed-form expressions for the optimal cruise airspeed and optimal final cruise time are obtained. Additional analytical conditions are derived that determine when all-electric operation is feasible, one of which is that sufficient electric charge must be available. Numerical simulations based on the BETA Technologies CX300 all-electric aircraft and a representative AAM scenario illustrate how the aircraft weight, cruising altitude, electrical system efficiency, and initial battery charge influence the optimal airspeed and the feasibility of all-electric cruise.

eess.SY↗

arXiv2026

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30\% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

cs.LG↗

arXiv2026

CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill

The prefill stage in long-context LLM inference remains a computational bottleneck. Recent token-ranking heuristics accelerate inference by selectively processing a subset of semantically relevant tokens. However, existing methods suffer from unstable token importance estimation, often varying between layers. Evaluating token-ranking quality independently from heuristic-specific architectures is challenging. To address this, we introduce an Answer-Informed Oracle, which defines ground-truth token importance by measuring attention from generated answers back to the prompt. This oracle reveals that existing heuristics exhibit high variance across layers: rankings can degrade sharply at specific layers, a failure mode invisible to end-to-end benchmarks. The diagnosis suggests a simple fix: aggregate scores across layers rather than relying on any single one. We implement this as Cross-Layer Attention Aggregation (CLAA), which closes the gap to the oracle upper bound and reduces Time-to-First-Token (TTFT) by up to 39\% compared to the Full KV Cache baseline.

cs.CL↗

arXiv2026

The EXoplanet Climate Infrared TElescope (EXCITE): A balloon-borne mission to measure spectroscopic phase curves of transiting hot Jupiters

The EXoplanet Climate Infrared TElescope (EXCITE) is a balloon-borne mission dedicated to measuring spectroscopic phase curves of hot Jupiter-type exoplanets. Phase curve measurements can be used to characterize an exoplanet's longitude-dependent atmospheric composition and energy circulation patterns. EXCITE carries a 0.5 m primary mirror and moderate resolution diffraction-limited spectrograph with spectral coverage from 0.8--3.5 um. EXCITE is designed to fly from a long-duration balloon (LDB). EXCITE will observe through the peak of a target's spectral energy distribution (SED) and through spectral signatures of hydrogen and carbon-containing molecules. In this paper, we present the science goals of EXCITE, detail the as-built instrument, and discuss its performance during a 2024 engineering flight from Fort Sumner, New Mexico.

astro-ph.IM↗

arXiv2025

Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction

End-to-end (E2E) spoken dialogue systems are increasingly replacing cascaded pipelines for voice-based human-AI interaction, processing raw audio directly without intermediate transcription. Existing benchmarks primarily evaluate these models on synthetic speech and single-turn tasks, leaving realistic multi-turn conversational ability underexplored. We introduce Audio MultiChallenge, an open-source benchmark to evaluate E2E spoken dialogue systems under natural multi-turn interaction patterns. Building on the text-based MultiChallenge framework, which evaluates Inference Memory, Instruction Retention, and Self Coherence, we introduce a new axis Voice Editing that tests robustness to mid-utterance speech repairs and backtracking. We further augment each axis to the audio modality, such as introducing Audio-Cue challenges for Inference Memory that require recalling ambient sounds and paralinguistic signals beyond semantic content. We curate 452 conversations from 47 speakers with 1,712 instance-specific rubrics through a hybrid audio-native agentic and human-in-the-loop pipeline that exposes model failures at scale while preserving natural disfluencies found in unscripted human speech. Our evaluation of proprietary and open-source models reveals that even frontier models struggle on our benchmark, with Gemini 3 Pro Preview (Thinking), our highest-performing model achieving a 54.65% pass rate. Error analysis shows that models fail most often on our new axes and that Self Coherence degrades with longer audio context. These failures reflect difficulty of tracking edits, audio cues, and long-range context in natural spoken dialogue. Audio MultiChallenge provides a reproducible testbed to quantify them and drive improvements in audio-native multi-turn interaction capability.

cs.SD↗

arXiv2025

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due to their substantial model size, especially when deployed on edge devices. Activation sparsity offers a promising solution to reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applications. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy continual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-based methods. In this paper, we introduce R-Sparse, a training-free activation sparsity approach capable of achieving high sparsity levels in advanced LLMs. We conducted two preliminary investigations into how different components contribute to the output within a single linear layer and found two key observations: (i) the non-sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by an appropriate combination of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with a rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experiments on Llama-2/3 and Mistral models across ten diverse tasks demonstrate that R-Sparse achieves comparable performance at 50% model-level sparsity, resulting in a significant 43% end-to-end efficient improvements with customized kernels.

cs.LG↗

arXiv2024

Thermal architecture for a cryogenic super-pressure balloon payload: design and development of the Taurus flight cryostat

We describe the cryogenic system being developed for Taurus: a super-pressure balloon-borne microwave polarimeter scheduled to fly in 2027. The Taurus cryogenic system consists of a 660L liquid helium cryostat which achieves a base temperature of <100mK with the help of a capillary-fed superfluid tank and a closed cycle dilution refrigerator. The main tank is supported with fiberglass flexures and is encased in two layers of vapor-cooled shields which allow Taurus to make full use of the extended flight time offered by the super-pressure balloon platform. The Taurus cryostat is projected to hold for over 50 days while weighing under 1000lbs. We present the design, testing, and thermal analysis of the Taurus cryogenic systems.

physics.ins-det↗

arXiv2024

Instrument Overview of Taurus: A Balloon-borne CMB and Dust Polarization Experiment

Taurus is a balloon-borne cosmic microwave background (CMB) experiment optimized to map the E-mode polarization and Galactic foregrounds at the largest angular scales ($\ell$ $\lt$ 30) and improve measurements of the optical depth to reionization ($\tau$). This will pave the way for improved measurements of the sum of neutrino masses in combination with high-resolution CMB data while also testing the $\Lambda CDM$ model on large angular scales and providing high-frequency maps of polarized dust foregrounds to the CMB community. These measurements take advantage of the low-loading environment found in the stratosphere and are enabled by NASA's super-pressure balloon platform, which provides access to 70% of the sky with a launch from Wanaka, New Zealand. Here we describe a general overview of Taurus, with an emphasis on the instrument design. Taurus will employ more than 10,000 100 mK transition edge sensor bolometers distributed across two low-frequency (150, 220 GHz) and one high-frequency (280, 350 GHz) dichroic receivers. The liquid helium cryostat housing the detectors and optics is supported by a lightweight gondola. The payload is designed to meet the challenges in mass, power, and thermal control posed by the super-pressure platform. The instrument and scan strategy are optimized for rigorous control of instrumental systematics, enabling high-fidelity linear polarization measurements on the largest angular scales.

astro-ph.IM↗

arXiv2023

Optimal Cruise Airspeed for Hybrid-Electric and Electric Aircraft: Applications to Air Mobility

Electric and hybrid-electric aircraft can help our society transition towards more sustainable aviation and lower greenhouse gas (GHG) emissions. This paper provides solutions to minimize the direct operating cost (DOC) for hybrid-electric aircraft. The solution is the positive real root of a quintic polynomial which is derived using Pontryagin's minimum principle. By properly selecting a hybridization factor, one can also find the cruise airspeed corresponding to the minimum DOC of an electric aircraft. The optimal airspeed is integrated into the Rapidly-exploring Random Trees Star (RRT*) path planning algorithm. The minimum DOC solutions are investigated in a hybrid-electric international travel scenario and the path planning approach is applied to an electric aircraft city scenario.

eess.SY↗

arXiv2020

The Balloon-Borne Large Aperture Submillimeter Telescope Observatory

The BLAST Observatory is a proposed superpressure balloon-borne polarimeter designed for a future ultra-long duration balloon campaign from Wanaka, New Zealand. To maximize scientific output while staying within the stringent superpressure weight envelope, BLAST will feature new 1.8m off-axis optical system contained within a lightweight monocoque structure gondola. The payload will incorporate a 300L $^4$He cryogenic receiver which will cool 8,274 microwave kinetic inductance detectors (MKIDs) to 100mK through the use of an adiabatic demagnetization refrigerator (ADR) in combination with a $^3$He sorption refrigerator all backed by a liquid helium pumped pot operating at 2K. The detector readout utilizes a new Xilinx RFSOC-based system which will run the next-generation of the BLAST-TNG KIDPy software. With this instrument we aim to answer outstanding questions about dust dynamics as well as provide community access to the polarized submillimeter sky made possible by high-altitude observing unrestricted by atmospheric transmission. The BLAST Observatory is designed for a minimum 31-day flight of which 70$\%$ will be dedicated to observations for BLAST scientific goals and the remaining 30$\%$ will be open to proposals from the wider astronomical community through a shared-risk proposals program.

astro-ph.IM↗