arXiv subjects

Masahiro Tanaka

Publications and source records attributed to Masahiro Tanaka.

At least 19 recordsLinked to original sources

arXiv2026

PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintains an incremental radix tree over the pending queue, exposing prefix-sharing clusters no existing engine surfaces. A low-overhead dual-walk matches the tree against the engine's prefix cache to yield longest-prefix-match for every waiting request; PEEK then admits cluster pioneers first so siblings inherit the freshly cached prefix, a co-designed eviction hook protects blocks ancestral to queued demand, and a multi-lane stride scheduler bounds starvation. On SGLang and vLLM across five workloads up to 4$\times$H100 (DP=2 over TP=2), PEEK delivers up to 3.0$\times$/2.6$\times$ cache hit, 7.9$\times$/7.1$\times$ TTFT, 6.7$\times$/5.5$\times$ E2E, and 3.6$\times$/4.5$\times$ throughput gains over each engine's strongest stock baseline (SGLang/vLLM), while matching baselines within noise on workloads with no exploitable prefix structure. Wins hold as KV-cache pressure and inference parallelism scale.

cs.DC↗

arXiv2026

CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training

In training Mixture-of-Experts (MoE) models, sharded data parallelism partitions each expert's parameters across GPUs, requiring an Allgather operation to reconstruct the full weight matrix before each layer executes. This communication often dominates iteration time. Prior work often reduces this overhead using lossy compression methods that sacrifice numerical fidelity, while existing lossless methods do not exploit cross-iteration exponent stability. In this paper, we propose Cross-Iteration Exponent Reuse Allgather (CIERA), a lossless, system-aware communication method for sharded MoE training. We observe that after a brief warmup phase, the exponent values of most weights remain unchanged across iterations. Based on this, we cache exponents locally and transmit only the sign and mantissa when exponents are unchanged. The receiver reconstructs the original weights exactly by combining the cached exponents with the received data. Since parameter matrices vary in shape across layers, compression is applied only when it yields net time savings. Moreover, the compression operation is overlapped with both Allgather communication and computation. Our real experiments and large-scale trace-driven simulator show that on OLMoE-1B-7B at 16 GPUs, CIERA achieves a 3.70x speedup over the lossless baseline and 3.68x over the lossy baseline, projected to reach 4.28x and 4.42x respectively at 128 GPUs, while preserving bitwise-exact parameter reconstruction in all evaluated runs.

cs.DC↗

arXiv2026

Curvature-Calibrated Quasi-Bayesian Updating for Moment-Restricted Models

Moment restrictions provide a flexible basis for quasi-Bayesian inference when a full likelihood is unavailable, but the weighting matrix in a quadratic moment criterion determines both the relative importance of the moments and the information scale of posterior updating. We propose curvature-calibrated quasi-Bayesian updating, which uses the inverse of the covariance (or long-run covariance) of the moment conditions evaluated at a self-consistent quasi-posterior center. The resulting fixed-point procedure alternates between covariance estimation and simulation from a fixed-weight quasi-posterior, thereby avoiding parameter-dependent weighting during each simulation run. Under a Bernstein-von Mises condition for the fixed-weight quasi-posterior at the efficient population weight, we show that the calibration map is locally contractive, that its fixed point is consistent at the standard parametric rate, and that the Gaussian approximation continues to hold under the calibrated data-dependent weight, with covariance given by the inverse Godambe information matrix. Under a uniform fourth-moment condition, the scaled quasi-posterior covariance converges to the same matrix, so quasi-posterior and repeated-sampling uncertainty agree to first order. Simulations show improved covariance calibration and interval coverage after a few updates. An application to longitudinal binary-response data illustrates the method with within-subject dependence and overidentified residual moments.

stat.ME↗

arXiv2026

Learning Bidirectional Causal Interactions with Heteroscedastic Neural Networks

Estimating contemporaneous bidirectional interactions from observational data is difficult because each outcome is endogenous to the other, while flexible regressions may capture only reduced-form dependence. This paper proposes SEM-DNN, a heteroscedastic neural simultaneous-equation estimator that learns reciprocal structural interactions without external instruments. Identification exploits conditional covariance diagonalization: when structural shocks have zero conditional means, are conditionally uncorrelated given predetermined covariates, and exhibit nonproportional conditional variances, only the true interaction coefficients diagonalize the conditional residual covariance across the feature space. The method jointly approximates nonlinear structural mean functions and feature-dependent variances using a diagonal Gaussian quasi-likelihood that incorporates the simultaneous-system Jacobian. We establish unique identification and positive-definite local curvature of the profiled population criterion and show that, under neural-profile compatibility conditions, the implemented neural criterion inherits this curvature despite nonunique network parameterizations. The coefficients admit a causal interpretation when the structural equations represent autonomous mechanisms that remain invariant under the relevant interventions. Monte Carlo experiments with nonlinear, high-dimensional nuisance functions and non-Gaussian shocks show that SEM-DNN recovers structural effects more reliably than parametric, kernel-based, and separate-equation neural alternatives as information increases, although at greater computational cost. An application to ready-to-eat cereal scanner data illustrates how the method can study contemporaneous price-sales feedback and assess identification strength, residual diagonalization, variance calibration, and optimization sensitivity.

stat.ML↗

arXiv2026

Shirakami: A Hybrid Concurrency Control Protocol for Tsurugi Relational Database System

Bill-of-materials and telecommunications billing applications need to process both short and long read-write transactions simultaneously. Recent work rarely addresses such evolving workloads. To deal with these workloads, we propose a new concurrency control protocol, Shirakami. Shirakami is a hybrid protocol. The first protocol, Shirakami-LTX, is for long read-write transactions based on multiversion view serializability. The second protocol, Shirakami-OCC, is for short transactions based on Silo. Shirakami naturally integrates them with write-preservation and epoch-based synchronization. It does not require dynamic protocol switching and provides stable performance. We implemented Shirakami as the transaction processing module of the Tsurugi system, which is a production-grade relational database system. The experimental results demonstrated that Tsurugi was at least 7.9x faster than PostgreSQL, and Shirakami-LTX exhibited 680x higher throughput than Shirakami-OCC.

cs.DB↗

arXiv2026

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.

cs.AI↗

arXiv2026

A Theory of Bootstrap Coverage Calibration for Generalized Posterior Credible Sets

Generalized posteriors replace the likelihood by an exponentiated empirical criterion, but their credible sets generally lack asymptotic justification for frequentist coverage. General posterior calibration selects a scalar learning rate by estimating coverage with the bootstrap. Using Edgeworth expansions under regular fixed-dimensional asymptotics, we derive higher-order coverage expansions and analyze the stochastic approximation step used in the implemented algorithm. For a fixed nominal level, the root of the bootstrap coverage equation is consistent under a uniform coverage approximation and local identification. The higher-order expansions separate two sources of coverage error: the sampling Edgeworth correction for the estimator and the posterior Edgeworth correction for credible set boundaries, centres, and shapes. A scalar learning rate can calibrate all nominal levels in the Gaussian limit only when the posterior covariance and the sampling covariance are proportional. Hence, bootstrap calibration is a level-specific scale correction, not a remedy for general shape misspecification.

stat.ME↗

arXiv2026

Delayed Acceptance Markov Chain Monte Carlo for Robust Bayesian Analysis

This study introduces a computationally efficient algorithm, delayed acceptance Markov chain Monte Carlo (DA-MCMC), designed to improve posterior simulation in quasi-Bayesian inference. Quasi-Bayesian methods, which do not require fully specifying a probabilistic model, are often computationally expensive owing to the need to evaluate the inverse and determinant of large covariance matrices. DA-MCMC addresses this challenge by employing a two-stage process: In the first stage, proposals are screened using an approximate posterior, whereas a final acceptance or rejection decision is made in the second stage based on the exact target posterior. This reduces the need for costly matrix computations, thereby improving efficiency without sacrificing accuracy. We demonstrate the effectiveness of DA-MCMC through applications to both synthetic and real data. The results demonstrate that, although DA-MCMC slightly reduces the effective sample size per iteration compared with the standard MCMC, it achieves substantial improvement in terms of effective sample size per second, approximately doubling the efficiency. This makes DA-MCMC particularly useful for cases where posterior simulation is computationally intensive. Thus, the DA-MCMC algorithm offers a significant advancement in computational efficiency for quasi-Bayesian inference, making it a valuable tool for robust Bayesian analysis.

stat.CO↗

arXiv2026

Estimating Bidirectional Causal Effects with Large Scale Online Kernel Learning

In this study, a scalable online kernel learning framework is proposed for estimating bidirectional causal effects in systems characterized by mutual dependence and heteroskedasticity. Traditional causal inference often focuses on unidirectional effects, overlooking the common bidirectional relationships in real-world phenomena. Building on heteroskedasticity-based identification, the proposed method integrates a quasi-maximum likelihood estimator for simultaneous equation models with large scale online kernel learning. It employs random Fourier feature approximations to flexibly model nonlinear conditional means and variances, while an adaptive online gradient descent algorithm ensures computational efficiency for streaming and high-dimensional data. Results from extensive simulations demonstrate that the proposed method achieves superior accuracy and stability than single equation and polynomial approximation baselines, exhibiting lower bias and root mean squared error across various data-generating processes. These results confirm that the proposed approach effectively captures complex bidirectional causal effects with near-linear computational scaling. By combining econometric identification with modern machine learning techniques, the proposed framework offers a practical, scalable, and theoretically grounded solution for large scale causal inference in natural/social science, policy making, business, and industrial applications.

stat.ML↗

arXiv2026

Generalized Posterior Calibration via Sequential Monte Carlo Sampler

As the amount and complexity of available data increases, the need for robust statistical learning becomes more pressing. To enhance resilience against model misspecification, the generalized posterior inference method adjusts the likelihood term by exponentiating it with a learning rate, thereby fine-tuning the dispersion of the posterior distribution. This study proposes a computationally efficient strategy for selecting an appropriate learning rate. The proposed approach builds upon the generalized posterior calibration (GPC) algorithm, which is designed to select a learning rate that ensures nominal frequentist coverage. This algorithm, which evaluates the coverage probability using bootstrap samples, has high computational costs because of the repeated posterior simulations needed for bootstrap samples. To address this limitation, the study proposes an algorithm that combines elements of the GPC algorithm with the sequential Monte Carlo (SMC) sampler. By leveraging the similarity between the learning rate in generalized posterior inference and the inverse temperature in SMC sampling, the proposed algorithm efficiently calibrates the posterior distribution with a reduced computational cost. For demonstration, the proposed algorithm was applied to several statistical learning models and shown to be significantly faster than the original GPC.

stat.CO↗

arXiv2026

Quasi-Bayesian Local Projections: Simultaneous Inference and Extension to the Instrumental Variable Method

Local projections (LPs) are widely used for impulse response analysis, but Bayesian methods face challenges due to the absence of a likelihood function. Existing approaches rely on pseudo-likelihoods, which often result in poorly calibrated posteriors. We propose a quasi-Bayesian method based on the Laplace-type estimator, where a quasi-likelihood is constructed using a generalized method of moments criterion. This approach avoids strict distributional assumptions, ensures well-calibrated inferences, and supports simultaneous credible bands. Additionally, it can be naturally extended to the instrumental variable method. We validate our approach through Monte Carlo simulations.

econ.EM↗

arXiv2026

Quasi-Bayesian Local Projection Instrumental-Variables Method: Application to Renewable Energy and Electricity Prices

This paper introduces a quasi-Bayesian approach for local projection instrumental-variables (LP-IV) estimation. It builds a moment-based quasi-posterior using the generalized method of moments (GMM) objective and applies a roughness-penalty prior to smooth impulse responses over different horizons. The approach maintains the key first-order features of traditional LP-IV methods, while enhancing stability in finite samples and allowing for joint inference through simultaneous bands. Simulations indicate that this regularization decreases root mean squared error compared to standard GMM, especially at medium and longer horizons. An application to Danish electricity markets highlights the method's practical usefulness.

econ.EM↗

arXiv2026

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism

Large-language-models (LLMs) demonstrate enormous utility in long-context tasks which require processing prompts that consist of tens to hundreds of thousands of tokens. However, existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training, instead focusing on optimizations for models with large parameter counts through ZeRO-3/FSDP, Tensor and Pipeline parallelism. This forces users to rewrite LLM training libraries to incorporate compositions of various complex long-context optimizations, such as sequence-parallelism, to training pipelines; a process that requires in-depth expertise, reducing developer productivity. To tackle these challenges, we introduce AutoSP: the first automated solution to automatically optimize LLM training for longer-contexts. AutoSP compiles models and applies a targeted set of optimizations: automated sequence parallelism, and long-context aware activation-checkpointing, to drastically enhance LLM trainability at negligible cost to throughput. Our evaluation demonstrates AutoSP's capability on both NVIDIA and AMD hardware, increasing training contexts by upto 2.7$\times$ and 2.5$\times$ respectively over competitive hand-written baseline at negligible cost to runtime performance.

cs.LG↗

arXiv2026

MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computation

Long-context decoding in LLMs is IO-bound: each token re-reads an ever-growing KV cache. Prior accelerations cut bytes via compression, which lowers fidelity, or selection/eviction, which restricts what remains accessible, and both can degrade delayed recall and long-form generation. We introduce MAC-Attention, a fidelity- and access-preserving alternative that accelerates decoding by reusing prior attention computations for semantically similar recent queries. It starts with a match stage that performs pre-RoPE L2 matching over a short local window; an amend stage rectifies the reused attention by recomputing a small band near the match boundary; and a complete stage fuses the rectified results with fresh attention computed on the KV tail through a numerically stable merge. On a match hit, the compute and bandwidth complexity is constant regardless of context length. The method is model-agnostic and composes with IO-aware kernels, paged-KV managers, and MQA/GQA. Across LongBench v2 (120K), RULER (120K), and LongGenBench (16K continuous generation), compared to the latest FlashInfer library, MAC-Attention reduces KV accesses by up to 99%, cuts token generation latency by over 60% at 128K, and achieves over 14.3x attention-phase speedups, up to 2.6x end-to-end, while maintaining full-attention quality. By reusing computation, MAC-Attention delivers long-context inference that is both fast and faithful. Code is available here: https://github.com/YJHMITWEB/MAC-Attention.git

cs.LG↗

arXiv2026

DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training

The rapid growth of deep learning models has increased the demand for efficient distributed training strategies. Fully sharded approaches like ZeRO-3 and FSDP partition model parameters across GPUs and apply optimizations such as prefetching and unsharding to reduce communication overhead. However, these systems lack fine-grained control over memory and communication scheduling, making it difficult to balance computation--communication overlap with memory requirements. Coordinating multiple optimizations such as prefetching and unsharding is also difficult, since their effects on memory usage can influence each other. To tackle these challenges, we propose DeepCompile, a compiler-based optimization framework for distributed training. DeepCompile transforms user-defined models into computation graphs and applies a series of profiling-guided optimization passes, each modifying the graph based on profiling information such as execution time and memory usage. This design allows each pass to flexibly insert, reorder, or remove operations such as all-gather and memory allocation, improving communication--computation overlap and reducing memory pressure. Each pass can access updated profiling feedback from earlier passes, enabling coordinated optimizations. We further enhance DeepCompile by three additional optimizations: proactive prefetching, selective unsharding, and adaptive offloading. Our evaluation shows that DeepCompile achieves up to 1.28$\times$ and 1.54$\times$ speedups over ZeRO-3 and FSDP baselines, respectively, and up to a 7.01$\times$ throughput increase in settings with limited GPU resources using offloading.

cs.DC↗

arXiv2026

Modified Delayed Acceptance MCMC for Quasi-Bayesian Inference with Linear Moment Conditions

We develop a computationally efficient framework for quasi-Bayesian inference based on linear moment conditions. The approach employs a delayed acceptance Markov chain Monte Carlo (DA-MCMC) algorithm that uses a surrogate target kernel and a proposal distribution derived from an approximate conditional posterior, thereby exploiting the structure of the quasi-likelihood. Two implementations are introduced. DA-MCMC-Exact fully incorporates prior information into the proposal distribution and maximizes per-iteration efficiency, whereas DA-MCMC-Approx omits the prior in the proposal to reduce matrix inversions, improving numerical stability and computational speed in higher dimensions. Simulation studies on heteroskedastic linear regressions show substantial gains over standard MCMC and conventional DA-MCMC baselines, measured by multivariate effective sample size per iteration and per second. The Approx variant yields the best overall throughput, while the Exact variant attains the highest per-iteration efficiency. Applications to two empirical instrumental variable regressions corroborate these findings: the Approx implementation scales to larger designs where other methods become impractical, while still delivering precise inference. Although developed for moment-based quasi-posteriors, the proposed approach also extends to risk-based quasi-Bayesian formulations when first-order conditions are linear and can be transformed analogously. Overall, the proposed algorithms provide a practical and robust tool for quasi-Bayesian analysis in statistical applications.

stat.CO↗

arXiv2025

SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips

The emergence of Superchips represents a significant advancement in next-generation AI hardware. These Superchips employ a tightly coupled heterogeneous architecture that integrates GPU and CPU on the same package, which offers unprecedented computational power. However, there has been scant research investigating how LLM training benefits from this new architecture. In this work, for the first time, we study LLM training solutions based on offloading for Superchips. We observe important differences between Superchips and traditional loosely-coupled GPU-CPU architecture, which necessitate revisiting prevailing assumptions about offloading. Based on that, we present SuperOffload, a Superchip-centric offloading system that simultaneously uses Hopper GPU, Grace CPU, and NVLink-C2C interconnect more efficiently. SuperOffload accomplishes this via a combination of techniques, such as adaptive weight offloading, bucketization repartitioning, Superchip-aware casting, speculative execution, and a highly optimized Adam optimizer for Grace CPUs. Our evaluation of SuperOffload on NVIDIA GH200 demonstrates up to 2.5x throughput improvement compared to state-of-the-art offloading-based systems, enabling training of up to 25B model on a single Superchip while achieving high training throughput. We also extend SuperOffload with ZeRO-style data parallelism and DeepSpeed-Ulysses sequence parallelism, enabling training of 13B model with sequence lengths up to 1 million tokens on 8 GH200 while achieving 55% MFU.

cs.LG↗

arXiv2025

ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloading framework that prioritizes important parameters and decouples updates between GPU and CPU. ZenFlow performs in-place updates of important gradients on GPU, while asynchronously offloading and accumulating less important ones on CPU, fully overlapping CPU work with GPU computation. To scale across GPUs, ZenFlow introduces a lightweight gradient selection method that exploits a novel spatial and temporal locality property of important gradients, avoiding costly global synchronization. ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, and reduces GPU stalls by over 85 percent, all while preserving accuracy.

cs.DC↗