arXiv Science⌕ Search

arXiv subjects

Mars Davletshin

Publications and source records attributed to Mars Davletshin.

3 recordsLinked to original sources

HADVQ: Hessian Aware Directional Vector Quantization

Ultra-low-bit quantization of large language models is a challenging task because aggressive compression severely distorts model parameters and disproportionately harms a small number of highly sensitive channels. We present a vector quantization method tailored for this regime. Our approach begins with random preconditioning, which regularizes the weight space and makes it more amenable to quantization. We then represent weights in polar coordinates and decouple the quantization of directions and magnitudes. To quantize directions, we leverage the structure induced by random preconditioning and design a structured codebook based on the $E_8$ lattice, yielding a compact yet expressive representation. To further reduce degradation, we detect extremely important channels using a proxy Hessian and preserve them with an additional cost of only 0.16 bits. Together, these components provide an effective quantization pipeline for LLMs in the extremely low-bit regime. Empirical results demonstrate improved accuracy, highlighting the benefit of combining geometric structure, lattice-based codebooks, and selective channel preservation.

cs.LG↗

Weight Distributions for Successive Cancellation Decoding of Polar Codes

In this paper, we derive the exact weight distributions that emerge during each stage of successive cancellation decoding of polar codes. Though we do not compute the distance spectrum of polar codes, the results allow us to get an estimate of the decoding error probability and to show a link between the first nonzero components of the weight distribution and the partial order between the synthetic channels. Also, we establish the minimal distance between two cosets associated with two paths that differ in two positions. This can be regarded as a first step toward analyzing the weight distributions for the successive cancellation list decoding.

cs.IT↗

Arnold diffusion in multidimensional a priori unstable Hamiltonian systems

We study the Arnold diffusion in a priori unstable near-integrable systems in a neighbourhood of a resonance of low order. We consider a non-autonomous near-integrable Hamiltonian system with $n+1/2$ degrees of freedom, $n\ge 2$. Let the Hamilton function $H$ of depend on the parameter $\varepsilon$, for $\varepsilon=0$ the system is integrable and has a homoclinic asymptotic manifold $Γ$. Our main result is that for small generic perturbation in an $\varepsilon$-neighborhood of $Γ$ there exist trajectories the projections of which on the space of actions cross the resonance. By ``generic perturbations'' we mean an open dense set in the space of $C^r$-smooth functions $\frac{d}{d\varepsilon}\big|_{\varepsilon=0} H$, $r=r_0,r_0+1,\ldots,\infty,ω$. Combination of this result with results of \cite{DT} answers the main questions on the Arnold diffusion in a priori unstable case: the diffusion takes place for generic perturbation, diffusion trajectories can go along any smooth curve in the action space with average velocity of order $\varepsilon/|\log \varepsilon|$.

math.DS↗