arXiv Science⌕ Search

arXiv · 2611.17526

User-Schedulable Approximations for Quantized Tensor Programs

Abstract

Quantization lets language models run on edge devices, but it sharply increases the number of tensor kernels a library must maintain: every weight format must be paired with every activation format on every instruction set and GPU, with an encoder and a decoder that agree on the stored bytes. In GGML, the tensor library used by llama.cpp, these kernels are written by hand, and only 8 of the 54 weight-activation pairs it runs on our Arm CPU have a tiled matrix-multiplication kernel. We extend Halide's separation of algorithm and schedule to the representation of values. Each approximation is a codec, an encoder paired with a decoder, composed from reusable units: quantizers, integer representations, bit packers and layouts. A new directive, approximate_by, applies a codec to one stage of a program and returns the encoder and decoder as ordinary stages that Halide can schedule, so both decoders fuse into the register tiles of a matrix product. Optional contracts, such as a round-trip law, are checked by property tests; the law exposed an inconsistent block sum in GGML's q8_1. One algorithm and one schedule library generate 347 kernels for 31 of GGML's and KleidiAI's formats on Arm CPUs and Apple GPUs, covering 38 weight-activation pairs for which GGML has no kernel, in 3.7x less code than GGML and KleidiAI. On an Apple M3 Max, our kernels reach at least 95% of the speed of GGML's format-specific kernels in 91.0% of the cases that have one, with a median speedup of 1.13x, and are faster still where GGML falls back to blocked dot products. Counting Apple's BLAS library, which is usually faster for larger f32 products, we still reach that threshold in 86.9% of CPU cases. A port to x86 with Intel's AMX adds 793 lines of code and matches GGML in 83.5% of its cases.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Alex Reinking. 2026-11-12. User-Schedulable Approximations for Quantized Tensor Programs. https://arxiv.science/abs/2611.17526

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

The Best of Times, the Worst of Times: Moment-Based Analysis of Probabilistic Cost Structures

Many practical costs---such as the completion time of a quantum repeater, the rounds of an RFID collision-resolution protocol, or the response time of a fork--join computation---are described by hierarchical cost expressions: a tree of independent subcomputations with nonnegative integer costs, combined by addition, max, and min. We compute their moments with sound error bounds. Moment-based analyses for additive costs do not apply, because the moments of $\max(X,Y)$ are not determined by those of $X$ and $Y$; propagating full distributions becomes expensive as trees grow; and scalable approximations are hand-derived for specific structures and lack error bounds. Our analysis proceeds bottom-up, summarizing each subtree by a surrogate distribution: an exact short-horizon prefix plus a compact parametric tail fitted to preserve the queried moment. Each prefix is computed exactly from the children's prefixes---for a retry, whose cost sums an unbounded number of attempts, by solving a local recurrence equation---so all approximation is confined to the tails. Abstraction introduces shape error but no moment error; nonlinear operators, such as max and min, convert shape error into moment error. The analysis tracks two bounds, one on shape error and one on moment error. Tracking both lets it report a sound interval for the mean, the second moment, and the variance. Lengthening the prefix, or solving subtrees exactly, narrows the interval at the cost of additional computation. For quantum-repeater chains of up to 4096 links with high swap-success probability, our Dickens tool bounds the mean completion time within 1%, whereas the exact Markov-chain solution exhausts memory with 16 links, and the best hand-derived approximation we evaluated has 47% error with 64 links. Dickens also bounds second moments, variances, and deadline-miss probabilities, and handles RFID and fork--join costs.

cs.PL↗

A Compilation Model for Divergent Parallel Workloads

Efficient execution on modern parallel machines requires more than abundant parallelism; it demands instruction coherence, memory coherence, and load balance to maximize SIMD utilization and memory bandwidth. Efficiently mapping \textit{divergent} parallel workloads to modern machines, where control flow, memory accesses, and unbounded iteration depend on per-thread runtime values, often requires global program restructuring that trades off increased coherence with reduced locality, parallelism, or extra work. We observe that the key optimizations used to accelerate divergent parallel programs can be expressed as compositions of well-known program transformations from the functional programming and loop optimization communities. We operationalize this insight through \textit{parallel dispatch graphs}: a compilation model that represents divergent programs by representing control flow, data flow between logical stages, and parallel loops within a single abstraction. To facilitate rapid exploration of points in the optimization space, we provide a scheduling language that includes composable transformations of data-dependent control flow and data flow, together with standard loop transformations and hardware bindings. This separation of concerns allows the core algorithm to remain portable, while the execution schedule can be specialized to the specifics of target hardware and input data values. We evaluate our system using popular applications from the domain of spatial computing, and show that the optimization strategies of hand-optimized parallel tree traversals, path tracers, and Monte Carlo PDE solvers can be reproduced as schedules in our system. Our generated code matches or outperforms the hand-optimized implementations (geomeans $1.02$--$2.95\times$ faster). Additionally, we demonstrate empirically that the best schedule varies across programs, machines, and inputs.

cs.PL↗

Datatypes In, Zippers Out!

A zipper represents a focused position in an algebraic data structure together with the context needed to reconstruct the whole. Datatype derivatives determine one-hole context types, but do not by themselves choose a zipper's concrete representation or specify navigation costs and demands. We present Kamot, a tool that converts regular algebraic datatype declarations in a restricted Haskell fragment into zipper implementations with operational contracts. Navigation, replacement, and reconstruction are proved correct against a reference traversal. Kamot supports mutual recursion and regular containers inside containers. A compositional analysis derives sound bounds on transitions and constructor demands for certified traversal directions. These bounds are affine in trail depth (the number of saved frames) and guarantee termination of each move on productive, potentially infinite inputs. For inhabited families the analysis is complete: when no move certificate is emitted, no bound depending only on trail depth exists, and some productive input makes a move diverge. We prove that the generated code preserves results and the reference order of constructor matches, including finite prefixes of divergent searches. Lean 4 verifies simulation, bounds, and completeness for finite values of single self-recursive datatypes, in both directions. On 19 datatype models and 81 generated families, every generated module passes Haskell type checking, and independent checkers validate certificates, traces, and bounds. In a prefetching workload, starting only searches whose bounds fit a budget answers every local query and avoids discarded source evaluations on long returns, but conservative bounds can lose answers that fuelled search delivers.

cs.PL↗