arXiv · 2611.19687
Adaptive Precision Cholesky with Error Guarantees
Abstract
Modern accelerators increasingly support low-precision formats such as 8-bit floats and integers to speed up machine learning. We show that a Cholesky factorization in which both GEMM and TRSM are implemented using the fixed-point Ozaki 1 scheme satisfies the same backward error bounds as the standard floating-point Cholesky factorization, and that the same holds for a Cholesky factorization computed using a low-precision MicroScaled GEMM. We then use our error analyses to construct an adaptive precision Cholesky factorization that exploits low-precision formats while respecting a prescribed element-wise error bound, as long as the factorization succeeds. We test it in two settings. First, we use it to choose the number of Ozaki 1 slices at each iteration, with SYRK/GEMM and TRSM emulated in INT8 fixed-point arithmetic to attain double-precision accuracy. Second, we use it to build a low-precision Cholesky preconditioner for the conjugate gradient method (PCG), in which SYRK/GEMM take low-precision inputs and accumulate in high precision, with the input format selected adaptively for each panel; if the factorization breaks down, diagonal perturbation still yields a valid preconditioner, at the cost of the error bound guarantees. On an NVIDIA GH200, the adaptive Ozaki 1-based factorization is up to 29% faster than a right-looking Cholesky using a fixed slice count of 7, and up to 69% faster than one using cuBLAS' Automatic Dynamic Precision (ADP), while maintaining the same error guarantees. For PCG, FP8 factorizations with diagonal perturbation form effective preconditioners on the 13 tested SuiteSparse matrices, reducing the number of PCG iterations by up to $12\times$ over CG on the same matrix diagonally scaled by powers of two. For synthetic matrices with arithmetic spectra, and geometric spectra with $κ\le 10^4$, FP8 preconditioners also reduce the number of CG iterations.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Sudhanva Kulkarni, James W. Demmel, Laura Grigori. 2026-11-15. Adaptive Precision Cholesky with Error Guarantees. https://arxiv.science/abs/2611.19687
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.