arXiv · 2611.19355
Numerical Integration Instabilities in Likelihood Fits: Mechanisms and Remedies
Abstract
Numerical instabilities in the normalization of probability density functions can produce artificially lower negative log-likelihood (NLL) values than the true minimum of a likelihood fit, even when the resulting fit provides a poorer description of the data. Motivated by anomalously low NLL values found in non-converged fits associated with a recent CMS search for fully charmed tetraquark candidates, we use a pseudo-dataset generated from a likelihood model based on that analysis to investigate how numerical integration can yield unreliable normalization results, how these errors affect likelihood minimization, and how they can be addressed. We find that, even at fixed integration tolerance, the stopping decision of a numerical integrator can change with the fit parameters, introducing discontinuities in the normalization integrals that propagate to the likelihood surface. Such discontinuities can trap the minimizer at a numerical step and yield an artificially lower NLL. Tightening the integration tolerance mitigates these numerical discontinuities, while adaptive integration methods, such as Gauss--Kronrod quadrature, can also improve the numerical stability. A more severe failure occurs when the integration grid cannot adequately resolve a sharply localized contribution. In this case, the integrator may satisfy its estimated-error criterion despite substantially underestimating the normalization, allowing the minimizer to enter a region with an artificially low NLL. Tightening integration tolerances or using alternative adaptive integration methods does not necessarily resolve sharply localized contributions. We demonstrate that a suitable variable transformation can improve their numerical resolution, yielding more accurate normalization integrals and restoring fit convergence.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Manqi Zhu, Gerry Bauer, Kai Yi, Xining Wang. 2026-11-14. Numerical Integration Instabilities in Likelihood Fits: Mechanisms and Remedies. https://arxiv.science/abs/2611.19355
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.