Modeling the nanohertz gravitational wave background with the t-process: Improved frequentist analysis of gravitational-wave power
Pulsar timing array (PTA) collaborations have recently reported evidence for a gravitational wave background (GWB) signal in the nanohertz band, which most likely originates from the incoherent superposition of GWs emitted by a population of supermassive black hole binaries (SMBHBs). Since the number of binaries emitting in each frequency bin is finite, and since individual loud sources can dominate the GWB, the power spectral density (PSD) of the background is expected to deviate significantly from the smooth power law that describes an idealized population. PTA searches usually describe the PSD either with a power law or with a free spectrum model, which fits the PSD in each frequency bin as a free parameter. In this paper, we explore the t-process as a third model; it assumes a power law as the underlying spectrum but leaves each frequency bin free to deviate from it. We compare the performance of these three models on simulated datasets, injecting $100$ realizations of a realistic SMBHB population, to evaluate how well each model recovers a complex PSD. Using Bayesian and frequentist tools, we find that the t-process model gives the most precise PSD results, with uncertainties $43\%$ and $26\%$ narrower than those obtained with the power law and the free spectrum, respectively, while the improvement in accuracy is smaller. Moreover, using the t-process model, we compute isotropic and anisotropic signal-to-noise ratios with median gains of about $46\%$ over the power law and $65\%$ over the free spectrum in the first nine frequency bins. The t-process also returns lower anisotropy p-values and among the frequency bins with $p<0.01$, it allows to localize the loudest binary within the angular resolution of the sky maps in $14.3\%$ of the cases, compared to the $10\%$ for the power law and the $6.2\%$ for the free spectrum.