CHARIZARD: A High-throughput Generic Pipeline for Present and Upcoming Interferometers

Abstract

Radio interferometers now deliver wide bandwidths and high sensitivity, and with them data volumes that are awkward to reduce by hand and slow to reduce automatically. The limit is not always the algorithms themselves but the way the work is laid out across the computing available. We present CHARIZARD, a pipeline that carries a raw measurement set through flagging, cross-calibration, self-calibration and imaging to full-Stokes images, and which is built around distributing that work. The observing band is repartitioned into sub-bands that stay independent through cross-calibration, the unit of work becomes the target field and sub-band pair at self-calibration, and stages touching disjoint data are run together. Each stage is an ordinary PBS or SLURM job, submitted and tracked by a scheduler-agnostic layer, and every computation runs inside a single rootless container, so a user holding no administrative rights gets the same behaviour on any cluster. We benchmark the pipeline on MeerKAT measurements with full polarization calibration. Residual on-axis instrumental polarization is $0.13$ per cent in linear and $0.06$ per cent in circular; the deepest field reaches $4.11\,μ$Jy~beam$^{-1}$ over $787$~MHz with $σ_I/σ_V = 1.010$; positions agree with RACS-low to better than $0.4\arcsec$ and flux densities to a median ratio of $1.06$. A $1.5$~TB track completes $48.6$ core-hours of work in $18.4$~hours with self-calibration and polarization calibration. The next generation of the radio telescopes will not be short of data. CHARIZARD attempts to show how the distribution of the work provides a useful alternative for handling the staggering processing requirements.