arXiv ScienceSearch

arXiv subjects

Sarod Yatawatta

Publications and source records attributed to Sarod Yatawatta.

At least 19 recordsLinked to original sources

Recovering Weak Signals with Normalizing Flows

In many scientific disciplines, weak signals of interest are obscured by dominant nuisance signals that are several orders of magnitude stronger. Recovering these weak signals requires subtracting the dominant ones; however, this calibration process inherently distorts or partially suppresses the underlying signal of interest. To address this problem, we propose the use of normalizing flow models to reconstruct calibration-affected weak signals. By leveraging the statistical invariance of the target signals and assuming minimal initial suppression, our framework effectively recovers the lost signal components. We provide a comprehensive theoretical overview of this normalizing flow-based recovery method and demonstrate its efficacy using simulated data.

stat.ML

Observations of the Cosmic Dawn and Epoch of Reionization with the SKAO: Observational Lessons Learned from Precursors and Pathfinder Instruments

This chapter summarizes the observational lessons learned after two decades of observations of the Cosmic Dawn (CD) and Epoch of Reionization (EoR) with SKAO pathfinders and precursors. We will describe the effort towards building accurate simulation pipelines for actual observations and summarize the approaches that different groups have taken to calibrate and mitigate systematic effects such as sky model incompleteness, limited instrument models and antenna mutual coupling. We conclude by discussing the impact that these lessons may have on the design and analysis of upcoming SKAO observations of the Cosmic Dawn and Epoch of Reionization.

astro-ph.IM

Overview of 21cm Experiments at high redshift with SKAO

We provide an overview of the eight SKAO Science Book chapters that motivate the Epoch of Reionisation and Cosmic Dawn experiments with SKA-Low. We describe the individual SKA-Low experiments and expected sensitivity - power spectrum, tomography, 21-cm forest, cross-correlations, building on the broad observational plan laid out in the 2015 SKA Science Book. Finally, we outline features of the telescope that will be critical for the success of EoR/CD science, e.g., beam apodization, substations, and multi-beaming.

astro-ph.CO

Direction-dependent calibration with image-domain gridding

Wide-field images made by radio interferometers are invariably affected by direction-dependent systematic effects such as the ionosphere or the beam pattern. Calibration along a set of discrete directions in the sky is the default technique to estimate and correct these systematic errors. However, additional processing such as smoothing and mosaicing are required to reconcile the step wise variation of the estimated systematic errors at the edges of the discrete directions (facets). We overcome the discrete nature of direction-dependent calibration by using image-domain gridding as the model for the calibration. Instead of discrete directions in the sky, calibration is performed using a basis that represents a set of sub-grids in the Fourier space. This automatically removes the need for extra operations to recover the wide-field systematics error model without any discontinuity. We provide results based on LOFAR data where we compare the traditional facet-based (discrete directional gains) calibration with the proposed approach. The comparison shows improved image quality, mainly because of the physical plausibility of the proposed approach as opposed to using a piecewise constant model for direction-dependent systematic errors.

astro-ph.IM

Polarization based direction of arrival estimation using a radio interferometric array

Direction of arrival (DOA) estimation is mostly performed using specialized arrays that have carefully designed receiver spacing and layouts to match the operating frequency range. In contrast, radio interferometric arrays are designed to optimally sample the Fourier space data for making high quality images of the sky. Therefore, using existing radio interferometric arrays (with arbitrary geometry and wide frequency variation) for DOA estimation is practically infeasible except by using images made by such interferometers. In this paper, we focus on low cost DOA estimation without imaging, using a subset of a radio interferometric array, using a fraction of the data collected by the full array, and, enabling early determination of DOAs. The proposed method is suitable for transient and low duty cycle source detection. Moreover, the proposed method is an ideal follow-up step to online radio frequency interference (RFI) mitigation, enabling the early estimation of the DOA of the detected RFI.

astro-ph.IM

Energy and polarization based on-line interference mitigation in radio interferometry

Radio frequency interference (RFI) is a persistent contaminant in terrestrial radio astronomy. While new radio interferometers are becoming operational, novel sources of RFI are also emerging. In order to strengthen the mitigation of RFI in modern radio interferometers, we propose an on-line RFI mitigation scheme that can be run in the correlator of such interferometers. We combine statistics based on the energy as well as the polarization alignment of the correlated signal to develop an on-line RFI mitigation scheme that can be applied to a data stream produced by the correlator in real-time, especially targeted at low duty-cycle or transient RFI detection. In order to improve the computational efficiency, we explore the use of both single precision and half precision floating point operations in implementing the RFI mitigation algorithm. This ideally suits its deployment in accelerator computing devices such as graphics processing units (GPUs) as used by the LOFAR correlator. We provide results based on real data to demonstrate the efficacy of the proposed method.

astro-ph.IM

Reinforcement learning

Observing celestial objects and advancing our scientific knowledge about them involves tedious planning, scheduling, data collection and data post-processing. Many of these operational aspects of astronomy are guided and executed by expert astronomers. Reinforcement learning is a mechanism where we (as humans and astronomers) can teach agents of artificial intelligence to perform some of these tedious tasks. In this paper, we will present a state of the art overview of reinforcement learning and how it can benefit astronomy.

astro-ph.IM

Diffuse radio sky models using large scale shapelets

Sky models used in radio interferometric data processing primarily consist of compact and discrete radio sources. When there is a need to model large scale diffuse structure such as the Galaxy, specialized source models are sought after for the sake of simplicity and computational efficiency. In this paper, we propose the use of shapelet basis functions for modeling large scale diffuse structure in various radio interferometric data processing pipelines. Conventional source model construction using shapelet basis functions is restricted to using images of smaller size due to limitations in computational resources such as memory. We propose a novel shapelet decomposition method to lift this restriction, enabling the use of images of millions of pixels (as well as a wide spectral bandwidth) for building models of large scale diffuse structure. Furthermore, the application of direction dependent errors onto diffuse sky models is an expensive operation often performed as a convolution. In this paper, we propose using some specific properties of shapelet basis functions to apply such direction dependent errors as a product of the model coefficients, thus the avoiding the need for convolution. We provide results based on simulations and real observations. In order to measure the efficacy of our proposed method in modeling large scale diffuse structure, we consider direction dependent calibration of simulated as well as real LOFAR observations that have a significant amount of large scale diffuse structure. Results show that by including large scale shapelet models of the diffuse sky, we are able to overcome a major problem faced by existing calibration techniques that do not model such large scale diffuse structure, i.e., the suppression of such large scale diffuse structure due to model incompleteness.

astro-ph.IM

Hint assisted reinforcement learning: an application in radio astronomy

Model based reinforcement learning has proven to be more sample efficient than model free methods. On the other hand, the construction of a dynamics model in model based reinforcement learning has increased complexity. Data processing tasks in radio astronomy are such situations where the original problem which is being solved by reinforcement learning itself is the creation of a model. Fortunately, many methods based on heuristics or signal processing do exist to perform the same tasks and we can leverage them to propose the best action to take, or in other words, to provide a `hint'. We propose to use `hints' generated by the environment as an aid to the reinforcement learning process mitigating the complexity of model construction. We modify the soft actor critic algorithm to use hints and use the alternating direction method of multipliers algorithm with inequality constraints to train the agent. Results in several environments show that we get the increased sample efficiency by using hints as compared to model free methods.

astro-ph.IM

Spatially constrained direction dependent calibration

Direction dependent calibration of widefield radio interferometers estimates the systematic errors along multiple directions in the sky. This is necessary because with most systematic errors that are caused by effects such as the ionosphere or the receiver beam shape, there is significant spatial variation. Fortunately, there is some deterministic behavior of these variations in most situations. We enforce this underlying smooth spatial behavior of systematic errors as an additional constraint onto spectrally constrained direction dependent calibration. Using both analysis and simulations, we show that this additional spatial constraint improves the performance of multi-frequency direction dependent calibration.

astro-ph.IM

Deep reinforcement learning for smart calibration of radio telescopes

Modern radio telescopes produce unprecedented amounts of data, which are passed through many processing pipelines before the delivery of scientific results. Hyperparameters of these pipelines need to be tuned by hand to produce optimal results. Because many thousands of observations are taken during a lifetime of a telescope and because each observation will have its unique settings, the fine tuning of pipelines is a tedious task. In order to automate this process of hyperparameter selection in data calibration pipelines, we introduce the use of reinforcement learning. We test two reinforcement learning techniques, twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), to train an autonomous agent to perform this fine tuning. For the sake of generalization, we consider the pipeline to be a black-box system where the summarized state of the performance of the pipeline is used by the autonomous agent. The autonomous agent trained in this manner is able to determine optimal settings for diverse observations and is therefore able to perform 'smart' calibration, minimizing the need for human intervention.

astro-ph.IM

Polarization-based online interference mitigation in radio interferometry

Mitigation of radio frequency interference (RFI) is essential to deliver science-ready radio interferometric data to astronomers. In this paper, using dual polarized radio interferometers, we propose to use the polarization information of post-correlation interference signals to detect and mitigate them. We use the directional statistics of the polarized signals as the detection criteria and formulate a distributed, wideband spectrum sensing problem. Using consensus optimization, we solve this in an online manner, working with mini-batches of data. We present extensive results based on simulations to demonstrate the feasibility of our method.

astro-ph.IM

Stochastic Calibration of Radio Interferometers

With ever increasing data rates produced by modern radio telescopes like LOFAR and future telescopes like the SKA, many data processing steps are overwhelmed by the amount of data that needs to be handled using limited compute resources. Calibration is one such operation that dominates the overall data processing computational cost, nonetheless, it is an essential operation to reach many science goals. Calibration algorithms do exist that scale well with the number of stations of an array and the number of directions being calibrated. However, the remaining bottleneck is the raw data volume, which scales with the number of baselines, and which is proportional to the square of the number of stations. We propose a 'stochastic' calibration strategy where we only read in a mini-batch of data for obtaining calibration solutions, as opposed to reading the full batch of data being calibrated. Nonetheless, we obtain solutions that are valid for the full batch of data. Normally, data need to be averaged before calibration is performed to accommodate the data in size-limited compute memory. Stochastic calibration overcomes the need for data averaging before any calibration can be performed, and offers many advantages including: enabling the mitigation of faint radio frequency interference; better removal of strong celestial sources from the data; and better detection and spatial localization of fast radio transients.

astro-ph.IM

GPU acceleration of the SAGECal calibration package for the SKA

SAGECal has been designed to find the most accurate calibration solutions for low radio frequency imaging observations, with minimum artefacts due to incomplete sky models. SAGECAL is developed to handle extremely large datasets, e.g., when the number of frequency bands greatly exceeds the number of available nodes on a compute cluster. Accurate calibration solutions are derived at the expense of large computational loads, which require distributed computing and modern compute devices, such as GPUs, to decrease runtimes. In this work, we investigate if the GPU version of SAGECal scales well enough to meet the requirements for the Square Kilometre Array and we compare its performance with the CPU version.

astro-ph.IM

A Stochastic LBFGS Algorithm for Radio Interferometric Calibration

We present a stochastic, limited-memory Broyden Fletcher Goldfarb Shanno (LBFGS) algorithm that is suitable for handling very large amounts of data. A direct application of this algorithm is radio interferometric calibration of raw data at fine time and frequency resolution. Almost all existing radio interferometric calibration algorithms assume that it is possible to fit the dataset being calibrated into memory. Therefore, the raw data is averaged in time and frequency to reduce its size by many orders of magnitude before calibration is performed. However, this averaging is detrimental for the detection of some signals of interest that have narrow bandwidth and time duration such as fast radio bursts (FRBs). Using the proposed algorithm, it is possible to calibrate data at such a fine resolution that they cannot be entirely loaded into memory, thus preserving such signals. As an additional demonstration, we use the proposed algorithm for training deep neural networks and compare the performance against the mainstream first order optimization algorithms that are used in deep learning.

astro-ph.IM

Statistical Performance of Radio Interferometric Calibration

Calibration is an essential step in radio interferometric data processing that corrects the data for systematic errors and in addition, subtracts bright foreground interference to reveal weak signals hidden in the residual. These weak and unknown signals are much sought after to reach many science goals but the effect of calibration on such signals is an ever present concern. The main reason for this is the incompleteness of the model used in calibration. Distributed calibration based on consensus optimization has been shown to mitigate the effect due to model incompleteness by calibrating data covering a wide bandwidth in a computationally efficient manner. In this paper, we study the statistical performance of direction dependent distributed calibration, i.e., the distortion caused by calibration on the residual statistics. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship for the influence of the input on the residual and use this to find the relationship between the input and output probability density functions. Using simulations we show that the smallest eigenvalue of the Jacobian of this mapping is a reliable indicator of the statistical performance of calibration. The analysis developed in this paper can also be applied to other data processing steps in radio interferometry such as imaging and foreground subtraction as well as to many other machine learning problems.

astro-ph.IM

Performance Analysis of Distributed Radio Interferometric Calibration

Distributed calibration based on consensus optimization is a computationally efficient method to calibrate large radio interferometers such as LOFAR and SKA. Calibrating along multiple directions in the sky and removing the bright foreground signal is a crucial step in many science cases in radio interferometry. The residual data contain weak signals of huge scientific interest and of particular concern is the effect of incomplete sky models used in calibration on the residual. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship between the input and output probability density functions which can be used to study the performance of calibration.

astro-ph.IM

Distributed Model Construction in Radio Interferometric Calibration

Calibration of a typical radio interferometric array yields thousands of parameters as solutions. These solutions contain valuable information about the systematic errors in the data (ionosphere and beam shape). This information could be reused in calibration to improve the accuracy and also can be fed into imaging to improve the fidelity. We propose a distributed optimization strategy to construct models for the systematic errors in the data using the calibration solutions. We formulate this as an elastic net regularized distributed optimization problem which we solve using the alternating direction method of multipliers (ADMM) algorithm. We give simulation results to show the feasibility of the proposed distributed model construction scheme.

astro-ph.IM