arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 1304.5302

"Superluminal" FITS File Processing on Multiprocessors: Zero Time Endian Conversion Technique

Satoshi Eguchi
View original source ↗Read source document ↗Export metadata ↓

Abstract

The FITS is the standard file format in astronomy, and it has been extended to agree with astronomical needs of the day. However, astronomical datasets have been inflating year by year. In case of ALMA telescope, a ~ TB scale 4-dimensional data cube may be produced for one target. Considering that typical Internet bandwidth is a few 10 MB/s at most, the original data cubes in FITS format are hosted on a VO server, and the region which a user is interested in should be cut out and transferred to the user (Eguchi et al., 2012). The system will equip a very high-speed disk array to process a TB scale data cube in a few 10 seconds, and disk I/O speed, endian conversion and data processing one will be comparable. Hence to reduce the endian conversion time is one of issues to realize our system. In this paper, I introduce a technique named "just-in-time endian conversion", which delays the endian conversion for each pixel just before it is really needed, to sweep out the endian conversion time; by applying this method, the FITS processing speed increases 20% for single threading, and 40% for multi-threading compared to CFITSIO. The speed-up by the method tightly relates to modern CPU architecture to improve the efficiency of instruction pipelines due to break of "causality", a programmed instruction code sequence.

Explore related subjects

astro-ph.IMcs.PF

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Satoshi Eguchi. 2013-04-19. "Superluminal" FITS File Processing on Multiprocessors: Zero Time Endian Conversion Technique. https://doi.org/10.1086/671105

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Using 23 Years of ACS/SBC Data to Understand Backgrounds: Significant Reductions in Expected Background Levels

We have used 23 years of Hubble Space Telescope ACS/SBC data to study what background levels are encountered in practice and how much they vary. The backgrounds vary considerably, with F115LP, F122M, F125LP, PR110L, and PR130L all showing over an order of magnitude of variation in background between observations, apparently due to changes in airglow. The F150LP and F165LP filters, which are dominated by dark rate, not airglow, exhibit a far smaller variation in backgrounds. For the filters where the background is generally dominated by airglow, the backgrounds measured from the data are significantly lower than what the ETC predicts (as of ETC v33.2). The ETC predictions for `average' airglow are greater than the median of our measured background values by factors of 2.51, 2.64, 105, and 3.64, for F115LP, F122M, F125LP, and F140LP, respectively. A preliminary analysis suggests this could be due to certain OI airglow lines usually being fainter than expected by the ETC. With reduced reduced background levels, the shorter-wavelength SBC filters can conduct background-limited observations much more rapidly than had previously been expected. As of ETC v34.1, a new option will be included for SBC calculations, allowing users to employ empirical background percentiles to estimate required exposure times.

astro-ph.IM↗
arXiv2026

Using 23 Years of ACS/SBC Data to Understand Backgrounds: Explaining & Predicting Background Variations

Recent analysis of 23 years of Hubble Space Telescope ACS/SBC data has shown that background levels can vary considerably between observations, with most filters showing over an order of magnitude variation. For the shorter-wavelength filters, the background is understood to be dominated by airglow; however, what precisely drives background variations is not well constrained for any filter. Here, we explore the causes of the background variation. Using over 8,000 archival SBC observations, we developed a machine learning model that can accurately predict the background for an observation, based upon a set of 23 observational parameters. This model indicates that, depending on filter, the SBC background is generally dominated by Solar elevation, Solar separation angle, Earth limb angle of observation, SBC temperature, and target Galactic latitude.

astro-ph.IM↗
arXiv2026

Beyond the BLUE I: the advantage ceiling - how much can any estimator beat the matched filter in mm/submm survey data?

Papers claiming that convolutional neural networks outperform the matched filter are now common in astronomy. Since that filter is the best linear unbiased estimator (BLUE) for any noise of given covariance, and the minimum-variance unbiased estimator under Gaussian noise, such a claim requires that the noise covariance varies from image to image or the noise is non-Gaussian. We introduce the advantage ceiling eta >= 1, which quantifies both cases: the Fisher information for the amplitude in units of the matched filter's, computable from noise-only simulations before any network is trained, and a bound on the variance of any estimator unbiased at each true amplitude. We compute eta for a suite of millimeter/submillimeter survey noise models and for compact and extended source templates, using a variational score-matching ladder whose rungs are estimator classes of increasing statistical order, from linear to convolutional. With an instrumental white-noise floor, Gaussian noise returns eta = 0.94-1.06 (+-0.04); covariance mixtures give exact ceilings of 10.2 (spectral tilt, extended source), 2.1 (leaked components) and 1.1-1.8 otherwise; residual artifacts give eta <= 1.09. Source confusion gives the largest advantage, eta >= 3.8 (extended) and >= 2.4 (compact). Three unmodelled channels - a zeroed DC mode, a missing white floor and floating-point rounding - can manufacture spurious advantages of up to two orders of magnitude. A ResNet regressor is 6-15% less efficient than the matched filter on Gaussian noise once prior shrinkage is divided out, and realizes a factor 1.8 of the 10.2 available on the spectral-tilt mixture; a companion paper tests such regressors against these ceilings.

astro-ph.IM↗