arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · astro-ph/0508145

Optimized Data Loading for a Multi-Terabyte Sky Survey Repository

Y. Dora Cai·Ruth Aydt·Robert J. Brunner
View original source ↗Read source document ↗Export metadata ↓

Abstract

Advanced instruments in a variety of scientific domains are collecting massive amounts of data that must be post-processed and organized to support scientific research activities. Astronomers have been pioneers in the use of databases to host highly structured repositories of sky survey data. As more powerful telescopes come online, the increased volume and complexity of the data collected poses enormous challenges to state-of-the-art database systems and data-loading techniques. When the data source is an instrument taking ongoing samples, the database loading must, at a minimum, keep up with the data-acquisition rate. These challenges are being faced not only by the astronomy community, but also by other scientific disciplines interested in building scalable databases to house multi-terabyte archives of complex structured data. In this paper we present SkyLoader, our novel framework for fast and scalable data loading that is being used to populate a multi-table, multi-terabyte database repository for the Palomar-Quest sky survey. Our framework consists of an efficient algorithm for bulk loading, an effective data structure to support data integrity and proper error handling during the loading process, support for optimized parallelism that matches the number of concurrent loaders with the database host capabilities, and guidelines for database and system tuning. Performance studies showing the positive effects of the adopted strategies are also presented. Our parallel bulk loading with array buffering technique has made fast population of a multi-terabyte repository a reality, reducing the loading time for a 40-gigabyte data set from more than 20 hours to less than 3 hours. We believe our framework offers a promising approach for loading other large and complex scientific databases.

Explore related subjects

astro-ph

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Y. Dora Cai, Ruth Aydt, Robert J. Brunner. 2005-08-04. Optimized Data Loading for a Multi-Terabyte Sky Survey Repository. https://arxiv.org/abs/astro-ph/0508145

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

On binary pulsars and the force of gravity

The energy-momentum budget of the astrophysical systems can be studied by the exact local conservation equation derived by Landau and Lifshitz. We show that a similar equation is valid for the Einstein-Cartan gravity. We reanalyze a binary pulsar system using the Landau-Lifshitz conservation equation and show that the orbital period change rate can be completely understood as a curvature backreaction process. Taking into account the detailed theoretical and observational research of relativistic binary pulsar systems, especially the system of Hulse and Taylor, we conclude that general relativity and astrophysical observations rule out the existence of gravitational radiation. We comment upon the LIGO GW events and their alternative explanation, as well as the recent pulsar timing arrays data.

astro-ph↗
arXiv2026

Oscillation frequencies and mode lifetimes in alpha Centauri A

We analyse our recently-published velocity measurements of alpha Cen A (Butler et al. 2004). After adjusting the weights on a night-by-night basis in order to optimize the window function to minimize sidelobes, we extract 42 oscillation frequencies with l=0 to 3 and measure the large and small frequency separations. We give fitted relations to these frequencies that can be compared with theoretical models and conclude that the observed scatter about these fits is due to the finite lifetimes of the oscillation modes. We estimate the mode lifetimes to be 1-2 d, substantially shorter than in the Sun.

astro-ph↗
arXiv2026

Hipparcos period-luminosity relations for Miras and semiregular variables

We present period-luminosity diagrams for nearby Miras and semiregulars, selecting stars with parallaxes better than 20 per cent and well-determined periods. Using K-band magnitudes, we find two well-defined P-L sequences, one corresponding to the standard Mira P-L relation and the second shifted to shorter periods by a factor of about 1.9. The second sequence only contains semiregular variables, while the Mira sequence contains both Miras and semiregulars. Several semiregular stars show double periods in agreement with both relations. The Whitelock evolutionary track is shown to fit the data, indicating that the semiregulars are Mira progenitors. The transition between the two sequences may correspond to a change in pulsation mode or to a change in the stellar structure. Large amplitude pulsations leading to classical Mira classification occur mainly near the tip of the local AGB luminosity function.

astro-ph↗