The Great Chat in the Sky: Aligning Exotic Astronomical Data to Language with AstroBridge

Abstract

Astronomical observations include images, spectra, and light curves, while the scientific literature describes these objects in natural language. We present AstroBridge, a framework that connects these observations to an LLM through a literature-assisted caption generation pipeline. We used catalog metadata to match observations with scientific texts that describe the same objects. A vision-language model generates descriptive captions from representations of observations, matched passages, and available metadata, creating observation-caption pairs without manual captioning. We train AstroBridge to predict captions directly from raw observations. Modality-specific frozen encoders extract representations that a shared, trainable fusion network aligns with the embedding space of an LLM. In caption-based evaluation of spectra, AstroBridge achieves 91.5% accuracy in distinguishing quasars from galaxies, compared to 69.0% from the base model prior to fine-tuning. Limitations remain in galaxy morphology, classification of minority supernova classes, and the factual quality of generated captions. We believe that AstroBridge's framework is a step towards the goal of aligning the totality of astronomy's diverse, exotic data through the lingua franca of text.