Do Astronomical Foundation Models Know When They Will Fail?

Abstract

Astronomical foundation models are increasingly used as reusable representations across surveys, but it is unclear when the same object remains consistently represented across instruments or whether a model can warn us before transfer fails. We present a conformal framework for auditing cross-survey representations using cross-matched Legacy Survey--HSC, HSC--JWST, and SDSS--HSC objects. Astronomy-specific AstroPT embeddings show lower translation errors and tighter conformal regions than general-purpose visual backbones at comparable coverage, while different model families fail on different objects. We then use adaptive conformal prediction to test whether the source embedding contains information about its own future error. Using a ridge difficulty model fitted to source embeddings, we find that this warning signal is consistent across the tested backbones for Legacy$\rightarrow$HSC, backbone-dependent for HSC$\rightarrow$JWST, and largely absent for SDSS$\rightarrow$HSC. When informative, adaptation improves coverage for the hardest objects. Matched-control analysis further reveals silent failures not explained by magnitude, apparent size, or nearest-neighbor distance. Hence, we show that conformal uncertainty provides a realistic way to identify when astronomical representations transfer reliably and when they fail without warning.