Nonparametric In-Context Learning and Inference via Measure-Indexed Kernel Ridge Regression

Abstract

In-Context learning enables an estimator, trained across multiple tasks, to adapt to a new unseen task solely through its in-context samples, without any parameter refitting. Despite growing interest in the statistical foundations of in-context learning, two gaps still remain. First, existing convergence results do not characterize the distributional behavior of the in-context learning estimators. Second, classical nonparametric methods are not designed to handle the measure-valued inputs that in-context learning naturally produces. We address both gaps by developing a measure-indexed reproducing kernel Hilbert space framework, in which the adaptivity of the estimator is captured through the dependence of a mean functional on the empirical measure of the context. Within this framework, we propose a measure-indexed kernel ridge regression estimator computed from multiple independent tasks and establish its minimax-optimal convergence rate. We further prove that the proposed estimator is pointwise asymptotically normal, enabling the construction of confidence intervals for the mean and prediction intervals for the response on an unseen task. Notably, the estimator achieves valid uncertainty quantification without any refitting when adapting to a new task. As a special case, our framework recovers classical distributed nonparametric regression when all tasks share the same underlying function. The proposed framework is illustrated through a latent Gaussian process example.