arXiv · 2611.18732
ShiftGuard: Evaluating Confidence and Feature Distance for Risk-Controlled Image Classification
Abstract
A classifier that is usually right still makes confident mistakes, and a deployed system needs some way to decide when to answer and when to abstain. Cheap post-hoc scores such as maximum softmax probability (MSP) and predictive entropy are the standard answer. Distances in feature space are a popular alternative, and fusing several signals into one score is a natural next step. We ask whether adding a feature distance signal to MSP and entropy improves error detection, and whether it increases the share of predictions that can be accepted at a certified error rate when only a small labelled calibration set is available. We train a CIFAR-adapted ResNet-18 from scratch on CIFAR-10 (92.92% test accuracy, one seed) and compare five reliability scores: MSP, entropy, distance to the predicted-class centroid, a logistic fusion of MSP and entropy, and a logistic fusion of all three signals. Acceptance thresholds are chosen on a held out calibration set with Clopper Pearson upper bounds, Bonferroni corrected over a prespecified family of 315 rules at 95% simultaneous confidence. Confidence based scores were best. Error AUROC was 0.9230 for MSP, 0.9233 for entropy and 0.9235 for MSP plus entropy, against 0.8814 for the three signal fusion and 0.8224 for distance alone. At a 5% risk target with 1,000 calibration images, entropy accepted 84.82% of test images with 2.05% observed error, while three signal fusion accepted 80.61% with 2.21%. With 250 calibration images, neither distance nor the three-signal fusion could certify the 5% target at all. The results come from one seed, one dataset and one architecture, with no corruption experiments, so they say nothing about distribution shift detection. Within these limits, the distance signal did not earn its added complexity.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Mitra Varun. 2026-11-10. ShiftGuard: Evaluating Confidence and Feature Distance for Risk-Controlled Image Classification. https://arxiv.science/abs/2611.18732
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.