arXiv · 2611.21403
Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs
Abstract
We audit face-related bias in VLMs using BALPORTRAITS, a new balanced dataset of 9,000 fictional portraits across ethnicity, gender, and age segments. Sixteen VLMs estimate portrait fitness scores for occupations, brands, activities, and banking or insurance products within an advertising-casting framework designed to avoid direct inference of protected attributes. Item-level estimations are aggregated with scenario-specific prestige scores to compare rankings across demographic groups. While the magnitude and direction of these biases vary across scenarios, model families, and demographic dimensions, with age, ethnicity, and gender contributing differently depending on the task, we find demographic dispersion across all evaluated models, with a majority of pairwise segment differences reaching statistical significance. This study introduces a controlled protocol to assess implicit face-related associations and demonstrates that single-scenario or single-attribute audits are insufficient.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Alexandre Fournier-Montgieux, Akram Elbouanani, Aboubacar Tuo, Adrian Popescu, Hervé Le Borgne, Bertrand Luvison. 2026-11-16. Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs. https://arxiv.science/abs/2611.21403
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.