arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.28066

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Haonan Huang·Yuwei Lu·Chengkun Cai
View original source ↗Read source document ↗Export metadata ↓

Abstract

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.

Explore related subjects

cs.CLcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Haonan Huang, Yuwei Lu, Chengkun Cai. 2026-10-20. Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models. https://arxiv.org/abs/2610.28066

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models

The choice of which tokens to mask is an under-examined design decision in masked language modeling (MLM). Standard pretraining masks tokens uniformly at random, whereas content-aware masking schemes can improve downstream transfer. We study masking as a task-adaptive component of the fine-tuning pipeline and introduce Typhoon, a masking strategy that uses the gradient of the MLM loss with respect to one-hot token inputs to estimate online how much each token type contributes to the objective. Typhoon maintains an exponential moving average of per-token-type saliency and calibrates these scores into a masking distribution whose expected masking rate matches a target budget under a token-independence approximation. We formalize the method and evaluate it against random masking and whole-word masking on two GLUE tasks, MRPC and CoLA, across three BERT-family backbones (TinyBERT, DistilBERT, and BERT-base) and five random seeds per configuration ($90$ training runs in total). Once seed variance is accounted for, no masking strategy is reliably better than the others on these tasks: on MRPC the gap between Typhoon and the best baseline stays within $0.004$ $F_1$, across all twelve Typhoon comparisons no paired test reaches significance, and every $95\%$ confidence interval contains zero. Inspecting Typhoon's learned distribution explains why: budget-preserving min-max calibration compresses the gradient ranking into a narrow band whose Kullback-Leibler divergence from uniform random masking is $7\times 10^{-4}$ nats per token, and widening the calibration spread by an order of magnitude in KL still leaves downstream accuracy unchanged within seed noise. Typhoon's apparent advantage in single-run experiments does not survive multi-seed evaluation, showing that gradient-based task-adaptive masking is competitive with, but does not outperform, uniform random masking at this scale.

cs.CL↗
arXiv2026

MedHal: a Synthetic Dataset for Medical Hallucination Detection

Hallucination, the generation of non factual content by AI systems, poses serious risks in medical contexts, where errors can directly affect patient outcomes. We present MedHal, a large-scale dataset specifically designed to assess capabilities and train models on the task of hallucination detection in medical texts. Current hallucination detection methods face significant limitations when applied to specialized domains like medicine, where they can have disastrous consequences. MedHal addresses this issue by incorporating diverse medical text sources and tasks covering both intrinsic and extrinsic hallucinations, and by providing a substantial volume of data samples suitable for training medical hallucination detection models. We demonstrate MedHal's utility by training and evaluating a baseline medical hallucination detection model, showing improvements over general-purpose hallucination detection approaches. This resource enables more efficient evaluation and training of medical text generation systems while reducing reliance on costly expert review, potentially accelerating the development of medical AI research.

cs.CL↗
arXiv2026

Computational Typology

Typology is a subfield of linguistics that focuses on the study and classification of languages based on their structural features. Unlike genealogical classification, which examines the historical relationships between languages, typology seeks to understand the diversity of human languages by identifying common properties and patterns, known as universals. In recent years, computational methods have played an increasingly important role in typological research, enabling the analysis of large-scale linguistic data and the testing of hypotheses about language structure and evolution. This article provides an illustration of the benefits of computational statistical modeling in typology.

cs.CL↗