arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2505.16125

KoBALT: Korean Benchmark For Advanced Linguistic Tasks

Hyopil Shin·Sangah Lee·Dongjun Jang·Wooseok Song·Jaeyoon Kim·Chaeyoung Oh·Hyemi Jo·Youngchae Ahn·Sihyun Oh·Hyohyeong Chang·Sunkyoung Kim·Jinsik Lee
View original source ↗Read source document ↗Export metadata ↓

Abstract

We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61\% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66\%) to considerable weaknesses in phonology (31\%) and morphology (36\%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models.

Explore related subjects

cs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Hyopil Shin, Sangah Lee, Dongjun Jang, Wooseok Song, Jaeyoon Kim, Chaeyoung Oh, Hyemi Jo, Youngchae Ahn, Sihyun Oh, Hyohyeong Chang, Sunkyoung Kim, Jinsik Lee. 2025-05-22. KoBALT: Korean Benchmark For Advanced Linguistic Tasks. https://arxiv.org/abs/2505.16125

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Limits of LLM Text Detectors in Education

Students increasingly use the assistance of large language models (LLMs) in their academic writing. While slight assistance (e.g., grammar and style correction, as well as feedback) is permitted under most institutional policies, it is usually forbidden to offload entire writing tasks to LLMs. Unfortunately, current approaches to LLM-generated text detection predominantly assume a binary distinction between human-written and LLM-generated text, ignoring the breadth of realistic human-AI collaboration practices and limiting the validity of detection systems for educational assessment. In this paper, we propose a contribution-aware evaluation framework for LLM-based detection systems in education. We introduce a scale of eight student contribution levels that model realistic writing scenarios ranging from fully human-written texts to LLM-assisted revisions to fully LLM-generated and adversarially humanized texts. Institutional policies regarding LLM use can then be translated to thresholds of acceptable LLM assistance on this scale. We further present Generative Essay Detection in Education (GEDE), a novel benchmark dataset comprising more than 900 human-written and over 12,500 generated essays across 886 tasks and all contribution levels. Using this benchmark, we conduct a systematic evaluation of four state-of-the-art zero-shot and supervised detection methods across policy boundaries, contribution levels, generative models, out-of-distribution data, and text length. We show that most detectors struggle to accurately classify texts at intermediate student contribution levels, in particular LLM-assisted revisions of human-written texts. Such errors pose a substantial risk of false accusations, indicating that current text detection systems are (still) unsuitable to reliably support the enforcement of institutional policies regarding LLM assistance in education.

cs.CL↗
arXiv2026

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 938 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 31.6% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, we observe that reasoning helps mitigate the cross-lingual knowledge gap in LLMs, which refers to disparities in their ability to manifest knowledge across languages. KoSimpleQA can be found at https://github.com/naver-ai/KoSimpleQA.

cs.CL↗
arXiv2026

When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs

Despite substantial advances, large language models (LLMs) continue to exhibit hallucinations, generating plausible yet incorrect responses. In this paper, we highlight a critical yet previously underexplored class of hallucinations driven by spurious correlations -- superficial but statistically prominent associations between features (e.g., surnames) and attributes (e.g., nationality) present in the training data. We demonstrate that these spurious correlations induce hallucinations that are confidently generated, immune to model scaling, evade current detection methods, and persist even after refusal fine-tuning. Through systematically controlled synthetic experiments and empirical evaluations on state-of-the-art open-source and proprietary LLMs (including GPT-5), we show that existing hallucination detection methods, such as confidence-based filtering and inner-state probing, fundamentally fail in the presence of spurious correlations. Our theoretical analysis further elucidates why these statistical biases intrinsically undermine confidence-based detection techniques. Our findings thus emphasize the urgent need for new approaches explicitly designed to address hallucinations caused by spurious correlations.

cs.CL↗