Krityapriya Bhaumik | arXiv Science

arXiv2026

Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

While language models provide powerful mechanisms for evaluating textual predictability, it remains unclear whether the relative extents of LM surprisal remain stable across models. This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources. We consider two stylistic environments: a corpus of contemporary poetry and a corpus of academic research papers. We demonstrate that mean and variance of cross-entropy maintain robust cross-model rank correlation ($ρ> 0.90$) across both regimes. To test whether this agreement merely reflects generic differences in textual complexity, we compare it against six parameter-free compression baselines. The extent of agreement among language models is found to be substantially higher than that among model-baselines, for most of the measures. Geometric analysis of the structurally constrained academic regime reveals that higher-order moments are sensitive to specific models, although strong agreement persists. The findings suggest that relative extents of textual unpredictability may be independent of the language model they are computed against.

cs.CL↗

arXiv2026

Robust Dual-Signal Fusion: Hybrid Neuro-Symbolic Gating with Compressed Chain-of-Thought Refinement for Irony Detection in Social Media Texts

Small-scale Large Language Models (LLMs) natively default to literal semantic interpretations, making few-shot irony detection a persistent challenge in noisy, user-generated text. In this study, we show that, despite this limitation, inference from compact LLMs can still be effectively harnessed for this task. We introduce the Robust Dual-Signal (RDS) Fusion framework, a hybrid neuro-symbolic architecture that utilizes a compressed Chain-of-Thought (CoT) of an LLM, alongside a static, pretrained RoBERTa and a symbolic prior module in two downstream fusions. RDS Fusion demonstrates better performance than an unrestricted reasoning ablation, while requiring substantially lower inference time. We also find that in our experiments, unrestricted reasoning does not yield better results than restricted reasoning. Evaluated on a strictly held-out TweetEval test set ($N=734$), RDS Fusion achieves $78.1\%$ accuracy and a Macro F1 of $0.777$, which is comparable to a finetuned BERTweet-base. On the heavily imbalanced iSarcasm dataset, the frozen CoT pipeline suppresses $22.5\%$ of baseline false positives, yielding a Macro F1 of $0.6726$ and Ironic F1 of $0.4821$, outperforming multiple heavily supervised SemEval transformer ensembles. Paired McNemar's tests show that, while adding the symbolic prior to the neural baseline yields an insignificant gain, and the RDS Fusion is statistically insignificant compared to the combined RoBERTa and symbolic prior ablation; the concurrent fusion achieves a statistically significant improvement over the standalone baseline ($p=0.005$).

cs.CL↗