"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws

Liang Yue; Eric A. Vance

Home/Archive/arXiv

arXiv · 2610.23446

"You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws

Liang Yue·Eric A. Vance

View original source ↗Read source document ↗Export metadata ↓

Abstract

What the Web community believes about toxic conversations, such as that toxicity rises in longer and deeper threads, was measured mostly with Google's Perspective API, which is shutting down; large language models (LLMs) will make the next measurements. We find that LLM labelers mistake second-person address for toxicity, and that this one bias manufactures laws that human labels do not show. On 1.25 million news comments with crowd labels, Claude Haiku flags 12% of the non-toxic comments that contain no "you" and 51% of those with four or more; a single crowd rater flags 6% and 10%. Deeper replies address someone more often, so under Haiku toxicity rises with reply depth (+0.93 percentage points per level), while under the crowd's labels it slightly declines (-0.12). Rewriting "you" in the third person removes two thirds of the false rise; flagging only as many comments as the crowd does not remove it, and showing Haiku the parent comment enlarges it. All seven LLMs we test over-flag "you", and at the crowd's flag rate, without the parent, any false rise of the open ones vanishes at a fixed number of second-person words; Haiku also over-flags "you" on Reddit, Gab and Wikipedia talk pages. Agreement with human labels does not reveal the bias: we prove that a fixed Cohen's kappa leaves a law nearly unconstrained. We therefore certify the law instead of the labeler. Threadsafe uses the labeler's scores to choose which comments human raters label and to predict the others, and corrects the predictions with the human labels; linear laws are estimated without bias whatever the labeler does. In simulated audits its intervals for the laws keep nominal coverage from 250 labels on, where uniform-sample methods such as PPI++ fall short, and at 1,000 to 8,000 labels are 0.75 times as wide as those of design-based supervised learning (DSL); a thousand human labels rule out Haiku's rising law.

Explore related subjects

cs.SI cs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Liang Yue, Eric A. Vance. 2026-10-16. "You" Is Not Toxic: How LLM Labelers Rewrite the Web's Conversation Laws. https://arxiv.org/abs/2610.23446

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

I CAN Debias: Uncovering and Enhancing Self-debiasing Mechanisms of Social Bias in LLMs

As LLMs rapidly advance and become increasingly agentic, mitigating social bias and ensuring fairness at scale have become critical challenges. Prior work has shown that LLMs exhibit an emergent self-debiasing capability-an intrinsic mechanism that enables them to avoid generating biased outputs-but this mechanism remains poorly understood, leaving existing bias-mitigation approaches heavily reliant on carefully curated prompts or inefficient training procedures that risk catastrophic forgetting. To address this gap, we propose I-CAN-Debias, a unified framework for uncovering and enhancing self-debiasing mechanisms in LLMs through behavior-grounded Consistency-contrast-Aware Neuron analysis, which defines self-debiasing neurons not by the magnitude of their responses to biased inputs but by activation patterns that distinguish successful debiasing from biased failures. Using I-CAN-Debias, we find that self-debiasing neurons account for less than 2% of model parameters yet are causally critical: deactivating them raises the biased-response rate above 95%. These neurons cluster in the query and value projections of later layers and exhibit highly asymmetric cross-category transferability. Furthermore, targeted fine-tuning of these neurons substantially improves fairness while preserving general capabilities. Importantly, models tuned with I-CAN-Debias generalize robustly across diverse evaluations beyond the training distribution, spanning open-ended fairness, jailbreak attacks, and implicit bias, without requiring test-time debiasing prompts. Together, these findings provide new mechanistic insights and a parameter-efficient approach toward developing fair and trustworthy AI systems.

cs.SI↗

arXiv2026

A Theoretical Perspective on Link Prediction Heuristics

In network analysis, nodes with different degrees are usually weighted differently in tasks such as link prediction. For example, two individuals connected to the same high-degree node (e.g., a celebrity) are often considered less likely to know each other than if they share a low-degree connection. In this paper, we theoretically justify the reasoning behind this heuristic using a graph generation model. In this model, nodes are embedded in a latent metric space, and edges are more likely to form between nearby nodes. We analytically derive degree-based weighting schemes for commonly used link prediction and similarity measures. In particular, we identify the optimal weighting scheme for the Common Neighbors index and extend our analysis to other popular heuristics, including the graph distance. Our theoretical findings align with previous empirical observations, and we further validate them through experiments on both synthetic and real-world networks.

cs.SI↗

arXiv2026

Optimizing Marketing Budgets for Social Influence: A Generalized Fractional Incentives Framework

Marketing managers routinely allocate limited budgets across individuals in social networks whose responsiveness and influence vary substantially. We study the problem of Budgeted Generalized Fractional Influence Maximization (BGFIM), which captures the allocation of continuous levels of marketing spend across users in the presence of heterogeneous response and network spillovers. In this framework, adoption probabilities are governed by user-specific response functions that capture both baseline propensity and marginal responsiveness, while incentive costs vary across users. We propose a scalable solution approach based on a discrete greedy procedure with provable approximation guarantees, enabling tractable optimization under realistic budget constraints. Our formulation generalizes prior influence maximization models by allowing heterogeneous response and cost structures, providing a more flexible representation of targeted marketing interventions. Using experiments on multiple large-scale networks, we show that continuous allocation strategies can outperform standard heuristics and improve influence outcomes under heterogeneous conditions, and we illustrate how the framework can support profit-oriented decision making by identifying budget levels that balance marginal returns and costs. Overall, our results highlight the value of fine-grained budget allocation in networked marketing environments and offer a scalable approach for data-driven marketing resource allocation.

cs.SI↗