arXiv Science⌕ Search

arXiv · 2611.23335

Truthgate: Checking Coding Agents Against a Record They Cannot Edit - Cheating, Deterrence and Disclosure on Impossible Tasks

Abstract

When a coding agent says it is done, is that true, and how would anyone know? We study this question in five steps; every paid study was pre-registered, and in each step the agent is checked against a record it cannot edit: tests re-run by the host and, from step 2 on, hidden checks in containers the agent never sees and a signed log of every tool call written by a signer outside the agent's sandbox. A pilot with a small local model (Qwen3.5-4B, 36 runs on impossible tasks) found no hidden failure, and showed that labels built from actions alone over-count lies. On Impossible-LiveCodeBench, Claude Sonnet 5.5 passed 39 of 180 impossible tasks (21.7%), every time by special-casing its code to fit the failing assertion; Opus 5.5 (2.8%) and Haiku 5.5 (1.7%) rarely did. One sentence appended to the system prompt cut Sonnet's cheats on the same 60 tasks from 22 to 8 (an AI reviewer will read the transcript), 6 (a person will review the work), 6 ("This session is recorded.") and 1 (do not special-case tests), all significant under exact McNemar tests, with honest passes unchanged. No remaining cheat moved to a shell falsely described as unrecorded (0 of 4). Asked for a final report, every one of 55 natural cheats named its special case; unasked, Sonnet and Opus said nothing in 52 of 52. An LLM judge caught all 55; deterministic rules over the host record caught 27. A scripted tamper demonstration, with keyword stand-ins for the judges, shows that one rewrite of the agent-side transcript hides two calls from a judge that reads it, while the signed record keeps them; in a 40-rollout pilot, Sonnet never touched its transcript. The judge and the record caught different things, which leads to our main conclusion: the record is the evidence a judge should read, not a replacement for one. All pre-registrations, results documents, code and evidence manifests are released.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Bravish Ghosh. 2026-11-17. Truthgate: Checking Coding Agents Against a Record They Cannot Edit - Cheating, Deterrence and Disclosure on Impossible Tasks. https://arxiv.science/abs/2611.23335

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Adaptive Finite State Projection with Quantile-Based Pruning for Solving the Chemical Master Equation

We present an adaptive Finite State Projection (FSP) method for efficiently solving the Chemical Master Equation (CME) with rigorous error control. Our approach integrates time-stepping with dynamic state-space truncation, balancing accuracy and computational cost. Krylov subspace methods approximate the matrix exponential, while quantile-based pruning controls state-space growth by removing low-probability states. Theoretical error bounds ensure that the truncation error remains bounded by the pruned mass at each step, which is user-controlled, and does not propagate forward in time. Numerical experiments on biochemical systems, including the Lotka-Volterra and Michaelis-Menten and bi-stable toggle switch models.

cs.CE↗

SoK: Market Microstructure for Decentralized Prediction Markets (DePMs)

Decentralized prediction markets (DePMs) allow open participation in event-based wagering without fully relying on centralized intermediaries. We review the history of DePMs which date back to 2011 and includes hundreds of proposals. Perhaps surprising, modern DePMs like Polymarket deviate materially from earlier designs like Truthcoin and Augur v1. We use our review to present a modular workflow comprising eight stages: underlying infrastructure, market topic, share structure and pricing, market initialization, trading, market resolution, settlement, and archiving. For each module, we enumerate the design variants, analyzing trade-offs around decentralization, expressiveness, and manipulation resistance. We also identify open problems for researchers interested in this ecosystem.

cs.CE↗

A Multigrid Finite State Projection Algorithm For The Solution Of The Reaction-Diffusion Master Equation

We develop a multigrid method for the finite state projection of the reaction-diffusion master equation (RDME). The RDME evolves the joint probabilities of molecular count configurations across well-mixed voxels coupled by local reactions and diffusive jumps. When fast diffusion or large reaction rates make this evolution stiff over long time horizons, implicit time stepping requires solving a large shifted linear system at every step. Grouping neighboring voxels and summing their molecular counts produces a hierarchy of coarser state spaces terminating at a one-voxel chemical master equation. Restriction sums residual entries with the same coarse counts, while multinomial prolongation preserves the RDME generator form on every level with explicit coarse reaction and diffusion rates. For reversible diffusion, the error omitted by coarsening is bounded by the graph Dirichlet energy of jumps inside the aggregates, with a constant independent of population. Numerical tests with standard multigrid cycles and point or block Gauss-Seidel smoothing reach 7.16 million states and resolve rare probabilities of order $10^{-7}$.

cs.CE↗