arXiv Science⌕ Search

arXiv subjects

Sungjin Kwon

Publications and source records attributed to Sungjin Kwon.

2 recordsLinked to original sources

A Governance Methodology Layer for AI-Assisted Software Development: Defect Taxonomy, Controlled Ablation, and a Test of Process-Over-Capability

Autonomous coding agents produce output that passes syntactic checks -- compilation, type safety, CI -- at high velocity. Yet syntactic correctness does not imply semantic correctness: design boundaries, security invariants, and maintainability contracts remain structurally invisible to automated pipelines. This paper makes four contributions. First, we present a defect-class taxonomy grounded in five AI agent permission and governance modules, separating defects structurally detectable by static analysis from those requiring semantic review. Second, we describe a runtime-decoupled governance gate -- a file-based protocol that reads generator output and emits a structured verdict without API coupling, hence portable across generators. Third, we formalize methodology-as-code: expressing a verification protocol as a version-controlled, executable artifact whose two layers separate portable methodology from host automation. Fourth, we report a controlled ablation experiment (E-ablation, N=5 artifacts, 8-item independent ground truth) comparing harness-structured review against a token-matched unstructured prompt. The structured condition records 62% lenient recall against 50%, with 25% strict against 0%. A severity-grade differential reported earlier does not survive blind re-grading and is withdrawn (Sec. 6.6). An independent-session re-test with blind scoring does not replicate that contrast: the conditions differ by one strict hit in 24 (6/24 against 5/24), the structured aggregate again 25% and the unstructured 0% not recurring (Sec. 6.7). Both conditions miss document-quality defects identified by a human QA reviewer, indicating complementarity between structured AI review and human process inspection. A later pre-registered test varying model tier (184 runs) did not support process design as the dominant factor: under strict scoring the tier effect was larger (Sec. 6.8).

cs.SE↗

We Fixed the Scorer First, and Cluster Correction Killed Our Own Planted-Defect Result in AI Code Review

Judging an AI code-review pipeline only by how much it catches hides an axis: what it dropped. Stronger selection raises precision for free; the cost never reaches the report. We ran four arms on the same eight cases, one scorer, asking whether planted-defect detection can rise without raising the claim error rate. Three repetitions are nested in eight cases, so p is reported per contrast and twice: under an exact permutation test over cases (2^8 arrangements) the two contrasts publishable under independence (Fisher p=0.017, 0.019) move to 0.0625 and 0.109, and after Holm correction no contrast separates. This is "not observable with this design," not "no effect": the sign test needs six discordant cases in one direction, its realized type-I rate under the null is 0.0078, and 80% power needs a per-case win probability near 0.80; closing it takes 12-25 cases, not repetitions. Within a run, a second pass never removes what the first found (0/24, 0/24 paired losses) while its gains (+3, +2) fall in one case; baselines on byte-identical prompts differed 15/24 vs 20/24, so run-to-run variance exceeds the effect. The claim error rate is 0.0% (0/178), with no upper bound on the rate claimed; scorer discrimination was measured with pre-registered positive controls (12/12 fabricated claims rejected, 0 false rejections). We retract our original causal explanation and report the instrument defects we hit. The contributions are not the numbers: (1) fixing the instrument before seeing results, then killing our own main effect with it; (2) the identity eps-hat = eps-hat(pi): a reported error rate is computed only on the subset a retention policy kept, silent about what was dropped, so it does not identify the quantity an irreversible gate needs; no improvement in the number promotes an automated verdict to publish, delete, or history-rewrite surfaces; a mechanized refusal enforces this.

cs.SE↗