When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence

Ali Shakiba

Home/Archive/arXiv

arXiv · 2610.36413

When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence

Ali Shakiba

View original source ↗Read source document ↗Export metadata ↓

Abstract

Generative AI can produce code, tests, documentation, and technical explanations that resemble artefacts traditionally used as evidence of student competence, weakening the inference from artefact quality to the engineering capability behind it. We report the redesign and first deployment of a 13-week, AI-permissive, software-intensive engineering course in which students may delegate substantial work to AI while retaining responsibility for planning, verification, review, integration, and explanation. The redesign couples an engineering accountability workflow, which structures human-AI work and preserves human decision authority, with an assessment evidence architecture comprising six complementary channels: product, process/origin, verification, explanation, reflection, and individual assurance. A collect--cross-check--escalate (CCE) reasoning logic connects these channels rather than treating any single artefact or trace as sufficient evidence of competence. By the end of Week 11, the deployment involved 87 students organised into 17 teams, with 957 individual weekly Git repositories and 34 group-project Git repositories supporting repeated assessment and engineering activity. The first deployment yielded four practical lessons: AI assistance changes where rubric criteria carry evidential weight; multiple evidence channels require deliberate cross-checking and meaningful complementarity; the evidential strength realised in practice may differ from that intended in the assessment design; and automation scales evidence collection more readily than interpretation. The contribution is a conceptually grounded, deployed design for making engineering responsibility and competence more visible when AI can perform substantial artefact production.

Explore related subjects

cs.SE cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Ali Shakiba. 2026-10-26. When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence. https://arxiv.org/abs/2610.36413

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

Large language models (LLMs) can describe what a piece of code should compute, but they do so in free-form text. Such text cannot be checked against program executions, and it does not reveal whether a statement is correct. We present SemLoc, which turns LLM reasoning into executable, verifiable semantic constraints. With structured semantic grounding, the LLM writes each property as a constraint with a fixed category, attached to an anchor in the code graph, such as a variable definition, branch, loop, or return. SemLoc inserts each constraint as a runtime check and runs the tests, which yields a semantic violation spectrum over passing and failing tests. Counterfactual verification then patches the code at an anchor and reruns the tests; a patch that a constraint suggests counts as verified only if all tests then pass. We study SemLoc on fault localization, where the tests serve as the oracle for verification, with four LLMs on QuixBugs and on SemBugs, a new dataset of 809 real bugs from 750 GitHub projects. The constraints are executable: 99.4% become runtime checks, and on programs where coverage ties the faulty line with at least half of the covered lines, their violations alone rank it first in 40.9% of the cases, compared with 2.2% for spectrum-based fault localization and 7.0% for Daikon invariants. They make the LLM's answers verifiable: 68.4% of SemLoc's top-ranked answers carry a patch that the tests confirm, and 93.6% of these are correct, which the LLM's own confidence cannot match. SemLoc's first-rank accuracy does not differ significantly, for any of the four LLMs, from that of an LLM given the same inputs (81.4% vs. 82.0%), and it far exceeds that of an LLM that sees only the function (50.0%).

cs.SE↗

arXiv2026

Choosing Coding-Agent Benchmarks for Practice: Lessons from Task Demands and Agent Behaviour

Software teams evaluating coding agents need evidence that reflects the changes they expect those agents to make. Benchmark scores summarize how often an agent resolves tasks under broad labels such as bug fixing and feature implementation, but provide little guidance on whether the evaluated work resembles a team's workload. To support this decision, we characterize 2,487 tasks from five public benchmarks and analyze 14,922 runs of six agent configurations. We introduce the Spread-Novelty-Centrality (SNC) profile to describe how the reference solution is distributed across files and directories, how much it adds relative to what it removes, and the dependencies, history, and complexity of the code it touches. We also compare the scope of agent patches and their file exploration. Every pair of benchmarks differs on at least two SNC axes, including benchmarks with the same task label. Among tasks resolved by Claude Opus 4.6, patches are larger than the reference on FeatBench and smaller on FeatureBench. Across the six configurations, resolved runs tend to involve less distributed changes and code with lower values on the Centrality indicators, while the patch sizes associated with resolution differ between the Claude and Qwen configurations. These observations suggest that practitioners should select evaluation tasks based on their change characteristics, inspect the information provided in task descriptions, and assess patch scope alongside test outcomes. We explain how teams can apply this analysis to completed changes and identify the additional evidence needed before using it for task assignment or review automation.

cs.SE↗

arXiv2026

Knowing When to Stop: The Evidence-Action Gap in Coding Agents

Coding agents are trained to complete tasks, and they keep trying even when the environment makes completion impossible. The evidence that only outside help can restore progress usually appears within a few steps, yet agents rarely act on it: they over-persist, attempt undesired workarounds, and claim success the record does not support -- a phenomenon we call evidence-action gap. To measure it, we introduce ImpasseBench, which injects twelve realistic environment blockers into otherwise solvable repair tasks and marks, in each trajectory, the step at which the collected evidence becomes sufficient. Scoring actions relative to that step separates appropriate effort from over-persistence and premature handoff. Across eight frontier models and 19,371 trajectories, we find that nearly every run that needs outside help collects sufficient evidence, yet a third still exhaust their budget, and every unsafe request we detect comes after that point. Fortunately, an escalation tool, or an external monitor for models that ignore the tool, already guards against much of this behavior: the tool cuts budget exhaustion from 34.0% to 6.7%, and the monitor nearly triples task success after environment restoration. Even then, a fired escalation is not yet a useful handoff: tickets reliably name the blocker but often omit the agent's own progress and verification status. Taken together, we argue that knowing when to stop therefore needs explicit support, both in the tool that lets an agent hand off and in the report it leaves behind.

cs.SE↗