arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.25233

Noise-Awareness in Quantum Software Testing: An Empirical Landscape Analysis

Mir Mohammad Yousuf·Shabir Ahmad Sofi·Bisma Majid
View original source ↗Read source document ↗Export metadata ↓

Abstract

Noisy Intermediate-Scale Quantum (NISQ) hardware has transformed quantum computing into a practical experimental platform while introducing fundamental challenges associated with hardware noise, decoherence, and probabilistic execution. Although quantum software testing (QST) has rapidly emerged as an active research area, it remains unclear to what extent existing testing methodologies explicitly engage with noisy and hardware-constrained execution environments. This paper presents a systematic empirical landscape analysis of noise-awareness in QST research through a deterministic, taxonomy-guided analytical framework applied to 61 papers published between 2018 and 2025 across eight testing paradigms, contrasting how contributions are framed with how they are evaluated. Among the analyzed papers, 47 (77.0%) are noise-oblivious and only 14 (23.0%) explicitly engage with noise-related execution concerns, while only 11 (18.0%) report evaluation on physical hardware or a noisy simulator. Test generation, the largest paradigm, exhibits a noise-awareness rate of only 14.3%, and debugging and repair 18.2%; coverage-based, metamorphic, and differential testing contain no noise-aware contributions. Noise-aware methodologies emerged late and remain outnumbered in all analyzed years. Framing and evaluation are strongly associated (odds ratio 40.5, p < 0.001) yet diverge for individual papers: 5 of the 14 noise-aware papers evaluate only on ideal simulators. Inferential statistical tests are reported in only 16.4% of papers. The classification framework was validated against a blinded, human-coded reference sample, achieving almost perfect inter-rater agreement on testing-technique classification and substantial agreement on noise-awareness. The framework provides a reusable empirical basis for longitudinal analysis of hardware-conscious testing practice.

Explore related subjects

cs.SEquant-ph

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Mir Mohammad Yousuf, Shabir Ahmad Sofi, Bisma Majid. 2026-10-18. Noise-Awareness in Quantum Software Testing: An Empirical Landscape Analysis. https://arxiv.org/abs/2610.25233

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 305 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 15.4% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 54% of cases.

cs.SE↗
arXiv2026

LLM-as-a-Judge for Contest-Style Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding

Large language models now judge open-ended outputs and co-create code with programmers, yet judgments of human-AI co-creation must be reliable, comparable across models, and interpretable over multi-turn interaction. This work presents a rubric-driven LLM-as-a-Judge framework for contest-style human-AI co-creation in coding, evaluated on 15 participants, 13 problems, 184 trajectories, and 517 attempts from an AI-permitted track run in parallel with a programming contest. Four judges are evaluated under grouped cross-validation with cluster-bootstrap 95% intervals and probed by injecting single defects into accepted programs against behavior-preserving controls. Three findings concern the judges as instruments and likely extend beyond this setting. First, the judge input is a first-order design variable: supplying the problem statement in place of an identifier rebuilds the judges' ranking (rank correlation 0.2617 [0.123, 0.391] for the three judges whose model was unchanged) and raises their ROC-AUC by 0.130 [0.061, 0.202] to 0.295 [0.214, 0.381]. Second, schema compliance does not imply scale validity: the judges emit as few as 8 distinct probability values, leaving as few as 6 of 99 thresholds effective. Third, accuracy and defect sensitivity are separable: Claude and Gemini are indistinguishable on accuracy (ROC-AUC 0.6985 [0.633, 0.763] against 0.7691 [0.719, 0.817]) yet separate on defect detection (0.8380 [0.806, 0.871] against 0.7512 [0.719, 0.783]). For co-creation, Success@Turn rises from 0.6033 [0.533, 0.674] at the first attempt to 0.8370 [0.783, 0.886] by the tenth, 44 of the 73 trajectories unsolved at first (60.3%) succeed later, and revision size is unrelated to judged improvement (Spearman 0.0007 [-0.085, +0.090]).

cs.SE↗
arXiv2026

Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing

Automated generation of end-to-end web tests remains costly: developers must identify stable selectors, encode multi-step workflows, attach assertions, and maintain scripts as interfaces evolve. Recent LLM-based web agents can complete natural-language tasks in a live browser session, but task completion is not the same as producing a test-ware artifact - code that is executable, replay-able from a reset state, inspectable by developers, and tied to an oracle. We present TESSARA, an adaptive test-script generation framework that converts natural-language test intent into Playwright scripts consisting of executable interaction code, goal-state assertions, and step-level provenance linking each step to its generated code fragment, selector evidence, execution outcome, and repair history. TESSARA grounds each planned step via Step-Aligned Scripting (SAS), enabling fine-grained failure localization, selective regeneration of failing suffixes, and reversibility-aware backtracking that avoids undoing likely commit points (e.g., form submissions). A three-level verification pipeline (syntactic, execution, and semantic/oracle), detects and repairs malformed code, interaction failures, and incorrect landing states. We evaluate TESSARA on three public web benchmarks, measuring script executability, oracle satisfaction, replay stability, and repair efficiency, and show substantial improvements over baselines and underlying VLMs used directly as script generators.

cs.SE↗