arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.14356

Actionable Meta-Requirements for AI Robustness Auditing

Joshua Owotogbe·Roya Nasiri·Indika Kumara·Damian Andrew Tamburri·Willem-Jan van den Heuvel
View original source ↗Read source document ↗Export metadata ↓

Abstract

AI-enabled systems must remain dependable when the conditions under which they operate change (e.g., due to input perturbations, distribution shifts, or changes in external dependencies). This makes robustness auditing of such systems important for judging whether the available evidence is sufficient to support a defined robustness conclusion. However, designing an AI robustness auditing framework is challenging. A key unresolved issue is determining the requirements such a framework should support; these requirements have not yet been consolidated. We address this gap through meta-requirements: higher-order requirements specifying what a class of auditing frameworks should support. In design science research, literature and domain experts are common sources of meta-requirements; accordingly, we derive the initial set from literature and validate and refine it with practitioners. Our systematic literature review retained 74 of 2,890 records and derived 43 meta-requirements across 16 categories. We evaluated them with 128 practitioners, obtaining 5,504 ratings and 586 substantive comments. Overall, 81.20% of ratings indicated agreement or strong agreement. In particular, 41 of the 43 literature-derived meta-requirements were classified as strongly supported. Practitioner feedback refined the requirements and identified an additional requirement concerning external components and supply chain dependencies. The final set comprises 44 meta-requirements across 16 categories, of which 43 received quantitative validation. These meta-requirements can help organizations implement AI robustness auditing frameworks, assess existing frameworks for completeness, and refine them.

Explore related subjects

cs.AIcs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Joshua Owotogbe, Roya Nasiri, Indika Kumara, Damian Andrew Tamburri, Willem-Jan van den Heuvel. 2026-10-09. Actionable Meta-Requirements for AI Robustness Auditing. https://arxiv.org/abs/2610.14356

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Python Tool for Visualizing Variability of Pareto Fronts over Multiple Runs

Hyperparameter optimization is crucial to achieving high performance in deep learning. On top of the performance, other criteria such as inference time or memory requirement often need to be optimized due to some practical reasons. This motivates research on multi-objective optimization (MOO). However, Pareto fronts of MOO methods are often shown without considering the variability caused by random seeds and this makes the performance stability evaluation difficult. Although there is a concept named \emph{empirical attainment surface} to enable the visualization with uncertainty over multiple runs, there is no major Python package for empirical attainment surface. We, therefore, develop a Python package for this purpose and describe the usage. The package is available at https://github.com/nabenabe0928/empirical-attainment-func. OptunaHub now provides our implementation at https://hub.optuna.org/visualization/plot_empirical_attainment_surface/.

cs.AI↗
arXiv2026

From Language Models to Control Policies: Search, Selection, and Verification

Industrial process control demands policies that are interpretable and auditable, requirements that black-box neural policies struggle to meet. LLM-driven heuristic synthesis offers an alternative: a language model iteratively proposes and refines human-readable Python controllers using rich behavioral feedback from a simulator. We study this form of programmatic policy search on hot steel rolling, a physics-simulated, multi-pass control problem with competing objectives and hard equipment constraints, and address three questions that arise once the search is run at scale: how to split a fixed budget of LLM iterations between restarting and refining, how to deploy the resulting pool of heuristics, and how to verify it. Treating each search run as a randomized algorithm, we find across 52 independent runs that runs of moderate length match the hindsight-optimal budget allocation, and that Luby-style universal restarts reach this regime without problem-specific tuning. Because each heuristic is cheap to simulate, a small portfolio selected per job outperforms the best single heuristic, also under cross-validation and on out-of-distribution operating conditions on which the best heuristic stalls. Finally, an automated audit pipeline combining static analysis, Z3 proofs, and property-based testing proves key safety properties for all 174 synthesized heuristics; the stall, which neither the search reward nor these properties reveal, shows that the specifications must also cover progress.

cs.AI↗
arXiv2026

Virtual Speech Therapist: A Clinician-in-the-Loop AI Agent for Speech Therapy Planning

This paper presents a proof of concept of the Virtual Speech Therapist (VST), an intelligent multi-agent based system designed to support stuttering assessment and individualized therapy planning through artificial intelligence (AI) driven workflows. VST integrates deep learning (DL) based stuttering detection fSD with large language model (LLM) based agentic system to assist clinicians in developing evidence informed therapy plans. The VST system begins with the acquisition and feature extraction of given speech samples, followed by the detection of stuttering types. Building on these speech driven information S, VST initiates an agentic reasoning process in which specialized LLM agents autonomously generate, critique, and iteratively refine individualized therapy plans P. A dedicated CriticAgent πca acts as a structured self-refinement mechanism, reviewing the generated therapy plans P and identifying areas for potential improvement in consistency, methodological soundness, and alignment with available evidence and professional guidelines. The resulting therapy plan P is presented then as a draft for clinician review. Clinician feedback can then be incorporated to produce a revised plan P, thereby maintaining a clinician-in-the-loop (CITL) paradigm. An initial evaluation, involving expert speech language pathologist (SLP) examines the clinical interpretability and feasibility of the generated therapy plans. Taken together, the results show the potential of VST to augment clinician-led therapy planning and provide a promising basis for future studies evaluating its integration into real world clinical settings. An interactive user interface for the proposed prototype is available online at: https://vocametrix.com/ai/stuttering-therapy-planning-agent, facilitating interactive stuttering assessment and stuttering therapy planning.

cs.AI↗