Muhammad Bilal | arXiv Science

arXiv2026

LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review

Large language models (LLMs) increasingly decide whether software behaves correctly, either by writing a test oracle or by acting as one. Yet two oracles can look identical and rest on different ground: one assertion encodes a written specification, another only what the model learned in training. Prior secondary studies sort oracles by form or by technique, rarely by the property that governs how far a verdict can be trusted: where its authority comes from. This systematic literature review, reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, screens 2,436 records to 54 included studies, extended by citation searching (snowballing) to 83 in total. We read the corpus along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. Just over half of the corpus reaches a verdict with no specification at all. That is what lets these oracles work on code with no specification to consult, and what leaves a challenged verdict with less to fall back on. Source and mechanism cross-cut rather than coincide, so a label such as LLM-as-a-judge names how a verdict is produced, not why it should be trusted. Oracle quality is most often judged by resemblance to a known oracle rather than by whether injected faults are caught. The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict. The protocol, search query, and per-study coding sheet are released.

cs.SE↗

arXiv2026

Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety

Large language models (LLMs) are increasingly being used in network operations (NetOps) and artificial intelligence for IT operations (AIOps) for tasks ranging from telemetry retrieval and incident diagnosis to configuration planning and bounded remediation. As these systems acquire greater access to operational tools, the central question is no longer only what an LLM can do, but whether operational assurance increases commensurately with the authority granted to it. This survey examines that question through a structured, evidence-stratified review of agentic NetOps and AIOps. We organise the field around autonomy, tool scope, evidence traces, assurance controls, evaluation, security, and governance, and introduce an operational assurance contract that links each autonomy level to permitted tools, required evidence, independent gates, execution budgets, rollout and rollback duties, and audit requirements. The synthesis reveals a capability--assurance gap: evidence is comparatively strong for read-oriented assistance and tool-grounded diagnosis, but becomes substantially less complete as systems approach configuration change, bounded execution, and closed-loop operation. We therefore argue that evaluation should move beyond static question answering and model accuracy towards workflow-level assessment of evidence quality, tool use, policy and invariant compliance, staged execution, recovery, calibration, cost, and human intervention. We also examine prompt-borne attacks, poisoned or stale operational evidence, excessive agency, privilege boundaries, and weak auditability. Taken together, the survey frames agentic NetOps and AIOps as constrained operational control, in which useful autonomy depends on independently enforced assurance rather than model capability alone.

cs.NI↗