arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.35978

Testing Agentic Service Systems: An Agentic Living Lab Methodology Across the Physical-Digital Continuum

Zhongjie Wang·Xiang He·Xiao Wang·Mingyi Liu·Shuang Yu·Hanchuan Xu
View original source ↗Read source document ↗Export metadata ↓

Abstract

Task success is insufficient for testing agents that act for principals, change persistent business state, and affect physical processes: a useful result may be unauthorized, leave obligations unresolved, or depend on the execution substrate. We propose an Agentic Living Lab methodology organized around three questions: what is the test unit, who and what participate, and under what reality is the test executed? The unit is a service episode containing delegated roles, obligations, world state, and evidence. The method separates agents under test from environment actors and distinguishes real humans, human traces, rule surrogates, and agent-based proxies. Realization Profiles, capability-declared Reality Substrate bindings, Grounding Profiles, and Semantic Delta disclosures make changes across software, enterprise systems, simulation, and physical settings explicit. A four-plane reference architecture separates experiment control, participants, world realization, and evidence evaluation, with cross-cutting authority and governance controls. Runtime semantics distinguish intent, normative admission, and factual reconciliation. Real stakeholder grounding, participant calibration or validation, and governed feedback are required for the Living Lab claim, but not synchronous human participation in every run. Released JSON Schemas, deterministic fixtures, and a limited reference runner make selected rules machine-checkable; they are not a complete platform or empirical validation. Constructed travel and industrial designs illustrate authorization and fulfillment failures, proxy sensitivity, and substrate changes. Agent quality, proxy validity, and cross-substrate effect stability remain empirical questions.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zhongjie Wang, Xiang He, Xiao Wang, Mingyi Liu, Shuang Yu, Hanchuan Xu. 2026-10-26. Testing Agentic Service Systems: An Agentic Living Lab Methodology Across the Physical-Digital Continuum. https://arxiv.org/abs/2610.35978

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.

cs.SE↗
arXiv2026

From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow

Scientific applications still rely heavily on legacy Fortran codebases built for homogeneous, CPU-based systems. As High-Performance Computing (HPC) shifts toward GPU-accelerated architectures, many accelerators lack native Fortran bindings, creating a need to translate and optimize legacy code for portability. Kokkos offers performance portability through a single-source C++ abstraction, but manual porting from Fortran demands significant expertise and time. While large language models (LLMs) show promise in source-to-source code generation, their use in autonomous workflows for translating and optimizing parallel code is largely unexplored. This paper presents an agentic AI workflow in which specialized LLM agents collaborate to translate, validate, compile, run, test, debug, and optimize Fortran kernels into portable Kokkos C++ programs without human intervention, bounded by configurable retry and optimization thresholds. We evaluate the pipeline over five runs per configuration on an NVIDIA A100, characterizing its variability in success rates, token cost, and optimization behavior, on benchmark kernels including a real-world multi-file kernel from a production climate model. Proprietary models such as GPT-5 and o4-mini-high completed the full workflow most reliably, producing optimized code that exceeded the Fortran baselines for only a few U.S. dollars, whereas open-weights models such as gpt-oss-120b and Qwen3.5-397B-A17B handled simpler kernels but struggled with complex Kokkos abstractions such as parallel reductions and hierarchical parallelism. This work demonstrates, at the kernel level, the feasibility of agentic AI for Fortran-to-Kokkos transformation and, more broadly, for structured software-engineering tasks such as automated code modernization and refactoring.

cs.SE↗
arXiv2026

ScratchWorld: Evaluating If World Models Compute Executable Consequences

Evaluations of world models commonly score a predicted future state by its overlap with the ground truth. Yet in interactive worlds, this rewards copying: a model that merely preserves persistent state appears accurate without predicting any consequence of an action. We introduce ScratchWorld, an execution-grounded diagnostic benchmark that treats Scratch projects as executable worlds and derives every label from replay on a pinned Scratch VM, including next-state transitions, hidden variables, causal traces, and counterfactual outcomes. Its primary metric, value-aware changed-field F1 ($F_1^{\mathrm{VA}}$), credits a prediction only when it names a changed field and computes its executed value. Model-free anchors make the confound explicit: a no-change prior attains 91.4% full-state field accuracy on synthetic long-horizon items and 0% changed-field credit on non-empty one-step items, whereas VM replay attains 100% wherever the corresponding metric is defined. Across seven frontier LLMs prompted as world models, state-only $F_1^{\mathrm{VA}}$ reaches at most 30.0% on controlled synthetic mechanisms and declines steeply as projects approach real-world complexity. Revealing the full program raises every model's value-aware score by 22.3-51.7 points and substantially reorders the models, yet the best reaches only 55.4%, an execution gap that persists even with the rules visible. Auxiliary diagnostics complete the error profile: causal-event accuracy remains within 2.5 points of the 50% no-event baseline, with no correct exact chain; counterfactual prediction trades changed-outcome accuracy against invariance; hidden-state accuracy degrades roughly twice as fast as full-state overlap over long horizons. In a word, ScratchWorld separates what a model knows about the rules from what it can actually execute.

cs.SE↗