arXiv subjects

Zimu Liu

Publications and source records attributed to Zimu Liu.

3 recordsLinked to original sources

arXiv2026

Can LLM Agents Learn to Test? Maintaining Scoped Experience as Testing Evidence for Mobile GUI Testing

Repeated mobile GUI tests can reuse routes, locators, and recovery strategies from earlier executions. However, a route that worked in one workflow or app version may be irrelevant or misleading in another. We study how an LLM-based tester can retain useful experience while withholding entries contradicted by current observations. We present a maintenance contract for AdaptiTest in which reusable experience has an explicit applicability scope, trace provenance, and a lifecycle for confirmation, revalidation, and withdrawal. The contract reserves persistence for a Reflector role and treats recalled experience as advice. The archived implementation does not establish end-to-end enforcement of every specified mechanism. The design changes retrieved context, not model weights. The evaluation uses read-only Android workflows and archived internal comparisons. A six-task, three-repeat ready subset reports 18/18 completed runs with memory versus 3/18 without it, but a grounded single-agent control also solves the earlier hard-task slice. A separate cross-session probe is excluded from efficacy conclusions because of confounded route schedules and unvalidated state labels. We examine 13 selected real-update hazard records across nine subjects, distinguishing stale-locator observations from policy-based withholding and offline retrieval diagnostics. The latter do not measure live-device failures or actions saved. These events are UI changes, not confirmed regression bugs. The evidence supports a bounded design and case study; it does not establish general efficacy, superiority over independent GUI-agent systems, or unrestricted exploration.

cs.SE↗

arXiv2026

When Does a Cross-Device Run Justify PASS? Obligation-Level Evidence Qualification under Partial Observability

A cross-device test can complete every command yet judge a business claim from evidence belonging to another device or attempt. We introduce an executable evidence contract that makes support qualification an explicit stage before predicate evaluation. It binds each obligation to admissible subjects, episodes, objects and observation scopes, and records unresolved support rather than treating an apparently successful observation as proof. We evaluate two implemented parts of the contract. A prospectively specified, 12-seed controlled study isolates association under reused identifiers across 2,304 claims. Given identical records and business policy, Atomic-Facts Flat produces 160 false PASS and 76.30% PASS recall; supplying computed associations reduces false PASS to five and increases PASS recall to 99.09%. Deterministic Closure produces zero false PASS and 100.00% PASS recall, improving on Atomic Flat in 11 seeds with one tie (two-sided sign p=0.00098). Most of the benefit comes from explicit association rather than graph notation. A separate prospective field characterization spans four workflows, 40 physical episodes and 80 claims. Against frozen reference labels, the physical-evidence verifier attains 95.00% accuracy and 100.00% PASS recall (27/27), with three false PASS judgments. The field results are descriptive: a policy-input mismatch prevents the planned matched-policy comparison, and post-hoc sensitivity does not establish superiority over a policy-informed textual baseline. Together, the studies characterize qualification on physical evidence and isolate the value of explicit association under a controlled matching contract.

cs.SE↗

arXiv2024

ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models

There is a growing interest in the role that LLMs play in chemistry which lead to an increased focus on the development of LLMs benchmarks tailored to chemical domains to assess the performance of LLMs across a spectrum of chemical tasks varying in type and complexity. However, existing benchmarks in this domain fail to adequately meet the specific requirements of chemical research professionals. To this end, we propose \textbf{\textit{ChemEval}}, which provides a comprehensive assessment of the capabilities of LLMs across a wide range of chemical domain tasks. Specifically, ChemEval identified 4 crucial progressive levels in chemistry, assessing 12 dimensions of LLMs across 42 distinct chemical tasks which are informed by open-source data and the data meticulously crafted by chemical experts, ensuring that the tasks have practical value and can effectively evaluate the capabilities of LLMs. In the experiment, we evaluate 12 mainstream LLMs on ChemEval under zero-shot and few-shot learning contexts, which included carefully selected demonstration examples and carefully designed prompts. The results show that while general LLMs like GPT-4 and Claude-3.5 excel in literature understanding and instruction following, they fall short in tasks demanding advanced chemical knowledge. Conversely, specialized LLMs exhibit enhanced chemical competencies, albeit with reduced literary comprehension. This suggests that LLMs have significant potential for enhancement when tackling sophisticated tasks in the field of chemistry. We believe our work will facilitate the exploration of their potential to drive progress in chemistry. Our benchmark and analysis will be available at {\color{blue} \url{https://github.com/USTC-StarTeam/ChemEval}}.

cs.CL↗