arXiv subjects

Zibin Zheng

Publications and source records attributed to Zibin Zheng.

At least 37 records · Page 2Linked to original sources

arXiv2026

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.

cs.SE↗

arXiv2026

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.

cs.SE↗

arXiv2026

SWE-Prime: Fewer Trajectories, Better Performance

To improve large language models' ability to resolve real-world software issues, prior work has focused on constructing large-scale agent trajectory datasets and performing supervised fine-tuning (SFT) on successful trajectories. However, task success does not guarantee high-quality supervision: successful trajectories may still contain ineffective, redundant, or risky steps. Directly using such trajectories for SFT can introduce noisy supervision and encourage models to imitate undesirable problem-solving behaviors. Therefore, we propose SWE-Prime, a multi-granularity, two-stage SFT data selection method that progressively filters training data at the trajectory and segment levels. Specifically, the first stage performs trajectory-level screening based on process quality, result quality, and data representativeness, selecting a high-quality and representative subset of successful trajectories. The second stage performs segment-level selection by grouping consecutive steps into semantic segments and assessing each segment based on its contribution to the final solution, learnability, and potential risks. During SFT, all segments remain in the sequence to preserve context, while only selected segments contribute to the loss computation. Experiments on SWE-Bench Pro and SWE-Bench Verified show that training on the 10% trajectory subset selected by SWE-Prime outperforms training on the full resolved dataset, yielding relative performance gains of up to 12.2% and 24.2%, respectively.

cs.SE↗

arXiv2026

IcFuzz: Fuzzing Isaac Sim with Semantic Stage Guidance and Multi-level Mutation

Robotics simulators serve as a foundational infrastructure for embodied AI, facilitating safe and scalable robotic system development. NVIDIA Isaac Sim has emerged as one of the most popular simulators, distinguished by its GPU-accelerated physics engine and photorealistic rendering, which enable high-fidelity modeling of complex environments. However, its inherent complexity inevitably introduces software bugs that can compromise simulation reliability. Existing fuzzing approaches struggle to test Isaac Sim effectively due to challenges of context-aware object semantics, hierarchical simulation control, and a vast simulation state space. In this paper, we propose IcFuzz, the first fuzzing approach for Isaac Sim. IcFuzz first performs an LLM-based semantic stage segmentation, decomposing simulation programs into structured stages that capture context-aware object semantics. Guided by this information, IcFuzz designs multi-level mutation operators to systematically exercise the simulator across hierarchical granularities. To efficiently navigate the vast simulation state space, IcFuzz employs a multi-armed bandit algorithm to adaptively schedule mutation operators. Experimental results show that IcFuzz outperforms the baselines in terms of both code coverage and bug detection. Specifically, IcFuzz achieves approximately 190\%--205\% of the code coverage of the baselines and detects an average of 3.7 unique crashes over three rounds of 12-hour tests, while no crashes are detected by the baselines. Moreover, IcFuzz has uncovered 11 bugs over approximately four months, 9 of which have been confirmed or fixed by the developers.

cs.RO↗

arXiv2026

RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

Recent large language models (LLMs) have shown strong performance on software engineering tasks, yet most existing benchmarks evaluate code reasoning at the function level, where all relevant information is localized. This setting fails to reflect real-world development, which requires reasoning across multiple files and complex dependency structures. We introduce RepoReasoner, a benchmark for evaluating repository-level code reasoning. It assesses two complementary abilities: Output Prediction, which measures fine-grained, stateful execution reasoning across files, and Call Chain Prediction, which evaluates high-level architectural dependency understanding under noisy context. Our benchmark is constructed through a multi-stage pipeline that leverages dynamic tracing of pytest executions to obtain ground-truth call chains, along with LLM-based I/O rewriting to reduce memorization effects. We evaluate seven state-of-the-art LLMs. Even under oracle context, the best-performing model achieves only 69.1% Pass@1 on Output Prediction, indicating that cross-file reasoning remains a major challenge. In Call Chain Prediction, models exhibit high precision but low recall, suggesting limited multi-hop dependency understanding. Furthermore, performance drops on rewritten data reveal partial reliance on memorization, and longer contexts do not consistently improve results due to noise. These findings highlight fundamental limitations in current LLMs' repository-level reasoning and motivate future work on structured architectural understanding and cross-file inference.

cs.SE↗

arXiv2026

Scalable and Efficient Joint Spiking Embedding Predictive Architecture for Large-Scale Dynamic Graphs

Dynamic graph learning aims to capture evolving structural and semantic patterns in real-world systems, such as fraud detection and recommender systems. Due to the scarcity of labeled data in real-world dynamic graphs, recent studies have introduced generative or contrastive paradigms (e.g., masked graph autoencoders or graph contrastive learning) to generate task-agnostic graph embeddings. However, these methods typically rely on complex edge-level reconstruction objectives and tailored graph augmentation strategies. This incurs substantial computational overhead when scaling to large-scale dynamic graphs. In this paper, we propose SG-JEPA, a joint spiking embedding predictive architecture for large-scale dynamic graphs. In contrast to existing self-supervised methods, SG-JEPA partitions nodes into context and target sets along the temporal dimension to learn embeddings that are predictive of each other via additional spatial-temporal information. Furthermore, through encoding sequential inputs into coarse-to-fine spike count embeddings, spiking neurons enable SG-JEPA to adapt to the varying computational constraints of downstream tasks. Extensive experiments demonstrate that SG-JEPA achieves competitive or even superior performance over discriminative baselines on node classification, while effectively scaling to the dynamic graph with 13 million edges. SG-JEPA avoids the complex machinery (negative sampling, graph augmentations, edge-level reconstruction, etc.), resulting in superior training efficiency and memory scalability compared with prior self-supervised dynamic graph baselines.

cs.LG↗

arXiv2026

Grad2Fair: A Gradient-driven Approach for Graph Fairness without Demographics

Graph neural networks (GNNs) frequently encounter group fairness issues, often yielding biased predictions against specific demographic groups defined by sensitive attributes such as gender or race. While this challenge has motivated extensive research, most existing solutions rely on the strong assumption that demographics are fully available. To bypass this strict requirement, a few recent studies have attempted to use predicted demographics as proxies to enforce fairness constraints. However, predicted demographics may be inaccurate, resulting in the failure to improve fairness. In this work, we investigate the problem of graph fairness without demographic information and avoid the utilization of predicted demographics. Motivated by our observation that the gradient distributions of misclassified nodes implicitly encode demographic information, we first propose GradDist, a gradient-based metric that quantifies bias by measuring the distance between local modes within these distributions. To mitigate this bias, we propose Gradient-to-Fairness (Grad2Fair), a gradient-guided approach for group fairness without demographics. Due to the potential demographics in gradients, Grad2Fair directly leverages gradients to debias and eliminates demographic prediction, thereby enabling stable fairness performance. Experiments on several real-world datasets demonstrate the effectiveness of Grad2Fair, as evidenced by superior performance over baselines in most cases. Our code is available at https://github.com/ZzoomD/Grad2Fair.

cs.LG↗

arXiv2026

RISKTAGGER: Evidence-Guided LLM Agent for Post-Incident Forensic Analysis of Money Laundering in Web3

Cryptocurrency money-laundering forensic analysis after Web3 incidents faces challenges such as fragmented evidence, expanding transaction paths, and cross-chain discontinuity. Existing Web3 AML methods largely rely on manual clues and heuristic or graph-search-based tracing, with outputs limited to lists of suspicious addresses and lacking path-level evidence and verifiable explanations. Directly applying general-purpose large language models to raw transaction flows also struggles to ensure evidence constraints and result verifiability. To address these limitations, this paper presents RISKTAGGER, an LLM-guided agent for forensic tracing of Web3 cryptocurrency money laundering. RISKTAGGER embeds the LLM as an evidence-constrained decision component within a controlled tracing loop. It extracts case clues from public incident materials, recursively expands a risk-labeled fund-flow graph over on-chain evidence, and generates evidence-organized reports for analyst review. We evaluate it on five real-world incidents spanning multiple years and covering heterogeneous attack patterns and laundering path structures. We further conduct cross-case generalization analysis, baseline comparison, component ablation, and LLM backend analysis. In the main Bybit case, the system achieves a 97.33% address recall and a 98.69% expert-reviewed sampled address precision. Across the other four incidents, it achieves 95.24-100.00% address recall and 91.27-100.00% expert-reviewed address precision. The cross-case results further show that the complexity of Web3 money laundering arises from heterogeneous mechanisms, including short-cycle fund fragmentation, long-range laundering paths, interwoven DeFi services, and deterministic denomination splitting. RISKTAGGER can recover case-related fund paths, identify high-priority risk accounts, and organize public evidence into verifiable forensic reports.

cs.CR↗

arXiv2026

VRExplorer: A Model-based Approach for Semi-Automated Testing of Virtual Reality Scenes

With the proliferation of Virtual Reality (VR) markets, VR applications are rapidly expanding in scale and complexity, thereby driving an urgent need for assuring VR software quality. Different from traditional mobile applications and computer software, VR testing faces unique challenges due to diverse interactions with virtual objects, complex 3D virtual environments, and intricate sequences to complete tasks. All of these emerging challenges hinder existing VR testing tools from effectively and systematically testing VR applications. In this paper, we present VRExplorer, a novel model-based testing tool to effectively interact with diverse virtual objects and explore complex VR scenes. Particularly, we design the Entity, Action, and Task (EAT) framework for modeling diverse VR interactions in a generic way. Built upon the EAT framework, we then present the VRExplorer agent, which can achieve effective scene exploration by incorporating meticulously designed path-finding algorithms into Unity's NavMesh. Moreover, the VRExplorer agent can also systematically execute interaction decisions on top of the Probabilistic Finite State Machine (PFSM). Experimental evaluation on 11 representative VR projects shows that VRExplorer consistently outperforms the state-of-the-art (SOTA) approach VRGuide by achieving significantly higher coverage and better efficiency. Specifically, VRExplorer yields up to 122.8% and 52.8% improvements over VRGuide in terms of executable lines of code (ELOC) coverage and method (function) coverage, respectively. Furthermore, ablation results also verify the essential contributions of each designed module. More importantly, our VRExplorer has successfully detected two functional bugs and one non-functional bug from real-world projects.

cs.SE↗

arXiv2026

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.

cs.SE↗

arXiv2026

LogNLQ: Natural-Language Log Querying with Parser-Induced and Semantically Grounded Schemas

Logs are essential for system monitoring and failure diagnoses in modern software systems, yet querying them through natural language remains an open challenge. Existing approaches either treat logs as plain text, generate queries for schema-light backends, or assume predefined relational schemas, but none addresses a fundamental obstacle: raw logs carry no executable schema over which structured queries can be defined and run. To address these limitations, we present LogNLQ, a framework that formulates natural-language log querying as executable SQL generation over parser-induced and semantically grounded schemas. LogNLQ parses raw logs into template-partitioned relational tables, then applies dual-granularity semantic grounding to annotate both templates and parameter columns with interpretable names and descriptions. At query time, relevant schema candidates are retrieved via semantic search, and a large language model (LLM) generates executable SQL constrained to the retrieved context. To support rigorous evaluation, we introduce LogNLQ-Bench, an execution-verified benchmark of 8,895 queries over four real-world log datasets. Experimental results demonstrate that LogNLQ consistently outperforms all representative baselines by wide margins, with especially pronounced gains on analytically complex scenario queries.

cs.SE↗

arXiv2026

Dynamic analysis enhances issue resolution

Resolving complex code defects from natural language descriptions remains a fundamental software engineering challenge. Recently, large language models (LLMs) have driven the creation of agent-based automated repair systems. While improving repository-level problem-solving, current methods struggle with complex defects like intricate polymorphic control flows and implicit type degradation. These approaches rely on static analysis and shallow execution feedback, lacking the ability to monitor intermediate execution states. Consequently, agents often fall into speculative exploration, consuming significant tokens without identifying the root cause. We introduce DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent), a pioneering automated repair framework deeply embedding dynamic analysis into the agent's decision loop. DAIRA employs a Test Tracing-Driven workflow, using lightweight tools to capture runtime evidence (e.g., call stacks and variable states) and convert it into structured semantic reports. By illuminating execution paths and causal dependencies, DAIRA enables precise fault localization and prevents context window flooding from irrelevant code retrievals. This shifts the agent's approach from speculative reasoning to deterministic inference. Evaluations on the SWE-bench Verified benchmark show DAIRA achieves a state-of-the-art 79.4% resolution rate when powered by Gemini 3 Flash Preview. Furthermore, it demonstrates robustness in addressing deep-seated logical defects, securing a 44.4% resolution rate on the most demanding tasks. Compared to baselines, DAIRA uniquely resolves complex edge cases and improves operational efficiency. Across various LLMs, it reduces inference costs by approximately 10% and input token consumption by 25%.

cs.SE↗

arXiv2026

AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion

Repository-level code completion remains a challenging task for existing code large language models (code LLMs) due to their limited understanding of repository-specific context and domain knowledge. While retrieval-augmented generation (RAG) approaches have shown promise by retrieving relevant code snippets as cross-file context, they suffer from two fundamental problems: misalignment between the query and the target code in the retrieval process, and the inability of existing retrieval methods to effectively utilize the inference information. To address these challenges, we propose AlignCoder, a repository-level code completion framework that introduces a query enhancement mechanism and a reinforcement learning based retriever training method. Our approach generates multiple candidate completions to construct an enhanced query that bridges the semantic gap between the initial query and the target code. Additionally, we employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval. We evaluate AlignCoder on two widely-used benchmarks (CrossCodeEval and RepoEval) across five backbone code LLMs, demonstrating an 18.1% improvement in EM score compared to baselines on the CrossCodeEval benchmark. The results show that our framework achieves superior performance and exhibits high generalizability across various code LLMs and programming languages.

cs.SE↗

arXiv2026

Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation

Automated unit test generation using large language models (LLMs) holds great promise but often struggles with generating tests that are both correct and maintainable in real-world projects. This paper presents KTester, a novel framework that integrates project-specific knowledge and testing domain knowledge to enhance LLM-based test generation. Our approach first extracts project structure and usage knowledge through static analysis, which provides rich context for the model. It then employs a testing-domain-knowledge-guided separation of test case design and test method generation, combined with a multi-perspective prompting strategy that guides the LLM to consider diverse testing heuristics. The generated tests follow structured templates, improving clarity and maintainability. We evaluate KTester on multiple open-source projects, comparing it against state-of-the-art LLM-based baselines using automatic correctness and coverage metrics, as well as a human study assessing readability and maintainability. Results demonstrate that KTester significantly outperforms existing methods across six key metrics, improving execution pass rate by 5.69% and line coverage by 8.83% over the strongest baseline, while requiring less time and generating fewer test cases. Human evaluators also rate the tests produced by KTester significantly higher in terms of correctness, readability, and maintainability, confirming the practical advantages of our knowledge-driven framework.

cs.SE↗

arXiv2026

From Hype to Collapse: Investigating Rug Pull Scams on Solana

Solana has experienced rapid growth due to its high performance and low transaction costs, but the extremely low barrier to token issuance has also enabled widespread Rug Pulls. Unlike Ethereum-based Rug Pulls, which often rely on malicious smart-contract logic, Solana's unified SPL Token program shifts fraudulent execution toward on-chain behavioral manipulation. However, existing research has not systematically examined these Solana-specific Rug Pull patterns, and no public Solana Rug Pull dataset is available for empirical research. To bridge this gap, we present a large-scale measurement study of Rug Pulls on Solana. We manually verify 68 community-reported incidents and curate a benchmark of 117 confirmed Rug Pull tokens, from which we distill three representative on-chain behavioral patterns: Freeze Authority Abuse, Liquidity Withdrawal, and Pump-and-Dump. Guided by these patterns, we design a behavior-guided candidate identification and human-validation pipeline. We apply this pipeline to 100,063 tokens newly issued on Orca, Raydium, and Meteora during the first half of 2025, identifying 76,469 Rug Pull tokens. A random manual audit of 382 samples estimates a labeling false-positive rate of 0.26\%, supporting the reliability of the dataset. We release the resulting dataset and use it to characterize the Solana Rug Pull ecosystem. Our analysis shows that Rug Pulls on Solana exhibit extremely short lifecycles, strong price-driven dynamics, severe economic losses, and highly organized group behaviors. These findings provide new insights into the Solana Rug Pull landscape and support the development of effective on-chain defense mechanisms.

cs.CR↗

arXiv2026

DataClawBench: An Agent Benchmark for Exploratory Real-World Financial Data Analysis

Autonomous data analysis agents are increasingly expected to conduct exploratory analysis with limited human guidance about data. However, existing benchmarks typically evaluate such agents in prior-guided settings, providing selected data sources, explicit data schemas, or cleaned data, thereby understating the exploratory burden. To evaluate this realistic exploratory data analysis task, we introduce DataClawBench, a benchmark built from financial think-tank consulting scenarios where agents must independently explore unfamiliar, noisy, cross-domain data and produce verifiable conclusions. DataClawBench provides a unified real-world data environment with approximately 2.06 million records across enterprise, industry, and policy domains, with native data noise preserved. On top of this data environment, it defines 492 multi-step cross-domain tasks, each annotated with intermediate milestones that diagnose exploration and reasoning failures beyond outcome accuracy. A systematic evaluation of eight advanced LLMs under the OpenClaw agent reveals that exploratory data analysis breaks agent reliability: more exploration does not reliably translate into task-relevant progress or correct final answers.

cs.AI↗

arXiv2026

CelerLog: Fast Log Parsing via Dynamic Routing

Log parsing is a fundamental step for automated log analysis, which transforms raw log messages into structured formats. Existing syntax-based parsers struggle with complex logs because they lack semantic reasoning ability. Emerging LLM-powered semantic parsers achieve high accuracy but suffer from prohibitive latency and token costs because they apply semantic inference across all logs. Our key observation is that not all logs necessitate complex semantic understanding: a vast majority of logs exhibit repetitive patterns that can be extracted via straightforward statistical analysis. Driven by this insight, we propose CelerLog, a fast and effective log parser. CelerLog introduces a dynamic routing mechanism to classify logs into dense and sparse groups. Logs with strong statistical patterns (dense groups) are processed by an efficient statistical processor, whereas the sparse groups lacking such patterns are routed to an LLM for semantic inference. This hybrid strategy avoids unnecessary LLM invocations. Extensive experiments on 14 public datasets show that CelerLog achieves leading performance over state-of-the-art baselines and is 7.9x to 18.6x faster than LLM methods and up to 1.5x faster than Drain. Additionally, it reduces costs by decreasing token consumption by 80.2% - 94.1% and LLM invocations by 86.4% - 90.9%.

cs.SE↗

arXiv2026

VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesis

Vision-Language-Action (VLA) models follow a data-driven paradigm and are constrained by the coverage of training data, making them prone to failure on edge-case configurations after deployment. To mitigate such risks, it is essential to expose high-quality failure modes and convert the resulting failures into supervisory data for model enhancement. Existing studies largely stop at failure detection and lack a mechanism for leveraging discovered failures for model repair. We propose VLAMotor, the first analysis framework for VLA enhancement, which integrates distance-aware model testing for failure exposure and agent-based data synthesis for model finetunning. First, VLAMotor estimates input uncertainty based on the distance to training samples, and combines uncertainty ranking with redundancy elimination to build compact test sets that expose diverse failures. Then, VLAMotor abstracts failure trajectories into structured semantic representations, and plans parameterized repair-skill sequences, which are then realized as executable trajectories through inverse kinematics and motion execution. The resulting successful trajectories are automatically labeled and used to fine-tune the original VLA model, yielding an enhanced VLA model. Evaluation on four representative robotic manipulation tasks shows that 92.33% of the in-simulation test cases generated by VLAMotor trigger VLA failures, and VLAMotor improves test coverage over the state-of-the-art tool by 18.93%. By fine-tuning VLA models with synthetic data derived from failed test cases, VLAMotor further enhances the overall success rate of VLA models by 49.25%. When deployed on real hardware, the simulation-enhanced models improve the success rate over the original VLA models by 57.50%, demonstrating an effective and low-cost direction for VLA enhancement.

cs.RO↗