arXiv subjects

Julien Yu

Publications and source records attributed to Julien Yu.

3 recordsLinked to original sources

arXiv2026

MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness

Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive "act-as-a-user" prompting often yields verbose, unrealistic utterances, motivating principled evaluation of *user proxy agents*. We present **MirrorBench**, a reproducible and extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational regimes, explicitly decoupled from downstream task success. **MirrorBench** combines three lexical-diversity metrics (**MATTR**, **Yule's~$K$**, and **HD-D**) with three LLM-judge-based metrics (**GTEval**, **Pairwise Indistinguishability**, and **Rubric-and-Reason**), and contextualizes judge scores using Human-Human and Proxy-Proxy calibration controls. Across four public datasets, **MirrorBench** yields variance-aware comparisons and reveals systematic gaps between user proxies and real human users. The framework is open sourced at https://github.com/SAP/mirrorbench and includes a command-line interface for running and managing user-proxy benchmarking experiments.

cs.AI↗

arXiv2025

Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky

Large language models (LLMs) are increasingly tasked with invoking enterprise APIs, yet they routinely falter when near-duplicate tools vie for the same user intent or when required arguments are left underspecified. We introduce DiaFORGE (Dialogue Framework for Organic Response Generation & Evaluation), a disambiguation-centric, three-stage pipeline that (i) synthesizes persona-driven, multi-turn dialogues in which the assistant must distinguish among highly similar tools, (ii) performs supervised fine-tuning of open-source models with reasoning traces across 3B - 70B parameters, and (iii) evaluates real-world readiness via a dynamic suite that redeploys each model in a live agentic loop and reports end-to-end goal completion alongside conventional static metrics. On our dynamic benchmark DiaBENCH, models trained with DiaFORGE raise tool-invocation success by 27 pp over GPT-4o and by 49 pp over Claude-3.5-Sonnet, both under optimized prompting. To spur further research, we release an open corpus of 5000 production-grade enterprise API specifications paired with rigorously validated, disambiguation-focused dialogues, offering a practical blueprint for building reliable, enterprise-ready tool-calling agents.

cs.AI↗

arXiv2025

A lossless a priori splitting rule for split-delivery routing problems

Resource allocation problems in which demand is splittable are usually solved using different solution methods from their unsplittable equivalents. Although splittable problem instances can be the easier of the two (for example, they might simply correspond to a linear relaxation of a discrete problem), there exist many problems, including routing problems, for which the converse is true. That is, the technology for solving unsplittable problems is mature, but the splittable counterpart is not. For such problems, one strategy that has recently shown potential is the use of an a priori splitting rule in which each customer's demand is split into smaller pieces in advance, which enables one to simply solve the splittable problem as an instance of the unsplittable version. An important factor to consider is the number of pieces that result after this splitting. A large numbers of pieces will allow more splitting patterns to be realizable, but will result in a larger problem instance. In this paper, we introduce a splitting rule that minimizes the number of pieces, subject to the constraint that all demand splitting patterns remain feasible. Computational experiments on benchmark instances for the vehicle routing problem and a time-windows extension show that the solution quality of our proposed splitting rule can match the performance of existing approaches.

cs.DS↗