arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.40112

Beyond IPS: Reliable Counterfactual Evaluation in Multi-Stage Ad Systems without Logged Propensities

Mohsen Malmir·Mohamed A. Radwan·Houssam Nassif·Murat Ali Bayir
View original source ↗Read source document ↗Export metadata ↓

Abstract

Offline evaluation of ad-ranking policies in multi-stage delivery systems is challenging because action propensities are often unavailable, limiting inverse propensity scoring (IPS). Reward-model-based estimators such as the Direct Method (DM) avoid propensities but can be biased when candidate policies induce impression distributions that differ from the logged data. We formalize DM-based counterfactual replay and propose mixture-weighted reward modeling, which reweights logged data toward a mixture of candidate-policy distributions using density ratios from a multiclass domain classifier. We bound each candidate's value-estimation error by overlap, cold-start, and replay terms, and derive an asymptotic rule that uses the cold-start mass to decide whether to trust offline selection or route candidates to online A/B testing. On an ads-delivery simulator and on production traffic from a large-scale CTR ranking system, the mixture-weighted reward model improves policy selection over its unweighted counterpart in both settings, with the largest gains at low overlap in simulation. Combined with a doubly robust correction, it attains the highest Top-1 accuracy in production and at low and medium overlap in simulation.

Explore related subjects

cs.IR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Mohsen Malmir, Mohamed A. Radwan, Houssam Nassif, Murat Ali Bayir. 2026-10-29. Beyond IPS: Reliable Counterfactual Evaluation in Multi-Stage Ad Systems without Logged Propensities. https://arxiv.org/abs/2610.40112

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DAT: Dynamic Alpha Tuning for Hybrid Retrieval in Retrieval-Augmented Generation

Hybrid retrieval techniques in Retrieval-Augmented Generation (RAG) systems enhance information retrieval by combining dense and sparse (e.g., BM25-based) retrieval methods. However, existing approaches struggle with adaptability, as fixed weighting schemes fail to adjust to different queries. To address this, we propose DAT (Dynamic Alpha Tuning), a novel hybrid retrieval framework that dynamically balances dense retrieval and BM25 for each query. DAT leverages a large language model (LLM) to evaluate the effectiveness of the top-1 results from both retrieval methods, assigning an effectiveness score to each. It then calibrates the optimal weighting factor through effectiveness score normalization, ensuring a more adaptive and query-aware weighting between the two approaches. Empirical results show that DAT consistently significantly outperforms fixed-weighting hybrid retrieval methods across various evaluation metrics. Even on smaller models, DAT delivers strong performance, highlighting its efficiency and adaptability.

cs.IR↗
arXiv2026

When Reviews Are Scarce: Structure-Aware Review Imputation for Recommendation in Realistic Sparse Review Settings

Textual reviews enrich recommender systems with fine-grained preference signals and enhanced explainability. However, in real-world scenarios, users rarely leave reviews, resulting in severe sparsity that undermines the effectiveness of review-aware recommenders under partial observability. A natural remedy is to synthesize the missing reviews to restore the textual signal these recommenders rely on. Yet, conventional imputation techniques, such as matrix completion and LLM-based augmentation, either lose contextualized semantics by embedding texts into vectors or overlook structural dependencies on the user-item graph. To address these shortcomings, we propose TWISTER (ToWards Imputation on Sparsity with Textual Edge Graph Representation), a unified framework for sparse-text recommendation that synthesizes missing reviews by jointly modeling semantic and structural signals. Specifically, we represent user-item interactions as a Textual-Edge Graph (TEG), treating reviews as edge attributes. To capture relational context, we construct line-graph views and employ an LLM as a graph-aware aggregator. For each interaction lacking a textual review, our model aggregates natural-language representations from neighboring interactions to generate a coherent and personalized review. Experiments on multiple datasets show that TWISTER consistently outperforms traditional numeric, graph-based, and LLM baselines, delivering higher-quality imputed reviews and, more importantly, improved downstream recommendation. In summary, TWISTER is a drop-in, structure-aware imputer that recovers the textual signal that review-aware recommenders need when reviews are scarce.

cs.IR↗
arXiv2026

Explaining Group Recommendations via Counterfactuals

Group recommender systems help users make collective choices but often lack transparency, leaving group members uncertain about why items are suggested. Existing explanation methods focus on individuals, offering limited support for groups where multiple preferences interact. In this work, we introduce and formalize the problem of finding counterfactual explanations for group recommendations. We introduce a comprehensive set of item- and explanation-level metrics that capture different aspects, such as relevance, popularity, explanation utility, interpretability, and fairness. We design five algorithms that employ techniques such as search-space pruning, greedy selection, early stopping, and incremental refinement to reduce computational cost and the number of recommender calls while producing high-quality explanations. The algorithms consider different trade-offs; for example, some prioritize computational efficiency at the expense of explanation quality, while others focus on improving explanation quality at a higher computational cost. Experiments on real-world datasets analyze these trade-offs in terms of explanation size, computational cost, utility, and fairness.

cs.IR↗