arXiv subjects

Le Yan

Publications and source records attributed to Le Yan.

At least 19 recordsLinked to original sources

arXiv2025

Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation

While Pairwise Ranking Prompting (PRP) with Large Language Models (LLMs) is one of the most effective zero-shot document ranking methods, it has a quadratic computational complexity with respect to the number of documents to be ranked, as it requires an enumeration over all possible document pairs. Consequently, the outstanding ranking performance of PRP has remained unreachable for most real-world ranking applications. In this work, we propose to harness the effectiveness of PRP through pairwise distillation. Specifically, we distill a pointwise student ranker from pairwise teacher labels generated by PRP, resulting in an efficient student model that retains the performance of PRP with substantially lower computational costs. Furthermore, we find that the distillation process can be made sample-efficient: with only 2% of pairs, we are able to obtain the same performance as using all pairs for teacher labels. Thus, our novel approach provides a solution to harness the ranking performance of PRP without incurring high computational costs during both distillation and serving.

cs.IR↗

arXiv2025

Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks

Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language processing. Specifically, encoder-only models remain dominant in tasks like classification, regression, and ranking. This is primarily due to the inherent structure of decoder-based models, which limits their direct applicability to these tasks. In this paper, we introduce Gemma Encoder, adapting the powerful Gemma decoder model to an encoder architecture, thereby unlocking its potential for a wider range of non-generative applications. To optimize the adaptation from decoder to encoder, we systematically analyze various pooling strategies, attention mechanisms, and hyperparameters (e.g., dropout rate). Furthermore, we benchmark Gemma Encoder against established approaches on the GLUE benchmarks, and MS MARCO ranking benchmark, demonstrating its effectiveness and versatility.

cs.CL↗

arXiv2024

MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers

The most recent pointwise Large Language Model (LLM) rankers have achieved remarkable ranking results. However, these rankers are hindered by two major drawbacks: (1) they fail to follow a standardized comparison guidance during the ranking process, and (2) they struggle with comprehensive considerations when dealing with complicated passages. To address these shortcomings, we propose to build a ranker that generates ranking scores based on a set of criteria from various perspectives. These criteria are intended to direct each perspective in providing a distinct yet synergistic evaluation. Our research, which examines eight datasets from the BEIR benchmark demonstrates that incorporating this multi-perspective criteria ensemble approach markedly enhanced the performance of pointwise LLM rankers.

cs.IR↗

arXiv2024

Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing

The powerful generative abilities of large language models (LLMs) show potential in generating relevance labels for search applications. Previous work has found that directly asking about relevancy, such as ``How relevant is document A to query Q?", results in sub-optimal ranking. Instead, the pairwise ranking prompting (PRP) approach produces promising ranking performance through asking about pairwise comparisons, e.g., ``Is document A more relevant than document B to query Q?". Thus, while LLMs are effective at their ranking ability, this is not reflected in their relevance label generation. In this work, we propose a post-processing method to consolidate the relevance labels generated by an LLM with its powerful ranking abilities. Our method takes both LLM generated relevance labels and pairwise preferences. The labels are then altered to satisfy the pairwise preferences of the LLM, while staying as close to the original values as possible. Our experimental results indicate that our approach effectively balances label accuracy and ranking performance. Thereby, our work shows it is possible to combine both the ranking and labeling abilities of LLMs through post-processing.

cs.IR↗

arXiv2023

Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels

Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.

cs.IR↗

arXiv2023

Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

cs.IR↗

arXiv2023

Learning to Rank when Grades Matter

Graded labels are ubiquitous in real-world learning-to-rank applications, especially in human rated relevance data. Traditional learning-to-rank techniques aim to optimize the ranked order of documents. They typically, however, ignore predicting actual grades. This prevents them from being adopted in applications where grades matter, such as filtering out ``poor'' documents. Achieving both good ranking performance and good grade prediction performance is still an under-explored problem. Existing research either focuses only on ranking performance by not calibrating model outputs, or treats grades as numerical values, assuming labels are on a linear scale and failing to leverage the ordinal grade information. In this paper, we conduct a rigorous study of learning to rank with grades, where both ranking performance and grade prediction performance are important. We provide a formal discussion on how to perform ranking with non-scalar predictions for grades, and propose a multiobjective formulation to jointly optimize both ranking and grade predictions. In experiments, we verify on several public datasets that our methods are able to push the Pareto frontier of the tradeoff between ranking and grade prediction performance, showing the benefit of leveraging ordinal grade information.

cs.IR↗

arXiv2023

RD-Suite: A Benchmark for Ranking Distillation

The distillation of ranking models has become an important topic in both academia and industry. In recent years, several advanced methods have been proposed to tackle this problem, often leveraging ranking information from teacher rankers that is absent in traditional classification settings. To date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide range of tasks and datasets make it difficult to assess or invigorate advances in this field. This paper first examines representative prior arts on ranking distillation, and raises three questions to be answered around methodology and reproducibility. To that end, we propose a systematic and unified benchmark, Ranking Distillation Suite (RD-Suite), which is a suite of tasks with 4 large real-world datasets, encompassing two major modalities (textual and numeric) and two applications (standard distillation and distillation transfer). RD-Suite consists of benchmark results that challenge some of the common wisdom in the field, and the release of datasets with teacher scores and evaluation scripts for future research. RD-Suite paves the way towards better understanding of ranking distillation, facilities more research in this direction, and presents new challenges.

cs.IR↗

arXiv2022

Towards Disentangling Relevance and Bias in Unbiased Learning to Rank

Unbiased learning to rank (ULTR) studies the problem of mitigating various biases from implicit user feedback data such as clicks, and has been receiving considerable attention recently. A popular ULTR approach for real-world applications uses a two-tower architecture, where click modeling is factorized into a relevance tower with regular input features, and a bias tower with bias-relevant inputs such as the position of a document. A successful factorization will allow the relevance tower to be exempt from biases. In this work, we identify a critical issue that existing ULTR methods ignored - the bias tower can be confounded with the relevance tower via the underlying true relevance. In particular, the positions were determined by the logging policy, i.e., the previous production model, which would possess relevance information. We give both theoretical analysis and empirical results to show the negative effects on relevance tower due to such a correlation. We then propose three methods to mitigate the negative confounding effects by better disentangling relevance and bias. Empirical results on both controlled public datasets and a large-scale industry dataset show the effectiveness of the proposed approaches.

cs.IR↗

arXiv2022

Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance

As Learning-to-Rank (LTR) approaches primarily seek to improve ranking quality, their output scores are not scale-calibrated by design. This fundamentally limits LTR usage in score-sensitive applications. Though a simple multi-objective approach that combines a regression and a ranking objective can effectively learn scale-calibrated scores, we argue that the two objectives are not necessarily compatible, which makes the trade-off less ideal for either of them. In this paper, we propose a practical regression compatible ranking (RCR) approach that achieves a better trade-off, where the two ranking and regression components are proved to be mutually aligned. Although the same idea applies to ranking with both binary and graded relevance, we mainly focus on binary labels in this paper. We evaluate the proposed approach on several public LTR benchmarks and show that it consistently achieves either best or competitive result in terms of both regression and ranking metrics, and significantly improves the Pareto frontiers in the context of multi-objective optimization. Furthermore, we evaluated the proposed approach on YouTube Search and found that it not only improved the ranking quality of the production pCTR model, but also brought gains to the click prediction accuracy. The proposed approach has been successfully deployed in the YouTube production system.

cs.IR↗

arXiv2021

Rank4Class: A Ranking Formulation for Multiclass Classification

Multiclass classification (MCC) is a fundamental machine learning problem of classifying each instance into one of a predefined set of classes. In the deep learning era, extensive efforts have been spent on developing more powerful neural embedding models to better represent the instance for improving MCC performance. In this paper, we do not aim to propose new neural models for instance representation learning, but to show that it is promising to boost MCC performance with a novel formulation through the lens of ranking. In particular, by viewing MCC as to rank classes for an instance, we first argue that ranking metrics, such as Normalized Discounted Cumulative Gain, can be more informative than the commonly used Top-$K$ metrics. We further demonstrate that the dominant neural MCC recipe can be transformed to a neural ranking framework. Based on such generalization, we show that it is intuitive to leverage advanced techniques from the learning to rank literature to improve the MCC performance out of the box. Extensive empirical results on both text and image classification tasks with diverse datasets and backbone neural models show the value of our proposed framework.

cs.LG↗

arXiv2021

Improving Neural Ranking via Lossless Knowledge Distillation

We explore a novel perspective of knowledge distillation (KD) for learning to rank (LTR), and introduce Self-Distilled neural Rankers (SDR), where student rankers are parameterized identically to their teachers. Unlike the existing ranking distillation work which pursues a good trade-off between performance and efficiency, SDR is able to significantly improve ranking performance of students over the teacher rankers without increasing model capacity. The key success factors of SDR, which differs from common distillation techniques for classification are: (1) an appropriate teacher score transformation function, and (2) a novel listwise distillation framework. Both techniques are specifically designed for ranking problems and are rarely studied in the existing knowledge distillation literature. Building upon the state-of-the-art neural ranking structure, SDR is able to push the limits of neural ranking performance above a recent rigorous benchmark study and significantly outperforms traditionally strong gradient boosted decision tree based models on 7 out of 9 key metrics, the first time in the literature. In addition to the strong empirical results, we give theoretical explanations on why listwise distillation is effective for neural rankers, and provide ablation studies to verify the necessity of the key factors in the SDR framework.

cs.IR↗

arXiv2020

Stream-Flow Forecasting of Small Rivers Based on LSTM

Stream-flow forecasting for small rivers has always been of great importance, yet comparatively challenging due to the special features of rivers with smaller volume. Artificial Intelligence (AI) methods have been employed in this area for long, but improvement of forecast quality is still on the way. In this paper, we tried to provide a new method to do the forecast using the Long-Short Term Memory (LSTM) deep learning model, which aims in the field of time-series data. Utilizing LSTM, we collected the stream flow data from one hydrologic station in Tunxi, China, and precipitation data from 11 rainfall stations around to forecast the stream flow data from that hydrologic station 6 hours in the future. We evaluated the prediction results using three criteria: root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R^2). By comparing LSTM's prediction with predictions of Support Vector Regression (SVR) and Multilayer Perceptions (MLP) models, we showed that LSTM has better performance, achieving RMSE of 82.007, MAE of 27.752, and R^2 of 0.970. We also did extended experiments on LSTM model, discussing influence factors of its performance.

cs.LG↗

arXiv2019

Competition between entropy and energy in network glass: the hidden connection between intermediate phase and liquid-liquid transition

In network glass including chalcogenides, the network topology of microscopic structures can be tuned by changing the chemical compositions. As the composition is varied, an intermediate phase (IP) singularly different from the adjacent floppy or rigid phases on sides has been revealed in the vicinity of the rigidity onset of the network. Glass formers in the IP appear to be reversible at glass transition and strong in dynamical fragility. Meanwhile, the calorimetry experiments indicate the existence of a first-order liquid-liquid transition (LLT) at a temperature above the glass transition in some strong glass-forming liquids. How are the intermediate phase and the liquid-liquid transition related? Recent molecular dynamics simulations hint that the intermediate phase is thermodynamically distinct that the transitions to IP as varying the chemical composition in fact reflect the LLT: out of IP, the glass is frozen in vibrational entropy-dominated heterogeneous structures with voids; while inside IP, energy dominates and the microscopic structures of liquids become homogeneous. Here we demonstrate such first-order thermodynamic liquid-liquid transition numerically and analytically in an elastic network model of network glass and discuss possible experimental approaches to testify the connection.

cond-mat.soft↗

arXiv2019

Mechanics of allostery: contrasting the induced fit and population shift scenarios

In allosteric proteins, binding a ligand can affect function at a distant location, for example by changing the binding affinity of a substrate at the active site. The induced fit and population shift models, which differ by the assumed number of stable configurations, explain such cooperative binding from a thermodynamic viewpoint. Yet, understanding what mechanical principles constrain these models remains a challenge. Here we provide an empirical study on 34 proteins supporting the idea that allosteric conformational change generally occurs along a soft elastic mode presenting extended regions of high shear. We argue, based on a detailed analysis of how the energy profile along such a mode depends on binding, that in the induced fit scenario there is an optimal stiffness $k_a^*\sim 1/N$ for cooperative binding, where $N$ is the number of residues involved in the allosteric response. We find that the population shift scenario is more robust to mutation affecting stiffness, as binding becomes more and more cooperative with stiffness up to the same characteristic value $k_a^*$, beyond which cooperativity saturates instead of decaying. We confirm numerically these findings in a non-linear mechanical model. Dynamical considerations suggest that a stiffness of order $k_a^*$ is favorable in that scenario as well, supporting that for proper function proteins must evolve a functional elastic mode that is softer as their size increases. In consistency with this view, we find a significant anticorrelation between the stiffness of the allosteric response and protein size in our data set.

physics.bio-ph↗

arXiv2018

Phylodynamics of rapidly adapting pathogens: extinction and speciation of a Red Queen

Rapidly evolving pathogens like influenza viruses can persist by accumulating antigenic novelty fast enough to evade the adaptive immunity of the host population, yet without continuous accumulation of genetic diversity. This dynamical state is often compared to the Red Queen evolving as fast as it can just to maintain its foothold in the host population: Accumulation of antigenic novelty is balanced by the build-up of host immunity. Such Red Queen States (RQS) of continuous adaptation in large rapidly mutating populations are well understood in terms of Traveling Wave (TW) theories of population genetics. Here we shall make explicit the mapping of the established Multi-strain Susceptible-Infected-Recovered (SIR) model onto the TW theory and demonstrate that a pathogen can persist in RQS if cross-immunity is long-ranged and its population size is large populations allowing for rapid adaptation. We then investigate the stability of this state focusing on the rate of extinction and the rate of "speciation" defined as antigenic divergence of viral strains beyond the range of cross-inhibition. RQS states are transient, but in a certain range of evolutionary parameters can exist for the time long compared to the typical time to the most recent common ancestor ($T_{MRCA}$). In this range the steady TW is unstable and the antigenic advance of the lead strains relative to the typical co-circulating viruses tends to oscillate. This results in large fluctuations in prevalence that facilitate extinction. We shall demonstrate that the rate of TW fission into antigenically uncoupled viral populations is related to fluctuations of $T_{MRCA}$ and construct a "phase diagram" identifying different regimes of viral phylodynamics as a function of evolutionary parameters.

q-bio.PE↗

arXiv2018

Multicellular rosettes drive fluid-solid transition in epithelial tissues

Models for confluent biological tissues often describe the network formed by cells as a triple-junction network, similar to foams. However, higher order vertices or multicellular rosettes are prevalent in developmental and {\it in vitro} processes and have been recognized as crucial in many important aspects of morphogenesis, disease, and physiology. In this work, we study the influence of rosettes on the mechanics of a confluent tissue. We find that the existence of rosettes in a tissue can greatly influence its rigidity. Using a generalized vertex model and effective medium theory we find a fluid-to-solid transition driven by rosette density and intracellular tensions. This transition exhibits several hallmarks of a second-order phase transition such as a growing correlation length and a universal critical scaling in the vicinity a critical point. Further, we elucidate the nature of rigidity transitions in dense biological tissues and other cellular structures using a generalized Maxwell constraint counting approach. This answers a long-standing puzzle of the origin of solidity in these systems.

physics.bio-ph↗

arXiv2017

Entropy favors heterogeneous structures of networks near the rigidity threshold

The dynamical properties and mechanical functions of amorphous materials are governed by their microscopic structures, particularly the elasticity of the interaction networks, which is generally complicated by structural heterogeneity. This ubiquitous heterogeneous nature of amorphous materials is intriguingly attributed to a complex role of entropy. Here, we show in disordered networks that the vibrational entropy increases by creating phase-separated structures when the interaction connectivity is close to the onset of network rigidity. The stress-energy, which conversely penalizes the heterogeneity, finally dominates a smaller vicinity of the rigidity threshold at the glass transition and creates a homogeneous intermediate phase. This picture of structures changing between homogeneous and heterogeneous phases by varying connectivity provides an interpretation of the transitions observed in chalcogenide glasses.

cond-mat.stat-mech↗