arXiv subjects

Ruizhe Shi

Publications and source records attributed to Ruizhe Shi.

10 recordsLinked to original sources

arXiv2026

Streaming Hypergraph Coloring via Palette Sparsification

For every fixed $k\ge2$, we give a randomized one-pass insertion-only algorithm that colors an $n$-vertex $k$-uniform hypergraph of maximum degree $Δ$ with $O(Δ^{1/(k-1)})$ colors using $\widetilde O_k(n)$ bits of working memory. As a graph-theoretic result of independent interest, we also prove a tight palette-sparsification theorem for general uniform hypergraphs. Independently sampled lists of $Θ(\sqrt{\log n})$ colors from a palette of size $O(Δ^{1/(k-1)})$ preserve colorability with high probability; the list-size dependence is asymptotically optimal. These results extend to bounded-rank hypergraphs. We complement the algorithm with a deterministic lower bound: for every fixed polylogarithmic semi-streaming space bound, there are polylogarithmic values of $Δ$ for which any deterministic one-pass algorithm requires $\exp(Δ^{Ω(1)})$ colors.

math.CO↗

arXiv2026

An Improved Upper Bound for Colorings Without Symmetrically Colored $k$-Term Arithmetic Progressions

Given a coloring $c$ and an even $k\ge 4$, a nontrivial $k$-term arithmetic progression~($k$-AP) $a,a+d,\ldots,a+(k-1)d$ is called symmetrically colored if $c(a+(i-1)d)=c(a+(k-i)d)$, $\forall i\in[k/2]$. Deng, Tidor, and Zhao asked whether $[N]$ admits a coloring with $N^{o(1)}$ colors and no such 4-APs, and gave an $O(N^{\log_{22}3})$-coloring of $[N]$. We give an $O_k(p)$-coloring of $\mathbb Z/p^{k^2/4}\mathbb Z$ without such $k$-APs for every even $k\ge 4$ and every prime $p>k$, and hence an $O_k(N^{4/k^2})$-coloring of $[N]$, improving the exponent in the upper bound for $4$-APs from $\log_{22}3$ to $1/4$. The construction combines a carry-control coloring of base-$p$ digits with a layered field norm mapping. Together with Behrend-style product colorings, our result for $4$-APs gives $h(N)\leq N^{1/4+o(1)}$ in Erdős's Problem~160 on coloring every nontrivial 4-AP with at least three colors. This result also yields $ρ_4(α)=O_\varepsilon(α^{5-\varepsilon})$ for every $\varepsilon>0$, improving the bound toward Ruzsa's question. Our result for $k$-APs disproves Gowers' conjectured lower bound for all even $k\ge6$ for the first time.

math.CO↗

arXiv2026

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

We present a fine-grained theoretical analysis of the performance gap between two-stage reinforcement learning from human feedback~(RLHF) and direct preference optimization~(DPO). Our study decomposes this gap into two sources: the explicit representation gap under exact optimization and the implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities. We show that RLHF, DPO, or online DPO can outperform one another depending on type of model mis-specifications. Notably, online DPO can outperform both RLHF and standard DPO when the reward and policy model classes are isomorphic and both mis-specified. In the approximate optimization setting, we provide a concrete construction where the ground-truth reward is sparse and show that RLHF requires significantly fewer samples than DPO to recover an effective reward model, highlighting a statistical advantage of two-stage learning. Together, these results provide a comprehensive understanding of the performance gap between RLHF and DPO under various settings, and offer practical insights into when each method is preferred.

cs.LG↗

arXiv2026

NFT Games: an Empirical Look into the Play-to-Earn Model

The past decade has witnessed the burgeoning and continuous development of blockchain and its applications. Besides various cryptocurrencies, an industry that has quickly embraced this trend is gaming. Thanks to the support of blockchain, games have started to incorporate non-fungible tokens (NFTs) that can enable a new gaming model, play-to-earn (P2E), which incentivizes users to participate and play. While recent studies looked at several NFT games qualitatively and individually, an in-depth understanding is still missing, particularly on how the P2E model has transformed traditional games. In this work, we set to conduct a measurement study of NFT games, aiming to gain a comprehensive understanding of the effectiveness of P2E in practice. For this purpose, we collect and analyze relevant NFT transaction data from the underlying blockchain (e.g., Ethereum) of 12 games, supplemented with various data scraped from their websites. Our study shows that (1) a few top wallets control unproportionally high percentage of NFTs, and the majority of wallets own only one or two NFTs and do not actively trade; (2) promotion events do boost the trade amount and the NFT price for some games, but their effect does not sustain; and (3) few players actually earned a profit, and players in 9 out of 12 games who traded NFTs have a negative profit on average. Motivated by these findings, we further investigate effective incentive mechanisms based on game theory to improve the trading profits that players can earn from these NFT games. Both modeling and simulation results confirm the effectiveness of the proposed incentive mechanism.

cs.CE↗

arXiv2025

The Crucial Role of Samplers in Online Direct Preference Optimization

Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $\textbf{linear}$ convergence, while our proposed online sampler achieves $\textbf{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over $7.4$% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.

cs.LG↗

arXiv2024

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to find a near-optimal policy using pre-collected datasets. In real-world scenarios, data collection could be costly and risky; therefore, offline RL becomes particularly challenging when the in-domain data is limited. Given recent advances in Large Language Models (LLMs) and their few-shot learning prowess, this paper introduces $\textbf{La}$nguage Models for $\textbf{Mo}$tion Control ($\textbf{LaMo}$), a general framework based on Decision Transformers to effectively use pre-trained Language Models (LMs) for offline RL. Our framework highlights four crucial components: (1) Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively, (3) using the non-linear MLP transformation instead of linear projections, to generate embeddings, and (4) integrating an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities on languages. Empirical results indicate $\textbf{LaMo}$ achieves excellent performance in sparse-reward tasks and closes the gap between value-based offline RL methods and decision transformers in dense-reward tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples.

cs.LG↗

arXiv2024

Decoding-Time Language Model Alignment with Multiple Objectives

Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose $\textbf{multi-objective decoding (MOD)}$, a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weightings over different objectives. We exploit a common form among a family of $f$-divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy. Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method. Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8% overall reward improvement when equally optimizing towards $3$ objectives. Moreover, we experiment with MOD on combining three fully-finetuned LLMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0% and achieves 7.9--33.3% improvement across other three metrics ($\textit{i.e.}$, Codex@1, GSM-COT, BBH-COT).

cs.LG↗

arXiv2024

Rethinking Transformers in Solving POMDPs

Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architecture, namely Transformers, in Partially Observable Markov Decision Processes (POMDPs) and reveals its theoretical limitations. We establish that regular languages, which Transformers struggle to model, are reducible to POMDPs. This poses a significant challenge for Transformers in learning POMDP-specific inductive biases, due to their lack of inherent recurrence found in other models like RNNs. This paper casts doubt on the prevalent belief in Transformers as sequence models for RL and proposes to introduce a point-wise recurrent structure. The Deep Linear Recurrent Unit (LRU) emerges as a well-suited alternative for Partially Observable RL, with empirical results highlighting the sub-optimal performance of the Transformer and considerable strength of LRU.

cs.LG↗

arXiv2023

H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation

Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose a human $\textbf{H}$and$\textbf{-In}$formed visual representation learning framework to solve difficult $\textbf{Dex}$terous manipulation tasks ($\textbf{H-InDex}$) with reinforcement learning. Our framework consists of three stages: (i) pre-training representations with 3D human hand pose estimation, (ii) offline adapting representations with self-supervised keypoint detection, and (iii) reinforcement learning with exponential moving average BatchNorm. The last two stages only modify $0.36\%$ parameters of the pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study 12 challenging dexterous manipulation tasks and find that H-InDex largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code is available at https://yanjieze.com/H-InDex .

cs.LG↗

arXiv2022

A Novel Gradient Descent Least Squares (GDLS) Algorithm for Efficient SMV Gridless Line Spectrum Estimation with Applications in Tomographic SAR Imaging

This paper presents a novel efficient method for gridless line spectrum estimation problem with single snapshot, namely the gradient descent least squares (GDLS) method. Conventional single snapshot (a.k.a. single measure vector or SMV) line spectrum estimation methods either rely on smoothing techniques that sacrifice the array aperture, or adopt the sparsity constraint and utilize compressed sensing (CS) method by defining prior grids and resulting in the off-grid problem. Recently emerged atomic norm minimization (ANM) methods achieved gridless SMV line spectrum estimation, but its computational complexity is extremely high; thus it is practically infeasible in real applications with large problem scales. Our proposed GDLS method reformulates the line spectrum estimations problem into a least squares (LS) estimation problem and solves the corresponding objective function via gradient descent algorithm in an iterative fashion with efficiency. The convergence guarantee, computational complexity, as well as performance analysis are discussed in this paper. Numerical simulations and real data experiments show that the proposed GDLS algorithm outperforms the state-of-the-art methods e.g., CS and ANM, in terms of estimation performances. It can completely avoid the off-grid problem, and its computational complexity is significantly lower than ANM. Our method has been tested in tomographic SAR (TomoSAR) imaging applications via simulated and real experiment data. Results show great potential of the proposed method in terms of better cloud point performance and eliminating the gridding effect.

eess.SP↗