arXiv subjects

Xun Li

Publications and source records attributed to Xun Li.

At least 19 recordsLinked to original sources

arXiv2026

Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.

math.OC↗

arXiv2026

Continuous-Time Mean-Variance Portfolio Selection Reinforcement Learning Problems for a Markov Regime-Switching System with No-Shorting Constraints

This paper is the first to study the continuous-time mean-variance portfolio selection problem with no-shorting constraints in a Markov regime-switching market using reinforcement learning (RL). The main challenge stems from the interaction between random regime shifts and portfolio non-negativity constraints, which typically causes the coupled Hamilton-Jacobi-Bellman (HJB) equations to lose smoothness across different state regions. Additionally, directly imposing no-shorting constraints on randomized policies disrupts their analytically tractable Gaussian structure, complicating both exploratory modeling and reinforcement learning. To address these challenges, we propose an auxiliary exploratory approach that limits the policy to a Gaussian distribution and applies the no-shorting constraint solely to its mean. We explicitly characterize the optimal policy for this auxiliary problem and demonstrate that, by appropriately selecting the exploration variance, its optimal mean policy exactly matches the optimal policy of the original constrained mean-variance problem. Building upon this analytical framework, we develop a model-free reinforcement learning algorithm that learns the optimal portfolio policy directly from trajectory data. Numerical experiments support the theoretical results and show that the proposed method can effectively recover the optimal no-shorting portfolio in a regime-switching market.

math.OC↗

arXiv2026

Policy Evaluation and Policy Gradient for Continuous-Time Markov Regime-Switching Systems

We study policy evaluation (PE) and policy gradient (PG) for continuous-time stochastic control with Markov regime-switching. The main challenges are the coexistence of diffusion and regime jumps, the coupling of regime-dependent value functions, and the construction of model-free learning rules from the coupled generators. We establish a martingale characterization of PE that incorporates both Brownian and switching martingales, and extend martingale loss and orthogonality methods to this setting. We further derive a coupled PG equation and a trajectory-based gradient representation that, under an action-independent switching generator, avoids explicit knowledge of the system coefficients and transition rates. These results lead to offline and online actor-critic algorithms. We also establish convergence and study the finite-grid approximation required for implementation. Numerical experiments for a regime-switching linear-quadratic problem illustrate the proposed methods.

math.OC↗

arXiv2026

Optimal Transport for Continuous-Time Mean-Variance Portfolio Selection

The continuous-time mean-variance portfolio selection problem is investigated in this paper through a rigorous integration of backward stochastic differential equation (BSDE) and Kantorovich duality theory. We propose a novel analytical framework for addressing feedback control in McKean-Vlasov stochastic systems. By solving three coupled systems of ordinary differential equations (ODEs), we derive an explicit, closed-loop optimal feedback investment strategy.

math.OC↗

arXiv2026

Entropy Regularization under No-Shorting: Structure Preservation and Convexity in Dynamic Mean-Variance Portfolio Selection

This paper studies entropy regularization for the discrete-time dynamic mean--variance portfolio problem with no-shorting constraints. In the unconstrained setting, entropy regularization can be introduced without changing the basic quadratic structure of the dynamic program. Under no-shorting, this property is lost: a standard fixed-temperature formulation generally preserves convexity but no longer preserves the homogeneous Bellman structure and the associated piecewise-linear feedback form. We introduce a state-dependent temperature that is matched to the homogeneity of the original mean--variance problem. This restores the piecewise-quadratic value function and the finite-dimensional Bellman recursion. The resulting formulation, however, makes convexity depend on the continuation coefficients generated by the soft recursion. We show that negative continuation coefficients lead to loss of convexity and prove that a common sufficiently small temperature preserves the Bellman structure and convexity over the entire finite horizon. The original no-shorting solution is recovered as the temperature vanishes. We also develop a sample-based implementation for unknown return distributions. Numerical and empirical results illustrate both the structural advantage of the state-dependent temperature and the practical relevance of the convexity-preserving temperature restriction.

q-fin.PM↗

arXiv2026

Mean-variance portfolio selection: connections between mean-field systems and forward-backward stochastic systems under initial-terminal cost coupling

This paper investigates the connection between mean-field systems and forward-backward stochastic systems arising from the mean-variance portfolio selection problem. By introducing the conditional expectation process of terminal wealth, the mean-field term is represented by a backward state, leading to a class of control problems for forward-backward stochastic systems with coupled initial-terminal values. The coupling between the terminal value of the forward state and the initial value of the backward state in the cost functional prevents classical methods for forward-backward stochastic optimal control from being directly applicable. To overcome this difficulty, we establish a stochastic maximum principle for control problems with running costs and coupled initial-terminal values, and provide sufficient conditions for optimality. Furthermore, we investigate a class of generalized stochastic linear-quadratic control problems with initial-terminal coupling, allowing the control weight in the running cost to be indefinite, and derive the optimal feedback representation via a Riccati equation. The classical mean-variance portfolio selection model is shown to be a special case, for which the corresponding efficient investment strategy and efficient frontier are obtained explicitly, and numerical simulations are presented to illustrate the results and examine parameter sensitivity and its financial implications.

math.OC↗

arXiv2026

Stochastic linear-quadratic control problem for regime-switching jump-diffusion system and its application to finance

This paper investigates a stochastic linear-quadratic (SLQ) control problem for a regime-switching jump-diffusion system. Unlike traditional regime-switching diffusion systems that couple a diffusion process with a Markov chain, we incorporate the jumps of the Markov chain into the state equation. This modeling methodology effectively captures potential gains or losses of the system during the regime transitions. It should be noted that the introduction of Markov chain jumps into the state equation leads to increased complexity in the corresponding coupled differential Riccati equations (CDREs), thereby rendering the solvability of the control problem more challenging. Under the assumption that the cost functional is uniformly convex, we establish the unique solvability of the corresponding CDREs. Building upon this foundation, we derive a closed-loop representation for the unique open-loop optimal control. Finally, we apply our theoretical results to the mean-variance portfolio selection problem in a regime-switching financial market with Markov chain jumps and obtain its efficient frontier.

math.OC↗

arXiv2026

Dynamic mean-variance portfolio selection with no-shorting constraints and unknown investment opportunity sets

We study continuous-time mean-variance portfolio selection with no-shorting constraints and unknown investment opportunity sets from a reinforcement learning (RL) perspective. The problem is a constrained stochastic linear -- quadratic control problem for which the entropy-regularized exploratory formulation of Wang et al. (2020) leads to difficulty in theoretical analysis, because enforcing the constraint on the support of randomized policies nullifies the tractable Gaussian exploration. To tackle this challenge, we introduce an auxiliary exploratory problem without entropy in which exploratory policies are still Gaussian whose samples may violate the no-shorting requirement but their means satisfy it. We then prove that, for a suitable choice of exploration variance, the mean of the optimal Gaussian policy of the auxiliary problem coincides with the optimal policy of the original problem. Motivated by this theoretical result, we develop a model-free RL algorithm that learns the optimal policy of the auxiliary (and hence the original) problem directly from trajectory data without estimating the investment opportunity set. A numerical example demonstrates the performance of the proposed algorithm.

math.OC↗

arXiv2026

Closed-loop $α$-Potential Stochastic Differential Games via a BSDE Approach

In this paper, we study the closed-loop $α$-potential stochastic differential game (SDG) problem as a continuation of our prior research on open-loop control (see \cite{GLZ2025}). By utilizing the backward stochastic differential equation (BSDE) approach, we derive a precise estimate for the parameter $α$. Compared to our earlier work \cite{GLZ2025}, this study incorporates both first- and second-order sensitivity state processes, as well as the sensitivity of the control process. A distinguishing feature of this work is that, in the context of $N$-player heterogeneous agent games involving mean-field type interactions, we derive an $N$-uniform upper bound for the minimal potential approximation error. In contrast to the corresponding open-loop estimates, the closed-loop bound contains feedback-induced contributions that need not vanish with $N$. Consequently, our present estimate does not in general guarantee $α\rightarrow 0$.

math.OC↗

arXiv2026

Continuous-time q-learning for Markov regime switching system under Tsallis entropy

This paper studies continuous-time q-learning (the continuous-time counterpart of Q-learning) for a Markov regime-switching system under Tsallis entropy regularization. The Tsallis entropy regularization yields an optimal policy distribution that may not necessarily be a Gibbs measure, thereby complicating algorithm design. Furthermore, to address the limited universality of current continuous-time regime-switching reinforcement learning algorithms (often restricted to the exploratory mean-variance framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time reinforcement learning method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. We further design two q-learning algorithms that differ based on whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory mean-variance portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.

math.OC↗

arXiv2026

Tree-of-Ideas: Automated Research Ideation via Cross-Trajectory Reasoning over Scholarly Evolution

Effective research ideation requires moving beyond a static understanding of prior work to trace how research problems and solutions evolve across the literature. Existing methods either treat papers as unstructured context or model scholarly evolution as isolated citation chains, overlooking interactions among research trajectories. We propose Tree-of-Ideas (ToI), a two-stage framework. EvoTrace reconstructs branching scholarly trajectories from citations, tracking evolving methods, resolved problems, and gaps. EvoAgent then reasons across trajectories to identify convergent problems and complementary solutions, generating grounded research ideas. Across six AI research topics, ToI achieves the highest score among automatic methods (6.27 vs. 5.36 for the strongest baseline on a 10-point scale), with strong Novelty (6.36) and Groundedness (7.00). Also, its score approaches that of human-paper references (6.29), demonstrating the value of cross-path evolutionary reasoning.

cs.AI↗

arXiv2026

Inverse reinforcement learning for indefinite mean-field social optimization with multiplicative noise

This paper studies the inverse reinforcement learning (RL) problem for linear-quadratic mean-field (MF) social optimization. The considered system features multiplicative noise and indefinite cost weights, which violate standard convexity assumptions and pose analytical challenges. The goal is to recover unknown social cost weights from expert demonstrations and reproduce the optimal control policies. This requires solving coupled stochastic algebraic Riccati equations and Lyapunov equations with unknown system dynamics. To this end, we first propose a model-based inverse RL algorithm with two sequential loops that separately handle individual and MF dynamics, and we prove its convergence and closed-loop stabilizability. Moreover, we characterize the non-uniqueness of the recovered cost weights. To eliminate reliance on system dynamics, we develop a model-free inverse RL algorithm using integral RL and least-squares identification, which requires only measured trajectory data satisfying mild rank conditions. Finally, numerical simulations validate the effectiveness of the proposed approaches.

math.OC↗

arXiv2026

Model-Free Q-Learning for Infinite-Horizon Stochastic Linear Quadratic Problems with Regime Switching

This paper addresses infinite-horizon continuous-time stochastic linear quadratic optimal control problems with regime switching. We propose a paradigm shift from model-based design by adopting an adaptive dynamic programming approach, specifically developing on-policy and off-policy Q-learning algorithms that learn the optimal controller solely from online state trajectory data. The theoretical core of our work consists of a complete proof of the equivalence between the on- and off-policy architectures, alongside a rigorous analysis establishing the stability of the closed-loop system and the convergence of the algorithms to the optimal solution. For computational tractability, we implement these algorithms using vectorization and Kronecker product algebra. The theoretical results are corroborated by numerical case studies that clearly demonstrate the operational effectiveness and practical feasibility of the proposed model-free control strategy.

math.OC↗

arXiv2026

Constrained Zero-Sum Stochastic Linear-Quadratic Differential Game for Jump-Diffusion Systems with Random Coefficients

This paper studies a two-player zero-sum stochastic linear-quadratic (SLQ) differential game for controlled jump-diffusion systems with random coefficients, where the controls of both players are constrained to nonempty closed convex cones. Under a uniform convexity--concavity condition, we establish the existence and uniqueness of an open-loop saddle point and characterize it by a forward--backward stochastic differential equation with jumps (FBSDEJ) together with cone-type variational inequalities. Assuming the existence of positive bounded solutions to the associated system of indefinite extended stochastic Riccati equations with jumps (IESREJs), we derive a feedback-form representation of the unique open-loop saddle point by constructing predictable minimax selectors and combining the Meyer--Itô formula with jumps, and the FBSDEJ characterization. Finally, under additional structural conditions, we prove the existence of positive bounded solutions to the IESREJs by a double-truncation approximation and a multidimensional BSDEJ comparison theorem.

math.OC↗

arXiv2026

An Optimization-Based Framework for Solving Forward-Backward Stochastic Differential Equations: Convergence Analysis and Error Bounds

Forward-backward stochastic differential equations have recently become a key focus in the computational field, and their role in continuous-time stochastic optimal control and reinforcement learning has grown increasingly prominent. In this paper, we develop an optimization-based framework for solving coupled forward-backward stochastic differential equations, which naturally arise in stochastic optimal control through the stochastic maximum principle and related Hamiltonian systems. We introduce an integral-form objective function and prove its equivalence to the error between consecutive Picard iterates. Our convergence analysis establishes that minimizing this objective generates sequences that converge to the true solution. We provide explicit upper and lower bounds that relate the objective value to the error between trial and exact solutions. We validate the proposed objective and its theoretical interpretation using two analytical test cases, and further illustrate its numerical applicability on a nonlinear stochastic optimal control problem with up to 1000 dimensions.

math.OC↗

arXiv2026

Douglas-Rachford Splitting for Group-Sparse Feedback Linear-Quadratic Control

In this paper, we study the distributed linear quadratic problem with fixed communication topology (DFT-LQ) and the sparse feedback linear quadratic (SF-LQ) problem through a unified optimization framework. Specifically, both problems are formulated as a nonconvex, nonsmooth optimization problem equipped with an $\ell_0$-penalty under affine constraints. To solve this problem, we first investigate the application of the Douglas-Rachford (DR) splitting algorithm. Under the local condition that the generated iterates remain on a fixed smooth manifold, we establish the convergence of the DR splitting to a stationary point. Furthermore, we characterize this stationary point as the global minimizer of a corresponding DFT-LQ problem. To bypass the restriction of the smooth manifold assumption, we introduce a projected subgradient descent algorithm that achieves global convergence without relying on smooth-manifold structures. This algorithm may serve as a warm-start mechanism that effectively drives the iterates toward the desired smooth manifolds, thereby establishing a favorable initialization where the convergence theory of the DR splitting algorithm becomes fully applicable. Numerical experiments shed light on the effectiveness of the proposed methods in distributed group-sparse controller design.

math.OC↗

arXiv2026

Inverse Optimal Control for Linear Quadratic Problem with Poisson Jumps: Model-Free Inverse Reinforcement Learning Approaches

This paper addresses the inverse optimal control (IOC) problem for stochastic linear systems subject to both Brownian motion and Poisson jumps, using an inverse reinforcement learning (IRL) framework. Given a target feedback gain from an expert, the objective is to identify an equivalent cost functional-specifically, the set of all cost weights-that yields this same gain. To solve this problem when system dynamics are unknown, we propose two model-free, off-policy IRL algorithms that operate entirely from data, circumventing the need to solve the generalized algebraic Riccati equation or compute the cost weights analytically. The first is an inverse Q-learning algorithm that constructs data-driven equations from expert demonstrations to compute the Q-function matrix, with equivalent cost weights updated algebraically and without requiring additional trajectory data. The second is a model-free off-policy inverse policy iteration algorithm that leverages data collected under an initial stabilizing policy, offering a complementary approach suited to different data availability scenarios. Crucially, by decoupling the data-collection behavior policies from the policies being iteratively updated, both algorithms can learn equivalent cost weights from sufficiently excited trajectories without identifying the system dynamics or jump intensity. Numerical simulations validate the effectiveness of the proposed methods.

math.OC↗

arXiv2026

Acoustic Cloning

Cloning refers to producing identical copies of existing objects. Here, we experimentally show how to clone acoustic scattering objects. We acquire a digital twin and bring it back to life - a simple two-step process. First, we use broadband speakers to illuminate the scattering object within a closed receiver aperture. From these recorded reverberative data, we retrieve the object's scattering Green's functions using multidimensional deconvolution. In the second step, the acoustic scatterer is holographically reconstructed using the acquired scattering Green's functions. The hologram scatters any wavefield in real-time exactly like the original object would. Low-latency feedback reproduces all orders of interactions between the physical wavefield and the numerically defined hologram. This two-step process is demonstrated by cloning and modifying several rigid scatterers in a two-dimensional acoustic waveguide. Applications range from fully realistic digital scattering models to efficient metamaterial experimentation.

physics.app-ph↗