arXiv Science⌕ Search

arXiv subjects

Vijay Gupta

Publications and source records attributed to Vijay Gupta.

At least 19 recordsLinked to original sources

Federated TD Learning in Heterogeneous Environments with Average Rewards: A Two-timescale Approach with Polyak-Ruppert Averaging

Federated Reinforcement Learning (FRL) provides a promising way to speedup training in reinforcement learning when one has access to multiple edge devices that can operate in parallel. Recently, it has been shown that even when these edge devices have access to different dynamic models---such as cars or robots with differing system configurations---an optimal convergence rate that has a linear speedup proportional to the number of devices is achievable. However, this result requires that the algorithm's stepsize be chosen in a manner that depends on the unknown model parameters. Also, it applies only to a discounted setting, which has been argued to fit episodic tasks better than continuing control tasks. In this paper, we obtain finite-time bounds for heterogeneous FRL with average rewards. We show that optimal convergence rate and linear speedup are achievable even with a universal stepsize choice, independent of the underlying dynamics. To achieve our result, we modify the existing one-timescale FRL method to a novel two-timescale variant that additionally incorporates iterate averaging. We remark that this idea also extends to other settings already studied in the literature, such as the homogeneous FRL or the heterogeneous case with a discounted cost function.

cs.LG↗

Free Lunch Uncertainty: Sequence-Level Rényi Entropy from Particle Decoding for Adaptive LLM Reasoning

Quantifying uncertainty over an entire LLM completion is critical for selective prediction and adaptive compute. Zero-cost signals such as mean token entropy and log-probability are local to tokens or to one completion, and self-consistency needs many extra completions. We show that Sequential Monte Carlo (SMC) decoding for power sampling already estimates the normalizing constant of the power distribution, which gives the sequence-level R'enyi entropy (SRE) of the model's completion distribution at zero marginal cost, with a guarantee at every particle budget. A R'enyi chain rule extends this readout to a family of stepwise entropies of any order. We prove that orders above one are tied to the most likely token of each step, which explains why they track log-probability, while the order-1/2 member accumulates the log of per-step search effort. This member is the best zero-cost selective-prediction signal in 10 of 13 settings. On MATH500 its AUROC is 0.022 to 0.044 above log-probability with all four models and within length quartiles, and the same order is selected when each setting is held out. An 8-particle probe identifies the problems that benefit from more particles, and a two-pass policy matches 64-particle accuracy on MATH500 with 34.2% fewer particles over three seeds.

cs.LG↗

Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learning

Average-reward reinforcement learning requires estimating the gain and a state-dependent bias, which solves the Poisson equation only up to an additive constant. Distributional analogues of the bias inherit this same ambiguity. We introduce a quotient-space formulation in which state-indexed laws are identified up to a common translation, together with a categorical parameterization that respects this symmetry. On this space, we define a projected average-reward distributional operator and show that it is well-defined, non-expansive in a coordinate Cramér metric, and admits fixed points. We interpret the projected fixed points through geometrically-stopped differential returns, whose categorical operators are contractive and whose fixed points converge in the Abel sense to a canonical fixed point of the projected undiscounted operator. We show that the means of the projected fixed points match the scalar bias up to an error caused by endpoint clipping of the categorical projections, and we bound this error. Finally, we establish almost-sure convergence and finite-iteration residual bounds for temporal-difference recursions with gain-centered rewards under i.i.d. and Markovian sampling. When the gain is unknown, we couple the recursion with an online gain estimator and prove the same guarantees under Markovian sampling, with a bound on the gain error.

cs.LG↗

Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length $S$ of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length $S$ is obtained. We establish that the algorithm achieves $\tilde{\mathcal{O}}({f(S,M_δ)\sqrt{T/S}})$ Bayes regret after $T$ time steps, where $M_δ$ characterizes the \emph{dissimilarity} between the two systems and $f(S,M_δ)$ is a function of $S$ and $M_δ$. When $M_δ$ is sufficiently small, the proposed algorithm achieves $\tilde{\mathcal{O}}({\sqrt{T/S}})$ Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.

eess.SY↗

Real-to-Sim Calibration of Bosch Etching Simulators with Few-Shot XSEM Data and Hybrid Search

The Bosch process is widely used in MEMS and TSV fabrication, but accurate prediction of etching outcomes remains difficult without calibration against real measurements, which are sparse and costly to obtain. This work presents a few-shot real-to-sim calibration framework that decouples simulator parameters from direct process variables to infer a recipe-specific simulator parameter vector from sparse measurements, enabling reliable prediction of etch depth and sidewall angle under held-out process conditions. We first formulate Bosch profile prediction as a few-shot real-to-sim calibration problem, where a small set of real XSEM measurements is used to identify a recipe-specific ViennaPS parameter vector $z_r^*$ that minimizes the discrepancy between simulated and measured etch depth and sidewall angle. To search this parameter space efficiently, the framework combines local exploration, LLM-guided candidate generation, and XGBoost-based surrogate screening, while retaining ViennaPS simulation as the final verification step for each selected candidate. Experiments on two Bosch etching recipes show that the proposed framework accurately predicts held-out etching results, achieving normalized global errors of 0.340 and 0.249 for the 2 recipes respectively.

cs.CE↗

Multi-Agent Deferred-Decision Trajectory Optimization with Anytime-Valid Target Retention for Moving Candidates

We study multi-agent terminal interception when one of several moving candidates is the unknown target and both target identity and candidate terminal locations are uncertain. We develop a receding-horizon deferred-decision trajectory optimization (DDTO) controller that combines conformal terminal-location prediction, sequential candidate rejection, capacity-constrained assignment, and candidate-contingent planning. Under exchangeability conditions for terminal and identity residuals, with predictable querying, we establish fixed-time terminal-location coverage and anytime-valid target retention. At the final replan, when one control step remains, we bound the joint probability that a tightened-set DDTO plan passes nonlinear rollout verification but capture fails. The closed-loop controller applies fallback guidance to any agent without a verified DDTO plan and, across 100 paired scenarios, captures 79% of targets under the strict control-effort budget and 85% when that budget is relaxed.

eess.SY↗

Reward-Rate Congestion Games and Replicator--Dinkelbach Dynamics

Reward rate is a key performance criterion in cyber-physical and robotic systems where time, workload, and coordination costs are limiting resources. We introduce reward-rate congestion games, where agents seek to maximize reward per unit execution time. The direct reward-rate game is generally not an exact potential game. We develop a Dinkelbach-based framework in which, for every fixed Dinkelbach parameter, the transformed game is an exact potential game. This yields a potential-level Dinkelbach iteration that terminates finitely at the optimal potential reward rate when the inner potential maximization problem is solved globally. We also provide a sufficient condition under which an equilibrium of the transformed game is an equilibrium of the original reward-rate game. To optimize aggregate performance, we introduce marginal externality corrections that make the corrected potential coincide with the Dinkelbach-transformed social reward-rate objective, thereby enabling optimization of the social reward rate. Finally, we develop a continuous-time replicator--Dinkelbach dynamics for reward-rate population games coupling fast replicator dynamics with a slow reward-rate update. We establish convergence of the fixed-parameter replicator dynamics, global asymptotic and local exponential stability of the reduced Dinkelbach dynamics, and local exponential stability of the coupled system for sufficiently slow Dinkelbach updates. The framework is illustrated on a continuous task-allocation problem.

eess.SY↗

Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes

Coupled-dynamics environments expose the one-step outcomes that would follow from several possible counterfactual actions under a common realization of exogenous randomness. The ordinary Markov decision process formalism allows one to reason about the marginal law of each action but discards dependence across these counterfactual outcomes. The Joint Markov decision process (JMDP) formalism preserves that dependence. Prior work established the formalism and solved the fixed-policy joint moment evaluation problem in JMDPs. This paper develops optimal-control methods. We define a nonparametric distributional Bellman optimality operator for JMDPs, and prove that when the induced marginal MDP has a unique optimal policy, its iterates converge in Wasserstein distance to the optimal joint return law. For the first two moments, we establish convergence under a weaker condition that permits several mean-optimal actions as long as their tie resolutions share a second-moment fixed point. We also derive sampled targets for neural approximation.

cs.LG↗

Stabilization Limits of Payoff-Based Higher-Order Replicator Dynamics

Replicator dynamics (RD) is a fundamental model in learning in games, connecting evolutionary game theory and online learning. This paper studies payoff-based higher-order variants of RD represented as a cascade interconnection between an integrator in parallel with an auxiliary linear time-invariant (LTI) system and the softmax mapping. We investigate learnability of Nash equilibria under this Nash-stationary learning rule. First, we revisit recent results that establish convergence to Nash Equilibrium whenever the auxiliary LTI system is strictly passive and prove a converse passivity result: if the auxiliary LTI system is not passive, then there exists a static strictly contractive game whose interior Nash equilibrium is unstable under the closed-loop learning dynamics. Second, we show that there exists a class of games with isolated interior Nash equilibria that cannot be locally asymptotically stabilized by any payoff-based higher-order RD whose auxiliary LTI system is asymptotically stable and strictly proper. Finally, we show that if Nash stationarity (i.e., all Nash equilibria are stationary points of the learning dynamics) is relaxed, then generalized exponential RD (Ex-RD) can locally asymptotically stabilize a logit equilibrium for any continuously differentiable game. The stabilized equilibrium can be viewed as an entropy-regularized approximate Nash equilibrium.

eess.SY↗

Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.

cs.LG↗

Control-Oriented System Identification: Classical, Learning, and Physics-Informed Approaches

We survey classical, machine learning, and data-driven system identification approaches to learn control-relevant and physics-informed models of dynamical systems. Recently, machine learning approaches have enabled system identification from noisy, high-dimensional, and complex data. However, their utility is limited by their ability to provide provable guarantees on control-relevant properties. Meanwhile, control theory has identified several properties that are useful in analysis and control synthesis, such as dissipativity, monotonicity, energy conservation, and symmetry-preserving structures. We posit that merging system identification with such control-relevant or physics-informed properties can provide useful inductive bias, enhance explainability, enable control synthesis with provable guarantees, and improve sample complexity. We formulate system identification as an optimization problem where control-relevant properties can be enforced through direct parameterization (constraining the model structure to satisfy a desired property by construction), soft constraints (encouraging control-relevant properties through regularization or penalty terms), and hard constraints (imposing control-relevant properties as constraints in the optimization problem). Through this lens, we survey methods to learn physics-informed and control-relevant models spanning classical linear and nonlinear system identification, machine learning approaches, and direct identification through data-driven and behavioral representations. We also provide several expository examples that are accompanied by code and brief tutorials on a public Github repository. We also describe challenging directions for future research, including identification in networked, switched, and time-varying systems, experiment design, and bridging the gaps between data-driven, learning-based, and control-oriented approaches.

eess.SY↗

Comparison of Non-Deterministic Nonlinear Systems

We characterize a notion of system comparison, termed as $(T_e,γ,δ)$-similarity, for non-deterministic nonlinear systems. Building on a similar notion recently proposed for stable linear systems, the proposed notion characterizes the dissimilarity between the outputs, measured using the $L_2$ norm, of two nonlinear dynamical systems in terms of their inputs and disturbances. By establishing a relationship between $(T_e,γ,δ)$-similarity and differential dissipativity, we establish equivalence between $(T_e,γ,δ)$-similarity of nonlinear systems and the $(T_e,γ,δ)$-similarity of their differential dynamics. We characterize the $(T_e,γ,δ)$-similarity for nonlinear systems as a Linear Matrix Inequality feasibility problem and also provide necessary and sufficient conditions for solving this feasibility problem. We demonstrate the utility of the proposed notion through its use in two applications: (i) robust hierarchical control applied to a planar aircraft and (ii) the improvement (or design) of abstract models applied to the Moore-Greitzer model and an electronic circuit.

eess.SY↗

Active Sensing and Deferred-Decision Trajectory Optimization for Robust Target Identification

We study trajectory optimization in mobile sensing systems that must identify which member of a finite candidate set is the true target, while maintaining reachability to all potential candidate targets, under resource constraints. Deferred-Decision Trajectory Optimization (DDTO) addresses this setting by computing trajectories that reach individual targets but remain coincident for as long as possible before separating toward different targets. We propose Active-Sensing DDTO (AS-DDTO), which extends DDTO by adding a trajectory-dependent information-acquisition term to the planning objective. The resulting planner maintains reachability to candidate targets while biasing the coincident portion of the trajectories toward regions that enable earlier target identification. The framework supports Bayesian updates and conformal candidate-set updates for distance-dependent sensing. We derive a mixed-integer conic reformulation and provide guarantees on recursive feasibility, belief concentration, and fixed-time coverage for the raw conformal candidate set. Numerical simulations show improved target identification compared with standard DDTO under distance-dependent sensing uncertainty and limited sensing budget.

eess.SY↗

Robust Filtering of Lévy-driven Stochastic Models

We study robust nonlinear filtering for stochastic models driven by Lévy processes, where the signal and observation processes are coupled through common Brownian and jump noise. Robustness, defined as the continuous dependence of the filter on the observation path, is essential whenever the observation process deviates from the idealized model, for instance when a path must be reconstructed from discrete-time samples. This question is well understood for continuous semimartingale systems but largely open in the presence of jumps. We construct a version of the filter and establish its continuity in two regimes. For processes with finitely many jumps on compact intervals, we prove continuity in both the rough $p$-variation and $p$-variation topologies on cadlag path space, without requiring a separability condition on the jump coefficients. For processes with infinitely many jumps, we prove continuity in a modified rough $p$-variation topology adapted to cadlag geometric rough paths, under an additional separability assumption. In both cases, our approach relies on Stratonovich and Marcus flow decompositions rather than the Itô-based methods of recent work. The resulting geometric rough-path lifts yield pathwise convergence guarantees and can be constructed directly from discrete observations without knowledge of the underlying probability law.

math.PR↗

The Variational Approach in Filtering and Correlated Noise

The variational formulation of nonlinear filtering due to Mitter and Newton characterizes the filtering distribution as the unique minimizer of a free energy functional involving the relative entropy with respect to the prior and an expected energy. This formulation rests on an absolute continuity condition between the joint path measure and a product reference measure. We prove that this condition necessarily fails whenever the signal and observation diffusions share a common noise source. Specifically we show that the joint and product measures are mutually singular, so no choice of reference measure can salvage the formulation. We then introduce a conditional variational principle that replaces the prior with a reference measure that preserves the noise correlation structure. This generalization recovers the Mitter--Newton formulation as a special case when the noises are independent, and yields an explicit free energy characterization of the filter in the linear correlated-noise setting.

math.PR↗

Parameter-free Optimal Rates for Nonlinear Semi-Norm Contractions with Applications to $Q$-Learning

Algorithms for solving \textit{nonlinear} fixed-point equations -- such as average-reward \textit{$Q$-learning} and \textit{TD-learning} -- often involve semi-norm contractions. Achieving parameter-free optimal convergence rates for these methods via Polyak--Ruppert averaging has remained elusive, largely due to the non-monotonicity of such semi-norms. We close this gap by (i.) recasting the averaged error as a linear recursion involving a nonlinear perturbation, and (ii.) taming the nonlinearity by coupling the semi-norm's contraction with the monotonicity of a suitably induced norm. Our main result yields the first parameter-free $\tilde{O}(1/\sqrt{t})$ optimal rates for $Q$-learning in both average-reward and exponentially discounted settings, where $t$ denotes the iteration index. The result applies within a broad framework that accommodates synchronous and asynchronous updates, single-agent and distributed deployments, and data streams obtained either from simulators or along Markovian trajectories.

cs.LG↗

Convergence of Payoff-Based Higher-Order Replicator Dynamics in Contractive Games

We study the convergence properties of a payoff-based higher-order version of replicator dynamics, a widely studied model in evolutionary dynamics and game-theoretic learning, in contractive games. Recent work has introduced a control-theoretic perspective for analyzing the convergence of learning dynamics through passivity theory, leading to a classification of learning dynamics based on the passivity notion they satisfy, such as \textdelta-passivity, equilibrium-independent passivity, and incremental passivity. We leverage this framework for the study of higher-order replicator dynamics for contractive games, which form the complement of passive learning dynamics. Standard replicator dynamics can be represented as a cascade interconnection between an integrator and the softmax mapping. Payoff-based higher-order replicator dynamics include a linear time-invariant (LTI) system in parallel with the existing integrator. First, we show that if this added system is strictly passive and asymptotically stable, then the resulting learning dynamics converge locally to the Nash equilibrium in contractive games. Second, we establish global convergence properties using incremental stability analysis for the special case of symmetric matrix contractive games.

eess.SY↗

Linear Quadratic Control with Non-Markovian and Non-Semimartingale Noise Models

The standard linear quadratic Gaussian (LQG) framework assumes a Brownian noise process and relies on classical stochastic calculus tools, such as those based on Itô calculus. In this paper, we solve a generalized linear quadratic optimal control problem where the process and measurement noises can be non-Markovian and non-semimartingale stochastic processes with sample paths that have low Hölder regularity. Since these noise models do not, in general, permit the use of the standard Itô calculus, we employ rough path theory to formulate and solve the problem. By leveraging signature representations and controlled rough paths, we derive the optimal state estimation and control strategies.

eess.SY↗