arXiv subjects

Adam Wierman

Publications and source records attributed to Adam Wierman.

At least 37 records · Page 2Linked to original sources

arXiv2025

SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer

Deploying reinforcement learning (RL) safely in the real world is challenging, as policies trained in simulators must face the inevitable sim-to-real gap. Robust safe RL techniques are provably safe, however difficult to scale, while domain randomization is more practical yet prone to unsafe behaviors. We address this gap by proposing SPiDR, short for Sim-to-real via Pessimistic Domain Randomization -- a scalable algorithm with provable guarantees for safe sim-to-real transfer. SPiDR uses domain randomization to incorporate the uncertainty about the sim-to-real gap into the safety constraints, making it versatile and highly compatible with existing training pipelines. Through extensive experiments on sim-to-sim benchmarks and two distinct real-world robotic platforms, we demonstrate that SPiDR effectively ensures safety despite the sim-to-real gap while maintaining strong performance.

cs.RO↗

arXiv2025

Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach

Traditional reinforcement learning (RL) assumes the agents make decisions based on Markov decision processes (MDPs) with one-step transition models. In many real-world applications, such as energy management and stock investment, agents can access multi-step predictions of future states, which provide additional advantages for decision making. However, multi-step predictions are inherently high-dimensional: naively embedding these predictions into an MDP leads to an exponential blow-up in state space and the curse of dimensionality. Moreover, existing RL theory provides few tools to analyze prediction-augmented MDPs, as it typically works on one-step transition kernels and cannot accommodate multi-step predictions with errors or partial action-coverage. We address these challenges with three key innovations: First, we propose the \emph{Bayesian value function} to characterize the optimal prediction-aware policy tractably. Second, we develop a novel \emph{Bellman-Jensen Gap} analysis on the Bayesian value function, which enables characterizing the value of imperfect predictions. Third, we introduce BOLA (Bayesian Offline Learning with Online Adaptation), a two-stage model-based RL algorithm that separates offline Bayesian value learning from lightweight online adaptation to real-time predictions. We prove that BOLA remains sample-efficient even under imperfect predictions. We validate our theory and algorithm on synthetic MDPs and a real-world wind energy storage control problem.

cs.LG↗

arXiv2025

KL-regularization Itself is Differentially Private in Bandits and RLHF

Differential Privacy (DP) provides a rigorous framework for privacy, ensuring the outputs of data-driven algorithms remain statistically indistinguishable across datasets that differ in a single entry. While guaranteeing DP generally requires explicitly injecting noise either to the algorithm itself or to its outputs, the intrinsic randomness of existing algorithms presents an opportunity to achieve DP ``for free''. In this work, we explore the role of regularization in achieving DP across three different decision-making problems: multi-armed bandits, linear contextual bandits, and reinforcement learning from human feedback (RLHF), in offline data settings. We show that adding KL-regularization to the learning objective (a common approach in optimization algorithms) makes the action sampled from the resulting stochastic policy itself differentially private. This offers a new route to privacy guarantees without additional noise injection, while also preserving the inherent advantage of regularization in enhancing performance.

cs.LG↗

arXiv2025

Conformal Risk Training: End-to-End Optimization of Conformal Risk Control

While deep learning models often achieve high predictive accuracy, their predictions typically do not come with any provable guarantees on risk or reliability, which are critical for deployment in high-stakes applications. The framework of conformal risk control (CRC) provides a distribution-free, finite-sample method for controlling the expected value of any bounded monotone loss function and can be conveniently applied post-hoc to any pre-trained deep learning model. However, many real-world applications are sensitive to tail risks, as opposed to just expected loss. In this work, we develop a method for controlling the general class of Optimized Certainty-Equivalent (OCE) risks, a broad class of risk measures which includes as special cases the expected loss (generalizing the original CRC method) and common tail risks like the conditional value-at-risk (CVaR). Furthermore, standard post-hoc CRC can degrade average-case performance due to its lack of feedback to the model. To address this, we introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control during model training or fine-tuning. Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance over post-hoc approaches on applications to controlling classifiers' false negative rate and controlling financial risk in battery storage operation.

cs.LG↗

arXiv2025

Enhancing Data Center Low-Voltage Ride-Through

Data center loads have expanded significantly in recent years. Compared to traditional loads, data centers are highly sensitive to voltage deviations and thus their protection mechanisms trip more proactively during voltage fluctuations. During a grid fault, simultaneous tripping of large-scale data centers can further destabilize the transmission system and even lead to cascading failures. In response, transmission system operators are imposing voltage ride-through (VRT) requirements for data centers. In this work, we enhance the VRT capability of data centers by designing voltage controllers for their internal power distribution network. We first systematically analyze VRT standards and the controllable resources related to data centers. These resources enable the design of voltage control strategies to regulate voltages internal to the data center, thereby allowing loads to remain online during voltage disturbances from the external transmission grid. We study and contrast both centralized and decentralized controllers that unify the control of heterogeneous flexible resources. Additionally, we construct an integrated test system that simulates both the transient fault response of the transmission system and the data center distribution network. Case studies demonstrate that the proposed voltage control mechanisms provide effective yet simple solutions to enhance data center low-voltage ride-through capability.

eess.SY↗

arXiv2025

Leveraging Predictions in Power System Voltage Control: An Adaptive Approach

High variability of solar PV and sudden changes in load (e.g., electric vehicles and storage) can lead to large voltage fluctuations in the distribution system. In recent years, a number of controllers have been designed to optimize voltage control. These controllers, however, almost always assume that the net load in the system remains constant over a sufficiently long time, such that the control actions converge before the load changes again. Given the intermittent and uncertain nature of renewable resources, it is becoming important to explicitly consider net load that is time-varying. This paper proposes an adaptive approach to voltage control in power systems with significant time-varying net load. We leverage advances in short-term load forecasting, where the net load in the system can be partially predicted using local measurements. We integrate these predictions into the design of adaptive controllers, and prove that the overall control architecture achieves input-to-state stability in a decentralized manner. We optimize the control policy through reinforcement learning. Case studies are conducted using time-varying load data from a real-world distribution system.

eess.SY↗

arXiv2025

Competitive Algorithms for Multi-Agent Ski-Rental Problems

This paper introduces a novel multi-agent ski-rental problem that generalizes the classical ski-rental dilemma to a group setting where agents incur individual and shared costs. In our model, each agent can either rent at a fixed daily cost, or purchase a pass at an individual cost, with an additional third option of a discounted group pass available to all. We consider scenarios in which agents' active days differ, leading to dynamic states as agents drop out of the decision process. To address this problem from different perspectives, we define three distinct competitive ratios: overall, state-dependent, and individual rational. For each objective, we design and analyze optimal deterministic and randomized policies. Our deterministic policies employ state-aware threshold functions that adapt to the dynamic states, while our randomized policies sample and resample thresholds from tailored state-aware distributions. The analysis reveals that symmetric policies, in which all agents use the same threshold, outperform asymmetric ones. Our results provide competitive ratio upper and lower bounds and extend classical ski-rental insights to multi-agent settings, highlighting both theoretical and practical implications for group decision-making under uncertainty.

cs.LG↗

arXiv2025

Fast Non-Episodic Adaptive Tuning of Robot Controllers with Online Policy Optimization

We study online algorithms to tune the parameters of a robot controller in a setting where the dynamics, policy class, and optimality objective are all time-varying. The system follows a single trajectory without episodes or state resets, and the time-varying information is not known in advance. Focusing on nonlinear geometric quadrotor controllers as a test case, we propose a practical implementation of a single-trajectory model-based online policy optimization algorithm, M-GAPS,along with reparameterizations of the quadrotor state space and policy class to improve the optimization landscape. In hardware experiments,we compare to model-based and model-free baselines that impose artificial episodes. We show that M-GAPS finds near-optimal parameters more quickly, especially when the episode length is not favorable. We also show that M-GAPS rapidly adapts to heavy unmodeled wind and payload disturbances, and achieves similar strong improvement on a 1:6-scale Ackermann-steered car. Our results demonstrate the hardware practicality of this emerging class of online policy optimization that offers significantly more flexibility than classic adaptive control, while being more stable and data-efficient than model-free reinforcement learning.

cs.RO↗

arXiv2025

Now More Than Ever, Foundational AI Research and Infrastructure Depends on the Federal Government

Leadership in the field of AI is vital for our nation's economy and security. Maintaining this leadership requires investments by the federal government. The federal investment in foundation AI research is essential for U.S. leadership in the field. Providing accessible AI infrastructure will benefit everyone. Now is the time to increase the federal support, which will be complementary to, and help drive, the nation's high-tech industry investments.

cs.CY↗

arXiv2025

Maximizing the Value of Predictions in Control: Accuracy Is Not Enough

We study the value of stochastic predictions in online optimal control with random disturbances. Prior work provides performance guarantees based on prediction error but ignores the stochastic dependence between predictions and disturbances. We introduce a general framework modeling their joint distribution and define "prediction power" as the control cost improvement from the optimal use of predictions compared to ignoring the predictions. In the time-varying Linear Quadratic Regulator (LQR) setting, we derive a closed-form expression for prediction power and discuss its mismatch with prediction accuracy and connection with online policy optimization. To extend beyond LQR, we study general dynamics and costs. We establish a lower bound of prediction power under two sufficient conditions that generalize the properties of the LQR setting, characterizing the fundamental benefit of incorporating stochastic predictions. We apply this lower bound to non-quadratic costs and show that even weakly dependent predictions yield significant performance gains.

math.OC↗

arXiv2025

Fusing Reward and Dueling Feedback in Stochastic Bandits

This paper investigates the fusion of absolute (reward) and relative (dueling) feedback in stochastic bandits, where both feedback types are gathered in each decision round. We derive a regret lower bound, demonstrating that an efficient algorithm may incur only the smaller among the reward and dueling-based regret for each individual arm. We propose two fusion approaches: (1) a simple elimination fusion algorithm that leverages both feedback types to explore all arms and unifies collected information by sharing a common candidate arm set, and (2) a decomposition fusion algorithm that selects the more effective feedback to explore the corresponding arms and randomly assigns one feedback type for exploration and the other for exploitation in each round. The elimination fusion experiences a suboptimal multiplicative term of the number of arms in regret due to the intrinsic suboptimality of dueling elimination. In contrast, the decomposition fusion achieves regret matching the lower bound up to a constant under a common assumption. Extensive experiments confirm the efficacy of our algorithms and theoretical results.

cs.LG↗

arXiv2025

Robustness and Consistency in Linear Quadratic Control with Untrusted Predictions

We study the problem of learning-augmented predictive linear quadratic control. Our goal is to design a controller that balances \textit{"consistency"}, which measures the competitive ratio when predictions are accurate, and \textit{"robustness"}, which bounds the competitive ratio when predictions are inaccurate. We propose a novel $λ$-confident policy and provide a competitive ratio upper bound that depends on a trust parameter $λ\in [0,1]$ set based on the confidence in the predictions and some prediction error $\varepsilon$. Motivated by online learning methods, we design a self-tuning policy that adaptively learns the trust parameter $λ$ with a competitive ratio that depends on $\varepsilon$ and the variation of system perturbations and predictions. We show that its competitive ratio is bounded from above by $ 1+{O(\varepsilon)}/({Θ(1)+Θ(\varepsilon)})+O(μ_{\mathsf{Var}})$ where $μ_\mathsf{Var}$ measures the variation of perturbations and predictions. It implies that when the variations of perturbations and predictions are small, by automatically adjusting the trust parameter online, the self-tuning scheme ensures a competitive ratio that does not scale up with the prediction error $\varepsilon$.

eess.SY↗

arXiv2025

Learning-Augmented Competitive Algorithms for Spatiotemporal Online Allocation with Deadline Constraints

We introduce and study spatiotemporal online allocation with deadline constraints ($\mathsf{SOAD}$), a new online problem motivated by emerging challenges in sustainability and energy. In $\mathsf{SOAD}$, an online player completes a workload by allocating and scheduling it on the points of a metric space $(X, d)$ while subject to a deadline $T$. At each time step, a service cost function is revealed that represents the cost of servicing the workload at each point, and the player must irrevocably decide the current allocation of work to points. Whenever the player moves this allocation, they incur a movement cost defined by the distance metric $d(\cdot, \ \cdot)$ that captures, e.g., an overhead cost. $\mathsf{SOAD}$ formalizes the open problem of combining general metrics and deadline constraints in the online algorithms literature, unifying problems such as metrical task systems and online search. We propose a competitive algorithm for $\mathsf{SOAD}$ along with a matching lower bound establishing its optimality. Our main algorithm, \textsc{ST-CLIP}, is a learning-augmented algorithm that takes advantage of predictions (e.g., forecasts of relevant costs) and achieves an optimal consistency-robustness trade-off. We evaluate our proposed algorithms in a simulated case study of carbon-aware spatiotemporal workload management, an application in sustainable computing that schedules a delay-tolerant batch compute job on a distributed network of data centers. In these experiments, we show that \textsc{ST-CLIP} substantially improves on heuristic baseline methods.

cs.DS↗

arXiv2025

Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning

Driven by inherent uncertainty and the sim-to-real gap, robust reinforcement learning (RL) seeks to improve resilience against the complexity and variability in agent-environment sequential interactions. Despite the existence of a large number of RL benchmarks, there is a lack of standardized benchmarks for robust RL. Current robust RL policies often focus on a specific type of uncertainty and are evaluated in distinct, one-off environments. In this work, we introduce Robust-Gymnasium, a unified modular benchmark designed for robust RL that supports a wide variety of disruptions across all key RL components-agents' observed state and reward, agents' actions, and the environment. Offering over sixty diverse task environments spanning control and robotics, safe RL, and multi-agent RL, it provides an open-source and user-friendly tool for the community to assess current methods and foster the development of robust RL algorithms. In addition, we benchmark existing standard and robust RL algorithms within this framework, uncovering significant deficiencies in each and offering new insights.

cs.LG↗

arXiv2025

Carbon- and Precedence-Aware Scheduling for Data Processing Clusters

As large-scale data processing workloads continue to grow, their carbon footprint raises concerns. Prior research on carbon-aware schedulers has focused on shifting computation to align with availability of low-carbon energy, but these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints (i.e., outputs of one task are inputs for another) that complicate decisions, since delaying an upstream ``bottleneck'' task to a low-carbon period will also block downstream tasks, impacting the entire job's completion time. In this paper, we show that carbon-aware scheduling for data processing benefits from knowledge of both time-varying carbon and precedence constraints. Our main contribution is $\texttt{PCAPS}$, a carbon-aware scheduler that interfaces with modern ML scheduling policies to explicitly consider the precedence-driven importance of each task in addition to carbon. To illustrate the gains due to fine-grained task information, we also study $\texttt{CAP}$, a wrapper for any carbon-agnostic scheduler that adapts the key provisioning ideas of $\texttt{PCAPS}$. Our schedulers enable a configurable priority between carbon reduction and job completion time, and we give analytical results characterizing the trade-off between the two. Furthermore, our Spark prototype on a 100-node Kubernetes cluster shows that a moderate configuration of $\texttt{PCAPS}$ reduces carbon footprint by up to 32.9% without significantly impacting the cluster's total efficiency.

cs.DC↗

arXiv2025

Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning

Standard multi-agent reinforcement learning (MARL) algorithms are vulnerable to sim-to-real gaps. To address this, distributionally robust Markov games (RMGs) have been proposed to enhance robustness in MARL by optimizing the worst-case performance when game dynamics shift within a prescribed uncertainty set. RMGs remains under-explored, from reasonable problem formulation to the development of sample-efficient algorithms. Two notorious and open challenges are the formulation of the uncertainty set and whether the corresponding RMGs can overcome the curse of multiagency, where the sample complexity scales exponentially with the number of agents. In this work, we propose a natural class of RMGs inspired by behavioral economics, where each agent's uncertainty set is shaped by both the environment and the integrated behavior of other agents. We first establish the well-posedness of this class of RMGs by proving the existence of game-theoretic solutions such as robust Nash equilibria and coarse correlated equilibria (CCE). Assuming access to a generative model, we then introduce a sample-efficient algorithm for learning the CCE whose sample complexity scales polynomially with all relevant parameters. To the best of our knowledge, this is the first algorithm to break the curse of multiagency for RMGs, regardless of the uncertainty set formulation.

cs.LG↗

arXiv2025

Optimal Decentralized Smoothed Online Convex Optimization

We study the multi-agent Smoothed Online Convex Optimization (SOCO) problem, where $N$ agents interact through a communication graph. In each round, each agent $i$ receives a strongly convex hitting cost function $f^i_t$ in an online fashion and selects an action $x^i_t \in \mathbb{R}^d$. The objective is to minimize the global cumulative cost, which includes the sum of individual hitting costs $f^i_t(x^i_t)$, a temporal "switching cost" for changing decisions, and a spatial "dissimilarity cost" that penalizes deviations in decisions among neighboring agents. We propose the first truly decentralized algorithm ACORD for multi-agent SOCO that provably exhibits asymptotic optimality. Our approach allows each agent to operate using only local information from its immediate neighbors in the graph. For finite-time performance, we establish that the optimality gap in the competitive ratio decreases with time horizon $T$ and can be conveniently tuned based on the per-round computation available to each agent. Our algorithm benefits from a provably scalable computational complexity that depends only logarithmically on the number of agents and almost linearly on their degree within the graph. Moreover, our results hold even when the communication graph changes arbitrarily and adaptively over time. Finally, ACORD, by virtue of its asymptotic-optimality, is shown to be provably superior to the state-of-the-art LPC algorithm, while exhibiting much lower computational complexity. Extensive numerical experiments across various network topologies further corroborate our theoretical claims.

math.OC↗

arXiv2025

Learning-Augmented Online Control for Decarbonizing Water Infrastructures

Water infrastructures are essential for drinking water supply, irrigation, fire protection, and other critical applications. However, water pumping systems, which are key to transporting water to the point of use, consume significant amounts of energy and emit millions of tons of greenhouse gases annually. With the wide deployment of digital water meters and sensors in these infrastructures, Machine Learning (ML) has the potential to optimize water supply control and reduce greenhouse gas emissions. Nevertheless, the inherent vulnerability of ML methods in terms of worst-case performance raises safety concerns when deployed in critical water infrastructures. To address this challenge, we propose a learning-augmented online control algorithm, termed LAOC, designed to dynamically schedule the activation and/or speed of water pumps. To ensure safety, we introduce a novel design of safe action sets for online control problems. By leveraging these safe action sets, LAOC can provably guarantee safety constraints while utilizing ML predictions to reduce energy and environmental costs. Our analysis reveals the tradeoff between safety requirements and average energy/environmental cost performance. Additionally, we conduct an experimental study on a building water supply system to demonstrate the empirical performance of LAOC. The results indicate that LAOC can effectively reduce environmental and energy costs while guaranteeing safety constraints.

eess.SY↗