arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2309.06921

Investigating the Impact of Action Representations in Policy Gradient Algorithms

Jan Schneider·Pierre Schumacher·Daniel Häufle·Bernhard Schölkopf·Dieter Büchler
View original source ↗Read source document ↗Export metadata ↓

Abstract

Reinforcement learning~(RL) is a versatile framework for learning to solve complex real-world tasks. However, influences on the learning performance of RL algorithms are often poorly understood in practice. We discuss different analysis techniques and assess their effectiveness for investigating the impact of action representations in RL. Our experiments demonstrate that the action representation can significantly influence the learning performance on popular RL benchmark tasks. The analysis results indicate that some of the performance differences can be attributed to changes in the complexity of the optimization landscape. Finally, we discuss open challenges of analysis techniques for RL algorithms.

Explore related subjects

cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jan Schneider, Pierre Schumacher, Daniel Häufle, Bernhard Schölkopf, Dieter Büchler. 2023-09-13. Investigating the Impact of Action Representations in Policy Gradient Algorithms. https://arxiv.org/abs/2309.06921

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

When fairness metrics fail: A utility-based perspective on $\varepsilon$-fairness

Fairness in decision-making processes is often quantified using probabilistic metrics. However, these metrics need not reflect the consequences of decisions for the affected individuals and groups. We develop a utility-based framework that incorporates these consequences into the assessment of fairness. Our main result shows that a decision-making process can satisfy $\varepsilon$-fairness while nevertheless being maximally unfair once the utilities associated with its outcomes are taken into account. To address applications in which information on false negatives is unavailable, we also formulate a reduced setting that retains the essential elements of the utility-based fairness assessment. We illustrate the framework through two applications: college admissions and credit-risk assessment. In both cases, probabilistic metrics may classify a decision-making process as approximately fair even though the corresponding utility outcomes are highly unequal. In the college-admissions example, our analysis shows that improving completion rates is necessary to achieve equality of utility across groups, while in the mortgage example, mitigating unfairness requires not only adjusting approval rates but also reducing the adverse consequences of default. These findings demonstrate that fairness assessments should account not only for the probabilities of different decisions but also for the consequences of those decisions.

cs.LG↗
arXiv2026

A Computational Tropical Geometry Framework for Neural Networks

We propose a computational tropical geometry framework for the symbolic analysis of neural networks with tropical activations. The number of linear regions of a neural network has been actively studied as a measure of the expressivity of a given architecture. To study these, we work in the setting of tropical geometry---a combinatorial and polyhedral variant of algebraic geometry---where there are known connections between tropical rational maps and feedforward neural networks. We expand this connection by developing concrete computational tools for studying the linear regions of neural networks. We present an algorithm, together with a proof of correctness, which computes the linear regions of a neural network as explicit unions of polyhedra. We further relate the computation of the number of linear regions of a tropical expression to the number of monomials that appear in it, and show how tropical expressions can often be pruned to remove redundant monomials. We introduce the Hoffman constant of a neural network's tropical expression, a geometric quantity that controls the distance from any point in the input space to the farthest linear region. We provide the open source Julia library TropicalNN.jl, which is built on top of the OSCAR computer algebra system and implements the algorithms mentioned above to analyze neural networks symbolically using their tropical representations. We present a set of proof-of-concept computational examples to demonstrate how our tropical geometric theory can be applied to reveal insights on the expressivity of a network architecture.

cs.LG↗
arXiv2026

Information-Geometric Inverse Distillation for Enhancing Adversarial Transferability

Transfer-based adversarial attacks rely on surrogate models to craft perturbations, yet often overfit the surrogate's decision boundary. To address this problem, we propose Inverse Knowledge Distillation (IKD), a simple and attack-agnostic mechanism that maximizes the prediction-distribution discrepancy between benign and adversarial samples on the surrogate model. IKD uses a CE/KL-equivalent soft-label objective to push adversarial predictions away from a fixed benign prediction anchor and enrich the attack with Fisher-sensitive surrogate directions. We prove that, under a matched fixed-anchor implementation, soft-label cross-entropy and KL divergence differ only by a constant entropy term and therefore induce identical gradients, Hessians, and adversarial optimization trajectories. Our information-geometric analysis further derives a quantitative lower bound on dominant Fisher-subspace overlap between surrogate and target models from local same-task stability and a Fisher eigengap, and establishes a sufficient target-margin crossing condition under oriented gradient coherence and target smoothness. This analysis connects IKD's surrogate Fisher sensitivity to cross-model transfer. In contrast, mean squared error uses a different Euclidean pullback in output probability space. IKD integrates seamlessly with standard gradient-based attacks without modifying their optimization pipelines. Extensive ImageNet experiments demonstrate consistent black-box gains across CNN, ViT, and defended models, while ablations confirm CE and KL equivalence and the pronounced disadvantage of MSE. These results establish IKD as an effective and lightweight component for improving adversarial transferability. Code is available at https://github.com/ImmortalTing/IKD.

cs.LG↗