arXiv subjects

Daniel Polani

Publications and source records attributed to Daniel Polani.

At least 19 recordsLinked to original sources

arXiv2026

Generalised Bellman recurrence and three dualities in sequential decision-making

What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.

cs.LG↗

arXiv2026

Life as Plasmas: Autonomy and Interactivism in-materio

When is a material system a candidate for life at all? We argue that this question is prior to behavior, functional architecture, or computational capacity, and that at root it is one of physical admissibility. We develop a framework in which minimal autonomy, taken in the interactivist sense of normativity grounded in self-maintaining far-from-equilibrium organization, corresponds to a distinct non-equilibrium phase of matter, and we take complex plasmas, a physical and non-biological system, as its in-materio exemplar. We formalize a diagnostic phase-space whose criteria (sustained free-energy throughput, organizational closure, active information maintenance, and regulated noise sensitivity) constitute necessary conditions for life-attribution. We instantiate the diagnostics across contrasting systems and fix the boundaries of the phase space via B\'enard convection as a driven baseline lacking closure, and a digital self-replicating soup that carries measured informational heredity while its physical closure remains a structural zero. We demonstrate that plasmas satisfy every admissibility condition for minimal physical autonomy while carrying none of the informational heredity that open-ended evolution requires, sharpening the distinction between physical admissibility and biological sufficiency, and bounding downstream questions of machine sentience.

cs.NE↗

arXiv2026

Active Inference: A method for Phenotyping Agency in AI systems?

The proliferation of agentic artificial intelligence has outpaced the conceptual tools needed to characterize agency in computational systems. Prevailing definitions mainly rely on autonomy and goal-directedness. Here, we argue for a minimal notion open to principled inspection given three criteria: intentionality as action grounded in beliefs and desires, rationality as normatively coherent action entailed by a world model, and explainability as action causally traceable to internal states; we subsequently instantiate these as a partially observable Markov decision process under a variational framework wherein posterior beliefs, prior preferences, and the minimization of expected free energy jointly constitute an agentic action chain. Using a canonical T-maze paradigm, we evidence how empowerment, formulated as the channel capacity between actions and anticipated observations, serves as an operational metric that distinguishes zero-, intermediate-, and high-agency phenotypes through structural manipulations of the generative model. We conclude by arguing that as agents engage in epistemic foraging to resolve ambiguity, the governance controls that remain effective must shift systematically from external constraints to the internal modulation of prior preferences, offering a principled, variational bridge from computational phenotyping to AI governance strategy

cs.AI↗

arXiv2026

Multi-Agent Empowerment and Emergence of Complex Behavior in Groups

Intrinsic motivations are receiving increasing attention, i.e. behavioral incentives that are not engineered, but emerge from the interaction of an agent with its surroundings. In this work we study the emergence of behaviors driven by one such incentive, empowerment, specifically in the context of more than one agent. We formulate a principled extension of empowerment to the multi-agent setting, and demonstrate its efficient calculation. We observe that this intrinsic motivation gives rise to characteristic modes of group-organization in two qualitatively distinct environments: a pair of agents coupled by a tendon, and a controllable Vicsek flock. This demonstrates the potential of intrinsic motivations such as empowerment to not just drive behavior for only individual agents but also higher levels of behavioral organization at scale.

cs.AI↗

arXiv2025

The Conditions of Physical Embodiment Enable Generalization and Care

As artificial agents enter open-ended physical environments -- eldercare, disaster response, and space missions -- they must persist under uncertainty while providing reliable care. Yet current systems struggle to generalize across distribution shifts and lack intrinsic motivation to preserve the well-being of others. Vulnerability and mortality are often seen as constraints to be avoided, yet organisms survive and provide care in an open-ended world with relative ease and efficiency. We argue that generalization and care arise from conditions of physical embodiment: being-in-the-world (the agent is a part of the environment) and being-towards-death (unless counteracted, the agent drifts toward terminal states). These conditions necessitate a homeostatic drive to maintain oneself and maximize the future capacity to continue doing so. Fulfilling this drive over long time horizons in multi-agent environments necessitates robust causal modeling of self and others' embodiment and jointly achievable future states. Because embodied agents are part of the environment, with the self delimited by reliable control, empowering others can expand self-boundaries, enabling other-regard. This provides a path from embodiment toward generalization and care based in shared constraints. We outline a reinforcement-learning framework for examining these questions. Homeostatic mortal agents continually learning in open-ended environments may offer efficient robustness and trustworthy alignment.

cs.AI↗

arXiv2025

Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights

Over the past decade, remarkable progress has been made in adopting deep neural networks to enhance the performance of conventional reinforcement learning. A notable milestone was the development of Deep Q-Networks (DQN), which achieved human-level performance across a range of Atari games, demonstrating the potential of deep learning to stabilise and scale reinforcement learning. Subsequently, extensions to continuous control algorithms paved the way for a new paradigm in control, one that has attracted broader attention than any classical control approach in recent literature. These developments also demonstrated strong potential for advancing data-driven, model-free algorithms for control and for achieving higher levels of autonomy. However, the application of these methods has remained largely confined to simulated and gaming environments, with ongoing efforts to extend them to real-world applications. Before such deployment can be realised, a solid and quantitative understanding of their performance on applied control problems is necessary. This paper conducts a comparative analysis of these approaches on four diverse benchmark problems with implementation results. This analysis offers a scrutinising and systematic evaluation to shed light on the real-world capabilities and limitations of deep reinforcement learning methods in applied control settings.

eess.SY↗

arXiv2025

Decentralized Traffic Flow Optimization Through Intrinsic Motivation

Traffic congestion has long been an ubiquitous problem that is exacerbating with the rapid growth of megacities. In this proof-of-concept work we study intrinsic motivation, implemented via the empowerment principle, to control autonomous car behavior to improve traffic flow. In standard models of traffic dynamics, self-organized traffic jams emerge spontaneously from the individual behavior of cars, affecting traffic over long distances. Our novel car behavior strategy improves traffic flow while still being decentralized and using only locally available information without explicit coordination. Decentralization is essential for various reasons, not least to be able to absorb robustly substantial levels of uncertainty. Our scenario is based on the well-established traffic dynamics model, the Nagel-Schreckenberg cellular automaton. In a fraction of the cars in this model, we substitute the default behavior by empowerment, our intrinsic motivation-based method. This proposed model significantly improves overall traffic flow, mitigates congestion, and reduces the average traffic jam time.

physics.soc-ph↗

arXiv2024

An Informational Parsimony Perspective on Probabilistic Symmetries

Extraction of structure, in particular of group symmetries, is increasingly crucial to understanding and building intelligent models. In particular, some information-theoretic models of parsimonious learning have been argued to induce invariance extraction. Here, we formalise these arguments from a group-theoretic perspective. We then extend them to the study of more general probabilistic symmetries, through compressions preserving geometric measures of complexity. More precisely, our framework implements a trade-off between compression and preservation of the divergence from a given hierarchical model, yielding a novel generalisation of the Information Bottleneck framework. Through appropriate choices of hierarchical models, we fully characterise (in the discrete and full support case) channel invariance, channel equivariance and distribution invariance under permutation. Allowing imperfect divergence preservation then leads to principled definitions of "soft symmetries", where the "coarseness" corresponds to the degree of compression of the system. In simple synthetic experiments, we demonstrate that our method successively recovers, at increasingly compressed "resolutions", nested but increasingly perturbed equivariances, where new equivariances emerge at bifurcation points of the trade-off parameter. Our framework suggests a new path for the extraction of generalised probabilistic symmetries.

cs.IT↗

arXiv2024

Normative Feeling: Socially Patterned Affective Mechanisms

Breaking a norm elicits both material and emotional consequences, yet how this coupling arose evolutionarily remains unclear. We investigate this question in light of emerging work suggesting that normativity's building blocks emerged earlier in evolution than previously considered, arguing that normative processes should inform accounts of how even ancient capacities such as mood evolved. Using a definition of normative processes we developed, we created an agent-based model with evolvable affect in a shared resource dilemma, comparing competition (non-normative) versus punishment (normative) conditions. Critically, different mood mechanisms emerge under each condition. Under competition, agents evolve a "bad mood -> consume more" response, creating a tragedy of the commons leading to resource depletion and population collapse. Under punishment, agents evolve a "bad mood -> consume less" mechanism, where negative affect functions as an implicit signal of social sanction, promoting resource conservation. Importantly, once normative logic is imprinted through punishment, it creates an evolutionary pathway for mood-based signalling that operates without costly physical enforcement. Our findings demonstrate how normative processes enable social preferences to emerge in a distributed manner within psychological mechanisms, showing how normative processes reprogram cognitive and physiological systems by embedding cultural patterns into psychological dispositions.

cs.MA↗

arXiv2024

SuPLE: Robot Learning with Lyapunov Rewards

The reward function is an essential component in robot learning. Reward directly affects the sample and computational complexity of learning, and the quality of a solution. The design of informative rewards requires domain knowledge, which is not always available. We use the properties of the dynamics to produce system-appropriate reward without adding external assumptions. Specifically, we explore an approach to utilize the Lyapunov exponents of the system dynamics to generate a system-immanent reward. We demonstrate that the `Sum of the Positive Lyapunov Exponents' (SuPLE) is a strong candidate for the design of such a reward. We develop a computational framework for the derivation of this reward, and demonstrate its effectiveness on classical benchmarks for sample-based stabilization of various dynamical systems. It eliminates the need to start the training trajectories at arbitrary states, also known as auxiliary exploration. While the latter is a common practice in simulated robot learning, it is unpractical to consider to use it in real robotic systems, since they typically start from natural rest states such as a pendulum at the bottom, a robot on the ground, etc. and can not be easily initialized at arbitrary states. Comparing the performance of SuPLE to commonly-used reward functions, we observe that the latter fail to find a solution without auxiliary exploration, even for the task of swinging up the double pendulum and keeping it stable at the upright position, a prototypical scenario for multi-linked robots. SuPLE-induced rewards for robot learning offer a novel route for effective robot learning in typical as opposed to highly specialized or fine-tuned scenarios. Our code is publicly available for reproducibility and further research.

cs.RO↗

arXiv2024

Mapping the Mind-Brain Duality to a Digital-Analog Perceptual Duality

Could the abstract ideas of our minds originate from neuronal interactions within our brains? To address this question, we examine interactions within 'brain-world' (BW) and 'brain-brain' (BB) domains, which represent the brain's physical interactions with its environment and the mental interactions between brains, respectively. BW interactions are characterized as analog - dynamic and continuous, whereas BB interactions are digital - non-dynamic and discrete. This distinction allows BB interactions to facilitate effective, albeit information-limited, communication through categorization. We review existing data showing that cascades of neural circuits can convert between analog and digital signals, thereby linking physical and mental processes. Importantly, we argue that these circuits cannot fully reduce one domain to the other, suggesting that the mind-brain duality can be mapped to the BB-BW duality. Such mapping suggests that the mind's foundation is inherently social, offering a novel explanation for the physical-mental gap while acknowledging the coexistence of the physical body and the non-physical mind.

q-bio.NC↗

arXiv2024

Software in the natural world: A computational approach to hierarchical emergence

Understanding the functional architecture of complex systems is crucial to illuminate their inner workings and enable effective methods for their prediction and control. Recent advances have introduced tools to characterise emergent macroscopic levels; however, while these approaches are successful in identifying when emergence takes place, they are limited in the extent they can determine how it does. Here we address this limitation by developing a computational approach to emergence, which characterises macroscopic processes in terms of their computational capabilities. Concretely, we articulate a view on emergence based on how software works, which is rooted on a mathematical formalism that articulates how macroscopic processes can express self-contained informational, interventional, and computational properties. This framework establishes a hierarchy of nested self-contained processes that determines what computations take place at what level, which in turn delineates the functional architecture of a complex system. This approach is illustrated on paradigmatic models from the statistical physics and computational neuroscience literature, which are shown to exhibit macroscopic processes that are akin to software in human-engineered systems. Overall, this framework enables a deeper understanding of the multi-level structure of complex systems, revealing specific ways in which they can be efficiently simulated, predicted, and controlled.

nlin.AO↗

arXiv2023

Dimensionality Reduction of Dynamics on Lie Manifolds via Structure-Aware Canonical Correlation Analysis

Incorporating prior knowledge into a data-driven modeling problem can drastically improve performance, reliability, and generalization outside of the training sample. The stronger the structural properties, the more effective these improvements become. Manifolds are a powerful nonlinear generalization of Euclidean space for modeling finite dimensions. Structural impositions in constrained systems increase when applying group structure, converting them into Lie manifolds. The range of their applications is very wide and includes the important case of robotic tasks. Canonical Correlation Analysis (CCA) can construct a hierarchical sequence of maximal correlations of up to two paired data sets in these Euclidean spaces. We present a method to generalize this concept to Lie Manifolds and demonstrate its efficacy through the substantial improvements it achieves in making structure-consistent predictions about changes in the state of a robotic hand.

cs.RO↗

arXiv2023

Towards Information Theory-Based Discovery of Equivariances

The presence of symmetries imposes a stringent set of constraints on a system. This constrained structure allows intelligent agents interacting with such a system to drastically improve the efficiency of learning and generalization, through the internalisation of the system's symmetries into their information-processing. In parallel, principled models of complexity-constrained learning and behaviour make increasing use of information-theoretic methods. Here, we wish to marry these two perspectives and understand whether and in which form the information-theoretic lens can "see" the effect of symmetries of a system. For this purpose, we propose a novel variant of the Information Bottleneck principle, which has served as a productive basis for many principled studies of learning and information-constrained adaptive behaviour. We show (in the discrete case and under a specific technical assumption) that our approach formalises a certain duality between symmetry and information parsimony: namely, channel equivariances can be characterised by the optimal mutual information-preserving joint compression of the channel's input and output. This information-theoretic treatment furthermore suggests a principled notion of "soft" equivariance, whose "coarseness" is measured by the amount of input-output mutual information preserved by the corresponding optimal compression. This new notion offers a bridge between the field of bounded rationality and the study of symmetries in neural representations. The framework may also allow (exact and soft) equivariances to be automatically discovered.

cs.IT↗

arXiv2023

The Effect of Noise on the Emergence of Continuous Norms and its Evolutionary Dynamics

We examine the effect of noise on societies of agents using an agent-based model of evolutionary norm emergence. Generally, we see that noisy societies are more selfish, smaller and discontent, and are caught in rounds of perpetual punishment preventing them from flourishing. Surprisingly, despite the effect of noise on the population, it does not seem to evolve away. We carry out further analysis and provide reasons for why this may be the case. Furthermore, we claim that our framework that evolves the noise/ambiguity of norms may be a new way to model the tight/loose framework of norms, suggesting that despite ambiguous norms detrimental effect on society, evolution does not favour clarity.

cs.MA↗

arXiv2023

Spatial, Social and Data Gaps in On-Demand Mobility Services: Towards a Supply-Oriented MaaS

After a decade of on-demand mobility services that change spatial behaviors in metropolitan areas, the Shared Autonomous Vehicle (SAV) service is expected to increase traffic congestion and unequal access to transport services. A paradigm of scheduled supply that is aware of demand but not on-demand is proposed, introducing coordination and social and behavioral understanding, urban cognition and empowerment of agents, into a novel informational framework. Daily routines and other patterns of spatial behaviors outline a fundamental demand layer in a supply-oriented paradigm that captures urban dynamics and spatial-temporal behaviors, mostly in groups. Rather than real-time requests and instant responses that reward unplanned actions, and beyond just reservation of travels in timetables, the intention is to capture mobility flows in scheduled travels along the day considering time of day, places, passengers etc. Regulating goal-directed behaviors and caring for service resources and the overall system welfare is proposed to minimize uncertainty, considering the capacity of mobility interactions to hold value, i.e., Motility as a Service (MaaS). The principal-agent problem in the smart city is a problem of collective action among service providers and users that create expectations based on previous actions and reactions in mutual systems. Planned behavior that accounts for service coordination is expected to stabilize excessive rides and traffic load, and to induce a cognitive gain, thus balancing information load and facilitating cognitive effort.

cs.CY↗

arXiv2022

Intrinsic Motivation in Dynamical Control Systems

Biological systems often choose actions without an explicit reward signal, a phenomenon known as intrinsic motivation. The computational principles underlying this behavior remain poorly understood. In this study, we investigate an information-theoretic approach to intrinsic motivation, based on maximizing an agent's empowerment (the mutual information between its past actions and future states). We show that this approach generalizes previous attempts to formalize intrinsic motivation, and we provide a computationally efficient algorithm for computing the necessary quantities. We test our approach on several benchmark control problems, and we explain its success in guiding intrinsically motivated behaviors by relating our information-theoretic control function to fundamental properties of the dynamical system representing the combined agent-environment system. This opens the door for designing practical artificial, intrinsically motivated controllers and for linking animal behaviors to their dynamical properties.

cs.AI↗

arXiv2021

Skilled motor control implies a low entropy of states but a high entropy of actions

The mastery of skills such as playing tennis or balancing an inverted pendulum implies a very accurate control of movements to achieve the task goals. Traditional accounts of skilled action control that focus on either routinization or perceptual control make opposite predictions about the ways we achieve mastery. The notion of routinization emphasizes the decrease of the variance of our actions, whereas the notion of perceptual control emphasizes the decrease of the variance of the states we visit, but not of the actions we execute. Here, we studied how participants managed control tasks of varying levels of complexity, which consisted in controlling inverted pendulums of different lengths. We used information-theoretic measures to compare the predictions of alternative theoretic accounts that focus on routinization and perceptual control, respectively. Our results indicate that the successful performance of the control task strongly correlates with the decrease of state variability and the increase of action variability. As postulated by perceptual control theory, the mastery of skills consists in achieving stable control goals by flexible means.

q-bio.NC↗