Homeostasis Revisited and Reformulated Through Hidden Markov Model Control

Abstract

A common formalization of homeostasis is the free energy principle, a framework that defines a set of desired observation values, or critical states, that the agent should reach or remain close to. Under the free energy principle, an agent should act to maximize the probability of receiving the desired observations. Here we revisit the common approach of solving the problem of maximizing the log probability of the desired observations by maximizing a variational lower bound, the so-called negative free energy. We show that, instead, an approach directly maximizing that probability under the agent's policy is better suited to, and provides a better solution for, the original homeostatic control problem. This is done using hidden Markov model (HMM) control by allowing the policy to act over hidden states or noisy versions thereof while trying to maximize the probability of repeatedly having the desired observations. HMM control largely improves performance over the variational, or free energy, approach. We also show that the optimal policy is strictly deterministic, while the variational approach leads to a stochastic policy approximation. We finally provide a homeostatic reinterpretation of the maximum occupancy principle -a principle proposing that agents ought to maximize the occupancy of action-state path space -by defining homeostatic states as any states that do not immediately entail the termination or death of the agent.