Policy
A policy defines how the agent behaves: it maps the current state to an action, either deterministically or by sampling
from a probability distribution over the action space. It is invoked by the agent through
draw_action(), and it is the object that most learning algorithms optimize.
For greedy evaluation (see evaluate()), a policy also exposes
draw_action_greedy(), returning the mode of the policy (e.g. the mean of a Gaussian,
the argmax of an \(\varepsilon\)-greedy or Boltzmann policy). The base implementation raises, so a policy supports
greedy evaluation only if it overrides it. For Torch policies the greedy action comes from the underlying distribution
(a Gaussian returns its mean, a categorical its mode/argmax, a squashed Gaussian its median, since its true
mode is ill-behaved and can pile up at the action bounds).
Two mixins add optional capabilities that can be combined with a policy: HasWeights equips
it with a set of trainable weights (used by policy-search and black-box optimization algorithms), while
HasGradient additionally provides the gradient of the log-probability required by
policy-gradient methods.
MushroomRL provides several families of policies:
Deterministic policies return a single action for each state;
Gaussian policies are differentiable parametric policies that sample from a Gaussian distribution;
TD policies are value-based policies that select the action from a Q-function (e.g. epsilon-greedy or Boltzmann);
Torch policies are implemented as neural networks and support tensor computation for deep RL;
Movement primitives are trajectory generators implementing DMPs and ProMPs;
Vector policies wrap a population of policies for vectorized black-box optimization.
Policies in MushroomRL can depend on the past in two orthogonal ways. A stateful policy
(StatefulPolicy) carries a latent internal state, updated at every step and stored in the
dataset because it cannot be reconstructed (e.g. a recurrent hidden state or Ornstein-Uhlenbeck noise). The context
is instead a deterministic function of the observed trajectory (e.g. a window of stacked observations); being
reconstructable from the stored transitions, it is assembled on the fly by the
HistoryManager rather than stored as policy state.
Interface representing a generic policy. |
|
Interface representing a stateful policy, i.e. a policy carrying a latent internal state updated at every step. |
|
Mixin adding a set of trainable parameters (the policy weights) to a policy. |
|
Mixin for a parametric policy that is also differentiable. |
|
Epsilon greedy policy. |
|
Boltzmann softmax policy. |
|
Mellowmax policy. |
|
Gaussian policy. |
|
Simple parametric policy representing a deterministic policy. |
|
Exploration policy adding temporally correlated noise drawn from an Ornstein-Uhlenbeck process. |
|
Gaussian policy whose sampled action is clipped to a given action range. |
|
Interface for a generic PyTorch policy. |
|
Interface for a stateful PyTorch policy, i.e. a |
|
Policy wrapping a vector of independent copies of a base policy, each one with its own weights. |
|
Class representing a Probabilistic Movement Primitive (ProMP). |
|
Class representing a Dynamic Movement Primitive (DMP). |