Reinforcement Learning utils

Eligibility trace

EligibilityTrace(shape, name='replacing')

Factory method to create an eligibility trace of the provided type.

Parameters:

• shape (list) – shape of the eligibility trace table;

• name (str, 'replacing') – type of the eligibility trace.

Returns:

The eligibility trace table of the provided shape and type.

class ReplacingTrace(*args, n_models=1, **kwargs)

Bases: Table

Replacing trace.

reset()

update(state, action)

class AccumulatingTrace(*args, n_models=1, **kwargs)

Bases: Table

Accumulating trace.

reset()

update(state, action)

Optimizers

class Optimizer(*args, **kwargs)

Bases: MushroomObject

Base class for gradient optimizers. These objects take the current parameters and the gradient estimate to compute the new parameters.

__init__(lr=0.001, maximize=True, *params)

Constructor

Parameters:

• lr ([float, Parameter]) – the learning rate;

• maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.

__call__(*args, **kwargs)

Call self as a function.

class AdaptiveOptimizer(*args, **kwargs)

Bases: Optimizer

This class implements an adaptive gradient step optimizer. Instead of moving of a step proportional to the gradient, takes a step limited by a given metric M. To specify the metric, the natural gradient has to be provided. If natural gradient is not provided, the identity matrix is used.

The step rule is:

\[ \begin{align}\begin{aligned}\Delta\theta=\underset{\Delta\vartheta}{argmax}\Delta\vartheta^{t}\nabla_{\theta}J\\s.t.:\Delta\vartheta^{T}M\Delta\vartheta\leq\varepsilon\end{aligned}\end{align} \]

Lecture notes, Neumann G. http://www.ias.informatik.tu-darmstadt.de/uploads/Geri/lecture-notes-constraint.pdf

__init__(eps, maximize=True)

Constructor.

Parameters:

• eps (float) – the maximum step defined by the metric;

• maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.

__call__(params, *args, **kwargs)

Call self as a function.

get_value(*args, **kwargs)

class SGDOptimizer(*args, **kwargs)

Bases: Optimizer

This class implements the SGD optimizer.

__init__(lr=0.001, maximize=True)

Constructor.

Parameters:

• lr ([float, Parameter], 0.001) – the learning rate;

• maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.

__call__(params, grads)

Call self as a function.

class AdamOptimizer(*args, **kwargs)

Bases: Optimizer

This class implements the Adam optimizer.

__init__(lr=0.001, beta1=0.9, beta2=0.999, eps=1e-07, maximize=True)

Constructor.

Parameters:

• lr ([float, Parameter], 0.001) – the learning rate;

• beta1 (float, 0.9) – Adam beta1 parameter;

• beta2 (float, 0.999) – Adam beta2 parameter;

• maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.

__call__(params, grads)

Call self as a function.

Parameters

class Parameter(*args, **kwargs)

Bases: MushroomObject

This class implements function to manage parameters, such as learning rate. It also allows to have a single parameter for each state of state-action tuple.

__init__(value, min_value=None, max_value=None, size=(1,), log_table=False)

Constructor.

Parameters:

• value (float) – initial value of the parameter;

• min_value (float, None) – minimum value that the parameter can reach when decreasing;

• max_value (float, None) – maximum value that the parameter can reach when increasing;

• size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.

• log_table (bool, False) – if True, the parameter is logged also when it is backed by a table with more than one element. By default tabular parameters are not logged, as logging a per-state or per-state-action value on every update is too expensive.

__call__(*idx, **kwargs)

Update and return the parameter in the provided index.

Parameters:

*idx (list) – index of the parameter to return.

Returns:

The updated parameter in the provided index.

get_value(*idx, **kwargs)

Return the current value of the parameter in the provided index.

Parameters:

*idx (list) – index of the parameter to return.

Returns:

The current value of the parameter in the provided index.

_compute(*idx, **kwargs)

Returns:

The value of the parameter in the provided index.

update(*idx, **kwargs)

Updates the number of visit of the parameter in the provided index.

Parameters:

*idx (list) – index of the parameter whose number of visits has to be updated.

property shape

Returns: The shape of the table of parameters.

property initial_value

Returns: The initial value of the parameters.

class LinearParameter(*args, **kwargs)

Bases: Parameter

This class implements a linearly changing parameter according to the number of times it has been used. The parameter changes following the formula:

\[v_n = \textrm{clip}(v_0 + \dfrac{v_{th} - v_0}{n}, v_{th})\]

where \(v_0\) is the initial value of the parameter, \(n\) is the number of steps and \(v_{th}\) is the upper or lower threshold for the parameter.

__init__(value, threshold_value, n, size=(1,), log_table=False)

Constructor.

Parameters:

• value (float) – initial value of the parameter;

• threshold_value (float, None) – minimum or maximum value that the parameter can reach;

• n (int) – number of time steps needed to reach the threshold value;

• size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.

• log_table (bool, False) – if True, the parameter is logged also when it is backed by a table with more than one element.

_compute(*idx, **kwargs)

Returns:

The value of the parameter in the provided index.

class DecayParameter(*args, **kwargs)

Bases: Parameter

This class implements a decaying parameter. The decay follows the formula:

\[v_n = \dfrac{v_0}{n^p}\]

where \(v_0\) is the initial value of the parameter, \(n\) is the number of steps and \(p\) is an arbitrary exponent.

__init__(value, exp=1.0, min_value=None, max_value=None, size=(1,), log_table=False)

Constructor.

Parameters:

• value (float) – initial value of the parameter;

• exp (float, 1.) – exponent for the step decay;

• min_value (float, None) – minimum value that the parameter can reach when decreasing;

• max_value (float, None) – maximum value that the parameter can reach when increasing;

• size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.

• log_table (bool, False) – if True, the parameter is logged also when it is backed by a table with more than one element.

_compute(*idx, **kwargs)

Returns:

The value of the parameter in the provided index.

Preprocessors

class Preprocessor(*args, **kwargs)

Bases: MushroomObject

Abstract preprocessor class.

__call__(obs)

Preprocess the observations.

Parameters:

obs (Array) – observations to be preprocessed.

Returns:

Preprocessed observations.

update(obs)

Update internal state of the preprocessor using the current observations.

Parameters:

obs (Array) – observations to be preprocessed.

class StandardizationPreprocessor(*args, **kwargs)

Bases: Preprocessor

Preprocess observations from the environment using a running standardization.

__init__(mdp_info, clip_obs=10.0, alpha=1e-32)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information of the MDP;

• clip_obs (float, 10.) – values to clip the normalized observations;

• alpha (float, 1e-32) – moving average catchup parameter for the normalization.

__call__(obs)

Preprocess the observations.

Parameters:

obs (Array) – observations to be preprocessed.

Returns:

Preprocessed observations.

update(obs)

Update internal state of the preprocessor using the current observations.

Parameters:

obs (Array) – observations to be preprocessed.

class MinMaxPreprocessor(*args, **kwargs)

Bases: StandardizationPreprocessor

Preprocess observations from the environment using the bounds of the observation space of the environment. For observations that are not limited falls back to using running mean standardization.

__init__(mdp_info, clip_obs=10.0, alpha=1e-32)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information of the MDP;

• clip_obs (float, 10.) – values to clip the normalized observations;

• alpha (float, 1e-32) – moving average catchup parameter for the normalization.

__call__(obs)

Preprocess the observations.

Parameters:

obs (Array) – observations to be preprocessed.

Returns:

Preprocessed observations.

Replay memory

class ReplayMemory(*args, **kwargs)

Bases: MushroomObject

This class implements function to manage a replay memory as the one used in “Human-Level Control Through Deep Reinforcement Learning” by Mnih V. et al..

__init__(mdp_info, agent_info, initial_size, max_size, history_manager=None, n_steps_return=1, store_policy_state=False, return_extra=False)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• initial_size (int) – initial size of the replay buffer;

• max_size (int) – maximum size of the replay buffer;

• history_manager (HistoryManager, None) – the manager used by the agent to assemble the stacked observation, reused offline so that the stacked observation matches the one built online;

• n_steps_return (int, 1) – number of steps used for the n-step return;

• store_policy_state (bool, False) – whether the policy internal state is stored in the replay memory. When False, no policy-state buffer is allocated and the policy state of the added datasets is dropped; a stateless algorithm should leave it False even if its policy is stateful;

• return_extra (bool, False) – whether get() appends, as a trailing element, the extra_data dictionary of the history windows not delivered in-band in the state, keyed as the online policy keyword arguments. When False these streams are not returned.

add(dataset)

Add elements to the replay memory.

Parameters:

dataset (Dataset) – dataset class elements to add to the replay memory.

get(n_samples)

Returns the provided number of states from the replay memory.

Parameters:

n_samples (int) – the number of samples to return.

Returns:

The requested number of samples.

reset()

Reset the replay memory.

property size

Returns: The number of elements contained in the replay memory.

property initialized

Returns: Whether the replay memory has reached the number of elements that allows it to be used.

_assemble_batch(idxs)

Read the transitions at the given buffer indices and assemble the batch. When a history is used the stacked observation windows are rebuilt from the buffer. The policy states are appended when stored, followed by the extra_data dictionary of the out-of-band history windows when return_extra is set.

Parameters:

idxs – the buffer indices of the transitions to read.

Returns:

The list of arrays forming the sampled batch.

_sample_idxs(n_samples)

Sample buffer indices to read, drawing uniformly among the anchors that can be sampled, i.e. those whose stacked observation window can be rebuilt and whose n-step return can be completed (see _compute_mask()).

Parameters:

n_samples (int) – the number of indices to sample.

Returns:

The sampled buffer indices.

_affected_window(positions)

The buffer positions whose sampling mask can change after a batch was written at positions: the newly written anchors, their forward n-step window (the n-1 anchors ending in the new batch) and the backward history reserve that trails the moved write head. Every other entry keeps its mask.

Parameters:

positions – the buffer positions where the last batch was written.

Returns:

The affected buffer positions, or None when no masking is in use.

_compute_mask(anchor_idxs)

Compute the sampling mask for a batch of anchors: True where the anchor cannot be sampled because its n-step window would cross a truncation or the write head, or because its backward history window would cross the write head of a full buffer.

Parameters:

anchor_idxs – buffer positions of the anchors to evaluate.

Returns:

The boolean mask (True = excluded from sampling) for the given anchors.

_write_to_buffer(dataset)

Write transitions from a dataset into the circular buffer.

Uses append_batch while the buffer still has capacity, then switches to direct slice assignment once the buffer is full, wrapping around as needed.

Parameters:

dataset (Dataset) – transitions to write.

Returns:

The buffer positions (indices into the circular buffer) where the transitions were written.

class SequenceReplayMemory(*args, **kwargs)

Bases: ReplayMemory

This class extend the base replay memory to allow sampling sequences of a certain length. This is useful for training recurrent agents or agents operating on a window of states etc.

The temporal length of the sampled sequences (truncation_length) and the history-stacking length of each timestep (history_length, carried by the injected HistoryManager) are orthogonal and compose: with a history_length greater than 1 each timestep of the sequence is itself a stacked window, so the sampled states have shape (n_samples, truncation_length, history_length, *obs_shape), collapsing to (n_samples, truncation_length, *obs_shape) when no stacking is used.

__init__(mdp_info, agent_info, initial_size, max_size, truncation_length, history_manager=None, return_extra=False)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• initial_size (int) – initial size of the replay buffer;

• max_size (int) – maximum size of the replay buffer;

• truncation_length (int) – truncation length to be sampled;

• history_manager (HistoryManager, None) – the manager used by the agent to assemble the stacked observation online, reused to rebuild the same stacked observation for every timestep of a sequence;

• return_extra (bool, False) – whether get() appends, as a trailing element, the extra_data dictionary of the history windows not delivered in-band in the state, keyed as the online policy keyword arguments. When False these streams are not returned.

get(n_samples)

Returns the provided number of states from the replay memory.

Parameters:

n_samples (int) – the number of samples to return.

Returns:

The requested number of samples. When return_extra is set, the extra_data dictionary assembled by the history manager, padded to the truncation length, is appended as the trailing element.

class PrioritizedReplayMemory(*args, **kwargs)

Bases: ReplayMemory

This class implements function to manage a prioritized replay memory as the one used in “Prioritized Experience Replay” by Schaul et al., 2015.

__init__(mdp_info, agent_info, initial_size, max_size, alpha, beta, epsilon=0.01, history_manager=None, n_steps_return=1, store_policy_state=False, return_extra=False)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• initial_size (int) – initial number of elements in the replay memory;

• max_size (int) – maximum number of elements that the replay memory can contain;

• alpha (float) – prioritization coefficient;

• beta ([float, Parameter]) – importance sampling coefficient;

• epsilon (float, .01) – small value to avoid zero probabilities;

• history_manager (HistoryManager, None) – the manager used by the agent to assemble the stacked observation, reused offline so that the stacked observation matches the one built online;

• n_steps_return (int, 1) – number of steps used for the n-step return;

• store_policy_state (bool, False) – whether the policy internal state is stored in the replay memory;

• return_extra (bool, False) – whether get() appends the extra_data dictionary of the history windows not delivered in-band in the state, inserted just before the idxs/is_weight pair.

add(dataset, p=None)

Add elements to the replay memory.

Parameters:

• dataset (Dataset) – dataset whose transitions will be added to the replay memory;

• p (Array, None) – priority of each sample in the dataset. When None, each new transition is inserted with the current maximum priority (max_priority), so that it is sampled at least once before its priority is corrected from its temporal-difference error.

get(n_samples)

Returns the provided number of states from the replay memory.

Parameters:

n_samples (int) – the number of samples to return.

Returns:

The requested number of samples, followed by the tree indices and importance-sampling weights of the drawn transitions. When return_extra is set, the extra_data dictionary assembled by the history manager is inserted just before this trailing pair.

reset()

Reset the replay memory.

update(error, idx)

Update the priority of the sample at the provided index in the dataset.

Parameters:

• error (Array) – errors to consider to compute the priorities;

• idx (Array) – indexes of the transitions in the dataset.

property max_priority

Returns: The maximum value of priority inside the replay memory.

_sync_tree_mask(window)

Mirror the sampling mask of window (see _compute_mask()) into the sum tree: mask the leaves that are excluded and unmask the ones that are sampleable. Both tree operations are idempotent and the leaf keeps its true priority while masked, so this is a plain copy of the mask over the affected leaves.

Parameters:

window – the buffer positions whose mask changed (see _affected_window()), or None when no masking is in use.

Running Statistics

class RunningStandardization(*args, **kwargs)

Bases: MushroomObject

Compute a running standardization of values according to Welford’s online algorithm: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford’s_online_algorithm

__init__(shape, backend, alpha=1e-32)

Constructor.

Parameters:

• shape (tuple) – shape of the data to standardize;

• backend (str) – name of the backend to be used;

• alpha (float, 1e-32) – minimum learning rate.

reset()

Reset the mean and standard deviation.

update_stats(value)

Update the statistics with the current data value.

Parameters:

value (Array) – current data value to use for the update.

property mean

Returns: The estimated mean value.

property std

Returns: The estimated standard deviation value.

class RunningExpWeightedAverage(*args, **kwargs)

Bases: MushroomObject

Compute an exponentially weighted moving average.

__init__(shape, alpha, backend, init_value=None)

Constructor.

Parameters:

• shape (tuple) – shape of the data to standardize;

• alpha (float) – learning rate;

• backend (str) – name of the backend to be used;

• init_value (np.ndarray) – initial value of the filter.

reset(init_value=None)

Reset the mean and standard deviation.

Parameters:

init_value (Array) – initial value of the filter.

update_stats(value)

Update the statistics with the current data value.

Parameters:

value (Array) – current data value to use for the update.

property mean

Returns: The estimated mean value.

class RunningAveragedWindow(*args, **kwargs)

Bases: MushroomObject

Compute the running average using a window of fixed size.

__init__(shape, window_size, backend, init_value=None)

Constructor.

Parameters:

• shape (tuple) – shape of the data to standardize;

• window_size (int) – size of the windows;

• backend (str) – name of the backend to be used;

• init_value (np.ndarray) – initial value of the filter.

reset(init_value=None)

Reset the window.

Parameters:

init_value (np.ndarray) – initial value of the filter.

update_stats(value)

Update the statistics with the current data value.

Parameters:

value (np.ndarray) – current data value to use for the update.

property mean

Returns: The estimated mean value.

Spaces

class Box(*args, **kwargs)

Bases: MushroomObject

This class implements functions to manage continuous states and action spaces. It is similar to the Box class in gym.spaces.box.

__init__(low, high, shape=None, data_type=<class 'float'>)

Constructor.

Parameters:

• low ([float, np.ndarray]) – the minimum value of each dimension of the space. If a scalar value is provided, this value is considered as the minimum one for each dimension. If a np.ndarray is provided, each i-th element is considered the minimum value of the i-th dimension;

• high ([float, np.ndarray]) – the maximum value of dimensions of the space. If a scalar value is provided, this value is considered as the maximum one for each dimension. If a np.ndarray is provided, each i-th element is considered the maximum value of the i-th dimension;

• shape (np.ndarray, None) – the dimension of the space. Must match the shape of low and high, if they are np.ndarray.

• data_type (class, float) – the data type to be used.

property low

Returns: The minimum value of each dimension of the space.

property high

Returns: The maximum value of each dimension of the space.

property shape

Returns: The dimensions of the space.

class Discrete(*args, **kwargs)

Bases: MushroomObject

This class implements functions to manage discrete states and action spaces. It is similar to the Discrete class in gym.spaces.discrete.

__init__(n)

Constructor.

Parameters:

n (int) – the number of values of the space.

property size

Returns: The number of elements of the space.

property shape

Returns: The shape of the space that is always (1,).

Value Functions

compute_advantage_montecarlo(V, s, ss, r, absorbing, last, gamma)

Function to estimate the advantage and new value function target over a dataset. The value function is estimated using rollouts (Monte Carlo estimation).

Parameters:

• V (Regressor) – the current value function regressor;

• s (torch.tensor) – the set of states in which we want to evaluate the advantage;

• ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;

• r (torch.tensor) – the reward obtained in each transition from state s to state ss;

• absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;

• gamma (float) – the discount factor of the considered problem.

Returns:

The new estimate for the value function of the next state and the advantage function.

compute_advantage(V, s, ss, r, absorbing, gamma)

Function to estimate the advantage and new value function target over a dataset. The value function is estimated using bootstrapping.

Parameters:

• V (Regressor) – the current value function regressor;

• s (torch.tensor) – the set of states in which we want to evaluate the advantage;

• ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;

• r (torch.tensor) – the reward obtained in each transition from state s to state ss;

• absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;

• gamma (float) – the discount factor of the considered problem.

Returns:

The new estimate for the value function of the next state and the advantage function.

compute_gae(V, s, ss, r, absorbing, last, gamma, lam)

Function to compute Generalized Advantage Estimation (GAE) and new value function target over a dataset.

“High-Dimensional Continuous Control Using Generalized Advantage Estimation”. Schulman J. et al.. 2016.

Parameters:

• V (Regressor) – the current value function regressor;

• s (torch.tensor) – the set of states in which we want to evaluate the advantage;

• ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;

• r (torch.tensor) – the reward obtained in each transition from state s to state ss;

• absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;

• last (torch.tensor) – an array of boolean flags indicating if the reached state is the last of the trajectory;

• gamma (float) – the discount factor of the considered problem;

• lam (float) – the value for the lamba coefficient used by GEA algorithm.

Returns:

The new estimate for the value function of the next state and the estimated generalized advantage.

Variance parameters

class VarianceParameter(*args, **kwargs)

Bases: Parameter

Abstract class to implement variance-dependent parameters. A target parameter is expected.

__init__(value, exponential=False, min_value=None, tol=1.0, size=(1,), log_table=False)

Constructor.

Parameters:

tol (float) – value of the variance of the target variable such that The parameter value is 0.5.

_compute(*idx, **kwargs)

Returns:

The value of the parameter in the provided index.

update(*idx, **kwargs)

Updates the value of the parameter in the provided index.

Parameters:

• *idx (list) – index of the parameter whose number of visits has to be updated.

• target (float) – Value of the target variable;

• factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.

class VarianceIncreasingParameter(*args, **kwargs)

Bases: VarianceParameter

Class implementing a parameter that increases with the target variance.

class VarianceDecreasingParameter(*args, **kwargs)

Bases: VarianceParameter

Class implementing a parameter that decreases with the target variance.

class WindowedVarianceParameter(*args, **kwargs)

Bases: Parameter

Abstract class to implement variance-dependent parameters. A target parameter is expected. differently from the “Variance Parameter” class the variance is computed in a window interval.

__init__(value, exponential=False, min_value=None, tol=1.0, window=100, size=(1,), log_table=False)

Constructor.

Parameters:

• tol (float) – value of the variance of the target variable such that the parameter value is 0.5.

• window (int)

_compute(*idx, **kwargs)

Returns:

The value of the parameter in the provided index.

update(*idx, **kwargs)

Updates the value of the parameter in the provided index.

Parameters:

• *idx (list) – index of the parameter whose number of visits has to be updated.

• target (float) – Value of the target variable;

• factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.

class WindowedVarianceIncreasingParameter(*args, **kwargs)

Bases: WindowedVarianceParameter

Class implementing a parameter that decreases with the target variance, where the variance is computed in a fixed length window.