Agent-Environment Interface

The three basic interface of mushroom_rl are the Agent, the Environment and the Core interface.

• The Agent is the basic interface for any Reinforcement Learning algorithm.

• The Environment is the basic interface for every problem/task that the agent should solve.

• The Core is a class used to control the interaction between an agent and an environment.

We provide the logging functionality with the Logger class. Finally, the MushroomObject interface implements serialization of MushroomRL data on the disk (load/save functionality) and forwards a logger down the object tree.

Agent

MushroomRL provides the implementations of several algorithms belonging to all categories of RL:

• value-based;

• policy-search;

• actor-critic.

One can easily implement customized algorithms following the structure of the already available ones, by extending the following interface:

class AgentInfo(*args, **kwargs)

Bases: MushroomObject

__init__(is_episodic, policy_state_shape, backend)

class Agent(*args, **kwargs)

Bases: MushroomObject

This class implements the functions to manage the agent (e.g. move the agent following its policy).

__init__(mdp_info, policy, is_episodic=False, backend='numpy', history_length=None, action_history_length=None, history_manager=None)

Constructor.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• policy (Policy) – the policy followed by the agent;

• is_episodic (bool, False) – whether the agent is learning in an episodic fashion or not;

• backend (str, 'numpy') – array backend to be used by the algorithm;

• history_length (int, None) – number of observations stacked as input to the policy. When greater than 1 the agent builds a HistoryManager that assembles the stacked observation on the fly; the history is reconstructable and never stored as policy state.

• action_history_length (int, None) – number of previous actions stacked as input to the policy. When greater than 0 the HistoryManager also assembles the stacked previous actions on the fly.

• history_manager (HistoryManager, None) – an already built history manager to use, mutually exclusive with history_length and action_history_length.

fit(dataset)

Fit step.

Parameters:

dataset (Dataset) – the dataset.

draw_action(state)

Return the action to execute in the given state. It is the action returned by the policy or the action set by the algorithm (e.g. in the case of SARSA). When the policy is stateful, its internal state is updated and can be read through get_policy_state().

Parameters:

state – the state where the agent is.

Returns:

The action to be executed.

property policy_state

The current internal state of the policy, in the agent’s own backend, or None if the policy is stateless.

episode_start(initial_state, episode_info)

Called by the Core when a new episode starts.

Parameters:

• initial_state (Array) – vector representing the initial state of the environment.

• episode_info (dict) – a dictionary containing the information at reset, such as context.

Returns:

A tuple containing the policy initial state and, optionally, the policy parameters

episode_start_vectorized(initial_states, episode_info, start_mask)

Called by the Core at the start of a new episode when using a vectorized environment.

Parameters:

• initial_states (Array) – the initial states of the environment.

• episode_info (dict) – a dictionary containing the information at reset, such as context;

• start_mask (Array) – boolean mask to select the environments that are starting a new episode

Returns:

A tuple containing the policy initial states and, optionally, the policy parameters

stop()

Method used to stop an agent. Useful when dealing with real world environments, simulators, or to cleanup environments internals after a core learn/evaluate to enforce consistency.

add_core_preprocessor(preprocessor)

Add preprocessor to the core’s preprocessor list. The preprocessors are applied in order.

Parameters:

preprocessor (object) – state preprocessors to be applied to state variables before feeding them to the agent.

add_agent_preprocessor(preprocessor)

Add preprocessor to the agent’s preprocessor list. The preprocessors are applied in order.

Parameters:

preprocessor (object) – state preprocessors to be applied to state variables before feeding them to the agent.

property core_preprocessors

Access to core’s state preprocessors stored in the agent.

property history_length

The number of observations stacked as policy input, 1 when no history is used.

property history_manager

The HistoryManager used to assemble the policy input.

_agent_preprocess(state)

Applies all the agent’s preprocessors to the state.

Parameters:

state (Array) – the state where the agent is;

Returns:

The preprocessed state.

_update_agent_preprocessor(state)

Updates the stats of all the agent’s preprocessors given the state.

Parameters:

state (Array) – the state where the agent is;

Environment

MushroomRL provides several implementation of well known benchmarks with both continuous and discrete action spaces.

To implement a new environment, it is mandatory to use the following interface:

class MDPInfo(*args, **kwargs)

Bases: MushroomObject

This class is used to store the information of the environment.

__init__(observation_space, action_space, gamma, horizon, dt=0.1, backend='numpy')

Constructor.

Parameters:

• observation_space ([Box, Discrete]) – the state space;

• action_space ([Box, Discrete]) – the action space;

• gamma (float) – the discount factor;

• horizon (int) – the horizon;

• dt (float, 1e-1) – the control timestep of the environment;

• backend (str, 'numpy') – the type of data library used to generate state and actions.

property size

Returns: The sum of the number of discrete states and discrete actions. Only works for discrete spaces.

property shape

Returns: The concatenation of the shape tuple of the state and action spaces.

class Environment(mdp_info)

Bases: object

Basic interface used by any MushroomRL environment.

classmethod register()

Register an environment in the environment list.

static list_registered()

List registered environments.

Returns:

The list of the registered environments.

static make(env_name, *args, **kwargs)

Generate an environment given an environment name and parameters. The environment is created using the generate method, if available. Otherwise, the constructor is used. The generate method has a simpler interface than the constructor, making it easier to generate a standard version of the environment. If the environment name contains a ‘.’ separator, the string is splitted, the first element is used to select the environment and the other elements are passed as positional parameters.

Parameters:

• env_name (str) – Name of the environment,

• *args – positional arguments to be provided to the environment generator;

• **kwargs – keyword arguments to be provided to the environment generator.

Returns:

An instance of the constructed environment.

__init__(mdp_info)

Constructor.

Parameters:

mdp_info (MDPInfo) – an object containing the info of the environment.

seed(seed)

Set the seed of the environment.

Parameters:

seed (float) – the value of the seed.

reset(state=None)

Reset the environment to the initial state.

Parameters:

state (np.ndarray, None) – the state to set to the current state.

Returns:

The initial state and a dictionary containing the info for the episode.

step(action)

Move the agent from its current state according to the action.

Parameters:

action (np.ndarray) – the action to execute.

Returns:

The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also, an additional dictionary is returned (possibly empty).

render(record=False)

Render the environment to screen.

Parameters:

record (bool, False) – whether the visualized image should be returned or not.

Returns:

The visualized image, or None if the record flag is set to false.

stop()

Method used to stop an env. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

property info

Returns: An object containing the info of the environment.

_bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;

• min_value – the minimum value;

• max_value – the maximum value;

Returns:

The bounded variable.

Core

class Core(agent, env, *args, **kwargs)

Bases: object

Implements the functions to run a generic algorithm.

This is a facade that, depending on the type of environment provided, dispatches to a single-environment (SequentialCore) or a vectorized (VectorizedCore) implementation. Both expose the same interface, so user code only ever instantiates Core.

__init__(agent, env, callbacks_fit=None, callback_step=None, logger=None)

Constructor.

Parameters:

• agent (Agent) – the agent moving according to a policy;

• env (Environment) – the environment in which the agent moves;

• callbacks_fit (list) – list of callbacks to execute at the end of each fit;

• callback_step (Callback) – callback to execute after each step;

• logger (Logger, None) – the logger to be used by the agent. If provided, it is set on the agent via agent.set_logger and the video fps is configured from the environment.

learn(n_steps=None, n_episodes=None, n_steps_per_fit=None, n_episodes_per_fit=None, render=False, record=False, quiet=False)

This function moves the agent in the environment and fits the policy using the collected samples. The agent can be moved for a given number of steps or a given number of episodes and, independently of this choice, the policy can be fitted after a given number of steps or a given number of episodes. The environment is reset at the beginning of the learning process.

Parameters:

• n_steps (int, None) – number of steps to move the agent;

• n_episodes (int, None) – number of episodes to move the agent;

• n_steps_per_fit (int, None) – number of steps between each fit of the policy;

• n_episodes_per_fit (int, None) – number of episodes between each fit of the policy;

• render (bool, False) – whether to render the environment or not;

• record (bool, False) – whether to record a video of the environment or not. If True, also the render flag should be set to True.

• quiet (bool, False) – whether to show the progress bar or not.

evaluate(initial_states=None, n_steps=None, n_episodes=None, render=False, quiet=False, record=False)

This function moves the agent in the environment using its policy. The agent is moved for a provided number of steps, episodes, or from a set of initial states for the whole episode. The environment is reset at the beginning of the learning process.

Parameters:

• initial_states (np.ndarray, None) – the starting states of each episode;

• n_steps (int, None) – number of steps to move the agent;

• n_episodes (int, None) – number of episodes to move the agent;

• render (bool, False) – whether to render the environment or not;

• quiet (bool, False) – whether to show the progress bar or not;

• record (bool, False) – whether to record a video of the environment or not. If True, also the render flag should be set to True.

Returns:

The collected dataset.

set_logger(logger)

Set the logger on the agent and configure the video fps from the environment.

Parameters:

logger (Logger) – the logger to be used by the agent.

_create_core_logic()

Returns:

The CoreLogic instance driving the step/episode counters of this core.

_prepare_dataset(n_steps, n_episodes, core_counts_episodes)

Build the empty dataset used to collect the samples of a run.

Parameters:

• n_steps (int, None) – number of steps used to size the dataset;

• n_episodes (int, None) – number of episodes used to size the dataset;

• core_counts_episodes (bool) – whether the run is driven by an episode count.

Returns:

The empty dataset to be filled during the run.

_preprocess(state)

Method to apply state preprocessors.

Parameters:

state (np.ndarray) – the state to be preprocessed.

Returns:

The preprocessed state.

class SequentialCore(agent, env, *args, **kwargs)

Bases: Core

Single-environment implementation of Core.

_create_core_logic()

Returns:

The CoreLogic instance driving the step/episode counters of this core.

_prepare_dataset(n_steps, n_episodes, core_counts_episodes)

Build the empty dataset used to collect the samples of a run.

Parameters:

• n_steps (int, None) – number of steps used to size the dataset;

• n_episodes (int, None) – number of episodes used to size the dataset;

• core_counts_episodes (bool) – whether the run is driven by an episode count.

Returns:

The empty dataset to be filled during the run.

_step(render, record)

Single step.

Parameters:

render (bool) – whether to render or not.

Returns:

A tuple containing the previous state, the action sampled by the agent, the reward obtained, the reached state, the absorbing flag of the reached state and the last step flag.

_reset(initial_states)

Reset the state of the agent.

class VectorizedCore(agent, env, *args, **kwargs)

Bases: Core

Vectorized (multienvironment) implementation of Core.

_create_core_logic()

Returns:

The CoreLogic instance driving the step/episode counters of this core.

_prepare_dataset(n_steps, n_episodes, core_counts_episodes)

Build the empty dataset used to collect the samples of a run.

Parameters:

• n_steps (int, None) – number of steps used to size the dataset;

• n_episodes (int, None) – number of episodes used to size the dataset;

• core_counts_episodes (bool) – whether the run is driven by an episode count.

Returns:

The empty dataset to be filled during the run.

_step(render, record, mask)

Single step.

Parameters:

render (bool) – whether to render or not.

Returns:

A tuple containing the previous states, the actions sampled by the agent, the rewards obtained, the reached states, the absorbing flags of the reached states and the last step flags.

_reset(initial_states, last, mask)

Reset the states of the agent.

Dataset

The Dataset stores the transitions collected while an agent interacts with an environment, keeping the environment data and the agent (policy state) data in their respective backends. DatasetInfo carries the static information used to build it, and VectorizedDataset handles data collected from parallel environments.

class DatasetInfo(*args, **kwargs)

Bases: MushroomObject

Static information needed to build a Dataset. A dataset keeps its data in two backend-aware groups: the environment data (state, action, reward, next state, absorbing and last flags) and the agent data (the policy state). This class stores the array backend and device of each group, together with the shapes and dtypes of the states and actions, the horizon, the discount factor and the number of parallel environments. Build it with the create_dataset_info() (on-policy collection) or create_replay_memory_info() (replay buffer) factories.

__init__(env_backend, agent_backend, env_device, agent_device, horizon, gamma, state_shape, state_dtype, action_shape, action_dtype, policy_state_shape, n_envs=1)

Constructor.

Parameters:

• env_backend (str) – array backend of the environment data ('numpy', 'torch' or 'list');

• agent_backend (str) – array backend of the agent (policy state) data;

• env_device (str, None) – device of the environment data, only allowed with the torch backend;

• agent_device (str, None) – device of the agent data, only allowed with the torch backend;

• horizon (int) – horizon of the MDP;

• gamma (float) – discount factor;

• state_shape (tuple) – shape of a single state;

• state_dtype – data type of the states;

• action_shape (tuple) – shape of a single action;

• action_dtype – data type of the actions;

• policy_state_shape (tuple, None) – shape of the policy state, or None if the agent is stateless;

• n_envs (int, 1) – number of parallel environments.

property env_array_backend

The ArrayBackend of the environment data.

property agent_array_backend

The ArrayBackend of the agent (policy state) data.

static create_dataset_info(mdp_info, agent_info, n_envs=1, device=None)

Build the dataset info for on-policy collection: the environment data uses mdp_info.backend (forced to 'list' for infinite-horizon MDPs) and the agent data uses agent_info.backend.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• n_envs (int, 1) – number of parallel environments;

• device (str, None) – torch device used by the torch-backed data groups.

Returns:

The dataset info.

static create_replay_memory_info(mdp_info, agent_info, store_policy_state=True, device=None)

Build the dataset info for a replay memory: the whole buffer (both the transition data and the policy state) lives in the agent backend, so the environment and agent backends/devices coincide.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• store_policy_state (bool, True) – whether the policy state is stored;

• device (str, None) – torch device used by the buffer.

Returns:

The dataset info.

class Dataset(*args, **kwargs)

Bases: MushroomObject

Collection of the transitions gathered while an agent interacts with an environment. The data is split into two backend-aware groups, each delegated to a backend-specific columnar container (NumpyDataset, TorchDataset or ListDataset):

• The environment data (state, action, reward, next state, absorbing and last flags), kept in the environment backend;

• The agent data (policy state and next policy state), kept in the agent backend so that it never needs a per-step conversion. Not created when the agent is stateless.

Step info, episode info and the per-episode policy parameters (theta) are stored alongside the transitions.

class _Field(value)

Bases: IntEnum

class _PolicyField(value)

Bases: IntEnum

__init__(dataset_info, n_steps=None, n_episodes=None, core_counts_episodes=False)

Constructor. Exactly one of n_steps and n_episodes must be given; it sizes the preallocated containers (the list backend grows on demand instead).

Parameters:

• dataset_info (DatasetInfo) – the static information used to build the dataset;

• n_steps (int, None) – number of steps the dataset is allocated for;

• n_episodes (int, None) – number of episodes the dataset is allocated for;

• core_counts_episodes (bool, False) – whether the collecting core counts episodes, which needs a slightly larger preallocation.

classmethod create_raw_instance(dataset=None)

Creates an empty instance of the Dataset and populates essential data structures

Parameters:

dataset (Dataset, None) – a template dataset to be used to create the new instance.

Returns:

A new empty instance of the dataset.

classmethod from_array(states, actions, rewards, next_states, absorbings, lasts, policy_state=None, policy_next_state=None, info=None, episode_info=None, theta_list=None, horizon=None, gamma=0.99, backend='numpy', policy_backend=None, device=None)

Creates a dataset of transitions from the provided arrays.

Parameters:

• states (array) – array of states;

• actions (array) – array of actions;

• rewards (array) – array of rewards;

• next_states (array) – array of next_states;

• absorbings (array) – array of absorbing flags;

• lasts (array) – array of last flags;

• policy_state (array, None) – array of policy internal states;

• policy_next_state (array, None) – array of next policy internal states;

• info (dict, None) – dictiornay of step info;

• episode_info (dict, None) – dictiornary of episode info;

• theta_list (list, None) – list of policy parameters;

• horizon (int, None) – horizon of the mdp;

• gamma (float, 0.99) – discount factor;

• backend (str, 'numpy') – backend to be used by the dataset;

• policy_backend (str, None) – backend to be used for the policy state arrays; defaults to backend.

Returns:

The list of transitions.

append_batch(other)

Append all transitions from another dataset without copying data.

Parameters:

other (Dataset) – dataset whose transitions will be appended.

property episodes_length

Compute the length of each episode in the dataset.

Returns:

An array with the length of each episode in the dataset.

parse(to=None)

Return the dataset as set of arrays. :param to: the backend to be used for the returned arrays. By default, the dataset backend is used. :type to: str, None

Returns:

A tuple containing the arrays that define the dataset, i.e. state, action, next state, absorbing and last

parse_policy_state(to=None)

Return the policy state arrays of the dataset.

Parameters:

to (str, None) – the backend to be used for the returned arrays. By default, the policy’s backend is used.

Returns:

A tuple containing the policy state and policy next state arrays.

to_backend(backend)

Return a copy of this dataset converted to the given backend.

Parameters:

backend (str) – target backend ('numpy', 'torch', or 'list').

Returns:

A new Dataset in the requested backend, or self if the backend already matches.

select_first_episodes(n_episodes)

Return the first n_episodes episodes in the provided dataset.

Parameters:

n_episodes (int) – the number of episodes to pick from the dataset;

Returns:

A subset of the dataset containing the first n_episodes episodes.

select_random_samples(n_samples)

Return the randomly picked desired number of samples in the provided dataset.

Parameters:

n_samples (int) – the number of samples to pick from the dataset.

Returns:

A subset of the dataset containing randomly picked n_samples samples.

get_init_states()

Get the initial states of a dataset

Returns:

An array of initial states of the considered dataset.

compute_J(gamma=1.0)

Compute the cumulative discounted reward of each episode in the dataset.

Parameters:

gamma (float, 1.) – discount factor.

Returns:

The cumulative discounted reward of each episode in the dataset.

compute_metrics(gamma=1.0)

Compute the metrics of each complete episode in the dataset.

Parameters:

gamma (float, 1.) – the discount factor.

Returns:

The minimum score reached in an episode, the maximum score reached in an episode, the mean score reached, the median score reached, the number of completed episodes.

If no episode has been completed, it returns 0 for all values.

class VectorizedDataset(*args, **kwargs)

Bases: Dataset

Dataset variant for data collected from several environments in parallel. Each step stores a batch of transitions together with a boolean mask (kept in its own env-backend container) marking which environments were active. The padded per-environment episodes are turned back into a flat Dataset with flatten(), typically once per fit.

__init__(dataset_info, n_steps=None, n_episodes=None, core_counts_episodes=False)

Constructor. Exactly one of n_steps and n_episodes must be given; it sizes the preallocated containers (the list backend grows on demand instead).

Parameters:

• dataset_info (DatasetInfo) – the static information used to build the dataset;

• n_steps (int, None) – number of steps the dataset is allocated for;

• n_episodes (int, None) – number of episodes the dataset is allocated for;

• core_counts_episodes (bool, False) – whether the collecting core counts episodes, which needs a slightly larger preallocation.

append_vectorized(step, info, mask)

Append one step of a batch of parallel environments.

Parameters:

• step (tuple) – the batched transition, one entry per environment field (plus the policy state fields when the agent is stateful);

• info (dict) – the batched step info;

• mask (Array) – boolean mask selecting the environments that are currently active.

append_theta_vectorized(theta, mask)

Append the policy parameters of the active environments to their per-environment theta lists.

Parameters:

• theta (Array) – the policy parameters, one entry per environment;

• mask (Array) – boolean mask selecting the environments that are currently active.

clear(n_steps_per_fit=None)

Clear the dataset. When n_steps_per_fit is given and more than that many (masked) steps were collected, the surplus tail is carried forward into the freshly cleared dataset so the next fit starts with it.

Parameters:

n_steps_per_fit (int, None) – number of steps consumed by the fit; the rest is carried forward.

Returns:

The number of steps carried forward.

flatten(n_steps_per_fit=None)

Turn the padded per-environment data into a flat Dataset, dropping the inactive entries via the mask and concatenating the environments end to end.

Parameters:

n_steps_per_fit (int, None) – if given, keep only the first this many flattened steps.

Returns:

A flat Dataset, or None if the dataset is empty.

property mask

Boolean mask marking, for every stored step, which environments were active.

History manager

class HistoryManager(*args, **kwargs)

Bases: MushroomObject

Object in charge of assembling the per-timestep context fed to the policy, i.e. the stacked window of the most recent entries of one or more streams.

The context is a deterministic function of the observed trajectory, hence it is always reconstructable from the stored transitions and is not part of the (latent) policy state. The manager holds an ordered set of named streams, each with its own stacking length and an offset telling how many steps behind the current one its window ends (0 for the observation, 1 for the previous action). The two reserved streams are sourced by the manager itself: the obs_history stream from the state passed to __call__() (delivered in-band as the policy state) and the action_history stream from the last action recorded through record_action(). Any further stream is fed as a keyword argument to __call__() and returned under its own name. With no active stream the manager is the identity context: it passes the state through unchanged and returns no keyword arguments, so an agent that does no stacking can hold one unconditionally instead of a None.

The manager works entirely in the agent backend. Online it stacks the most recent entry of each stream, and the same stacking rule is exposed offline through build_history() (regular buffer) and build_history_circular_buffer() (circular replay buffer), so the window built while interacting with the environment and the one rebuilt from a stored buffer are guaranteed to match. Each per-step window is (length, *shape), squeezed to (*shape) when length is 1.

Each stream is described by a specification dictionary with the keys length, shape and dtype plus any number of options; the only option acted upon by this class is offset (default 0). Subclasses may store and honor richer options without changing the base machinery. The reserved streams and their conventional lengths/offsets are wired up from the MDP and action spaces by the default_streams() factory.

__init__(agent_info, streams=None)

Constructor.

Parameters:

• agent_info (AgentInfo) – information about the agent, providing the array backend in which the manager keeps its buffers and returns the stacked windows;

• streams (dict, None) – the named streams assembled by the manager, given as a mapping name -> spec, where each spec is a dictionary with the keys length, shape and dtype and, optionally, offset (default 0), the number of steps behind the current one at which the window ends. Each spec is forwarded to add_stream(), and each stream’s window is returned under name in the output of __call__(). The two reserved names are sourced by the manager itself: obs_history (from the in-band state, returned positionally) and action_history (from the last recorded action, conventionally offset 1). These reserved streams, with their shapes and data types read from the MDP and action spaces, are wired up by default_streams().

__call__(state=None, **extra)

Append the current entries to the buffers and return the per-timestep context split for the policy call: the observation input to be passed positionally as state and a dictionary of the additional conditioning streams to be forwarded as keyword arguments. The reserved obs_history stream, when active, replaces state with its stacked window; otherwise the raw state is passed through unchanged. The action_history stream is sourced from the last action recorded through record_action(). Each remaining stream is forwarded under its own name.

Parameters:

• state – the current observation, already in the agent backend, consumed by the observation stream;

• **extra – the current value of each other stream, keyed by its name and already in the agent backend; a stream whose value is not provided is zero-padded for that step.

Returns:

A tuple (state, policy_kwargs) ready to be used as policy.draw_action(state, **policy_kwargs). Each window has shape (length, *shape) (single-environment) or (n_envs, length, *shape) (vectorized), squeezed along the length axis when the stream length is 1.

add_stream(name, length, shape, dtype, offset=0, **options)

Register a named stream to be stacked by the manager. Streams are usually declared through the constructor, but this method allows building a manager and adding buffers programmatically.

Parameters:

• name (str) – the name under which the stream’s window is returned by __call__() (obs_history is reserved for the in-band observation stream);

• length (int) – number of entries stacked in the stream’s window;

• shape (tuple) – shape of a single entry of the stream;

• dtype – data type of the stream, converted to the agent backend;

• offset (int, 0) – number of steps behind the current one at which the window ends;

• **options – additional per-stream options stored in the specification; ignored by the base class.

reset()

Reset the buffers at the beginning of a single-environment episode.

reset_vectorized(start_mask)

Reset the buffers for the environments selected by start_mask, leaving the others untouched. The buffers are (re)allocated the first time this is called and whenever the number of environments changes; otherwise the selected environments are zeroed in place.

Parameters:

start_mask – boolean mask selecting the environments that are starting a new episode.

record_action(action)

Record the action just drawn by the agent so that it becomes the most recent entry of the action_history window at the next step (its offset 1). Called by the agent after every draw_action. The last action is always kept (it is cheap and available for custom logging), even when the previous-action stream is not active, and it is not stacked into any window.

Parameters:

action – the action just drawn, already in the agent backend.

parse_history(dataset, to=None)

Parse a dataset into its arrays, the analog of Dataset.parse() with the history stacking rules applied: the state and next-state windows replace the raw observations with the stack of their most recent entries, i.e. the temporal context fed to the policy, and the window of every other active stream (e.g. the previous actions) is returned aside.

Parameters:

• dataset (Dataset) – the dataset to parse;

• to (str, None) – the backend of the returned arrays; when None the agent backend is used.

Returns:

The tuple (state, action, reward, next_state, absorbing, last, extra), as Dataset.parse() plus a dictionary mapping every other active stream name to its window; state and next_state carry the stacked windows. A stream stacking a single entry collapses to the raw value.

parse_history_circular_buffer(dataset, anchor_idxs, size, full, max_size, to=None)

Parse the transitions at anchor_idxs of a dataset stored in a circular replay buffer, as in parse_history(). A circular buffer overwrites its oldest entry once full, so a window is read modulo the capacity and is never stitched across the write head.

Parameters:

• dataset (Dataset) – the dataset to parse;

• anchor_idxs – the buffer position of each transition to parse;

• size (int) – the number of entries currently stored;

• full (bool) – whether the buffer has wrapped around;

• max_size (int) – the buffer capacity;

• to (str, None) – the backend of the returned arrays; when None the agent backend is used.

Returns:

The tuple (state, action, reward, next_state, absorbing, last, extra), as in parse_history().

parse_nstep_history(dataset, gamma=1.0, n_steps_return=1, anchor_idxs=None, to=None)

Parse a dataset into its n-step arrays, i.e. parse_history() with the n-step return folded in: the reward becomes the discounted n-step reward and the next-state window, the absorbing and the last flags belong to the n-ahead endpoint, while the state and previous-action windows belong to the current step. Only the transitions whose n-step return is well-defined are returned (see build_nstep_return()); the ones whose window would cross a truncated episode or run past the newest stored transition are dropped.

Parameters:

• dataset (Dataset) – the dataset to parse;

• gamma (float, 1.) – the discount factor;

• n_steps_return (int, 1) – the number of steps summed in the return;

• anchor_idxs (None) – the buffer position of each transition; when None every stored transition is used;

• to (str, None) – the backend of the returned arrays; when None the agent backend is used.

Returns:

The tuple (state, action, reward, next_state, absorbing, last, extra), as in parse_nstep_history_circular_buffer().

parse_nstep_history_circular_buffer(dataset, anchor_idxs, gamma, n_steps_return, size, full, max_size, write_head, to=None)

Parse the transitions at anchor_idxs of a dataset stored in a circular replay buffer into their n-step arrays, as in parse_nstep_history(). Only the valid transitions are returned; extra carries the endpoint index of each of them under endpoint and the surviving anchor index under anchor, so the caller can gather any further column (e.g. the policy state) aligned to the returned batch.

Parameters:

• dataset (Dataset) – the dataset to parse;

• anchor_idxs – the buffer position of each transition;

• gamma (float) – the discount factor;

• n_steps_return (int) – the number of steps summed in the return;

• size (int) – the number of entries currently stored;

• full (bool) – whether the buffer has wrapped around;

• max_size (int) – the buffer capacity;

• write_head (int) – the next write position of the buffer;

• to (str, None) – the backend of the returned arrays; when None the agent backend is used.

Returns:

The tuple (state, action, reward, next_state, absorbing, last, extra), as parse_history() but restricted to the valid transitions and with the discounted n-step reward and the endpoint next-state, absorbing and last. The endpoint and surviving anchor indices are provided under extra['endpoint'] and extra['anchor'].

build_history(name, buffer, last, anchor_idxs=None, backend=None)

Rebuild the name stream window offline for a batch of anchor indices, reading from a regular (non-circular) buffer such as an in-memory dataset. Each window is built by walking backwards from its anchor up to the stream length, stopping at the start of the buffer or at an episode boundary and zero-padding the missing older entries, which reproduces exactly the window assembled online by __call__().

Parameters:

• name (str) – the stream to rebuild, providing its length and offset;

• buffer – the buffer to read from;

• last – the last flags of the buffer, used to stop at episode boundaries;

• anchor_idxs (None) – buffer indices of the current step of each window; when None every timestep of the buffer is an anchor;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

An array of shape (n_samples, length, *entry_shape) (squeezed along length when it is 1), with older entries at lower indices.

build_history_circular_buffer(name, buffer, last, anchor_idxs, size, full, max_size, backend=None)

Same as build_history(), but reading from a circular replay buffer: positions are taken modulo the buffer size, the walk stops both at episode boundaries and at the buffer limits (the start of a not-yet-wrapped buffer, which is the first stored episode start, and the write head of a full one, which the anchors are assumed to stay clear of, see max_reach).

Parameters:

• name (str) – the stream to rebuild, providing its length and offset;

• buffer – the circular buffer to read from (e.g. the state or action column of a replay memory);

• last – the last flags of the buffer, used to stop at episode boundaries;

• anchor_idxs – buffer indices of the current step of each window;

• size (int) – the number of valid entries currently stored in the buffer;

• full (bool) – whether the circular buffer has wrapped around;

• max_size (int) – the maximum size of the circular buffer;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

An array of shape (n_samples, length, *entry_shape) (squeezed along length when it is 1), with older entries at lower indices.

build_nstep_return(reward, absorbing, last, anchor_idxs=None, gamma=1.0, n_steps_return=1, backend=None)

Compute the n-step return of a batch of transitions, keeping only the valid ones. The n-step return of a transition is the discounted sum of the rewards collected over the next n_steps_return steps; its bootstrap target is the transition n_steps_return steps ahead (the endpoint), or the terminal transition if the episode ends earlier. A transition is invalid, and dropped, when its window would cross a truncated (non-absorbing) episode or run past the newest stored transition.

Parameters:

• reward – the reward of each transition;

• absorbing – the absorbing flag of each transition;

• last – the episode-boundary flags of the buffer;

• anchor_idxs (None) – the buffer position of each transition; when None every stored transition is used;

• gamma (float, 1.) – the discount factor;

• n_steps_return (int, 1) – the number of steps summed in the return;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

the discounted n-step reward, the surviving anchor index and the index of the bootstrap transition.

Return type:

The tuple (reduced_reward, anchor, endpoint) restricted to the valid transitions

build_nstep_return_circular_buffer(reward, absorbing, last, anchor_idxs, gamma, n_steps_return, size, full, max_size, write_head, backend=None)

Compute the n-step return of a batch of transitions stored in a circular replay buffer, keeping only the valid ones, as in build_nstep_return(). An absorbing terminal ends the return early and is the bootstrap target; a non-absorbing truncation, or a window reaching past the newest stored transition (the write head), makes the transition invalid.

Parameters:

• reward – the reward column of the buffer;

• absorbing – the absorbing column of the buffer;

• last – the episode-boundary flags of the buffer;

• anchor_idxs – the buffer position of each transition;

• gamma (float) – the discount factor;

• n_steps_return (int) – the number of steps summed in the return;

• size (int) – the number of entries currently stored;

• full (bool) – whether the buffer has wrapped around;

• max_size (int) – the buffer capacity;

• write_head (int) – the next write position; the newest stored transition is the one before it;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

The tuple (reduced_reward, anchor, endpoint) restricted to the valid transitions, as in build_nstep_return().

nstep_valid(absorbing, last, anchor_idxs=None, n_steps_return=1, backend=None)

Compute, for a batch of transitions, whether their n-step return is well-defined, without reducing the reward or building the batch. This is the validity check behind the replay-memory sampling mask: it walks the next n_steps_return steps of each transition and reports whether its window crosses a non-absorbing truncation or runs past the buffer limits. The flag is aligned to anchor_idxs (nothing is dropped), so it can be used as a per-transition mask.

Parameters:

• absorbing – the absorbing flag of each transition;

• last – the episode-boundary flags of the buffer;

• anchor_idxs (None) – the buffer position of each transition; when None every stored transition is used;

• n_steps_return (int, 1) – the number of steps summed in the return;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

The boolean valid flag aligned to anchor_idxs, True where the n-step return is well-defined.

nstep_valid_circular_buffer(absorbing, last, anchor_idxs, n_steps_return, size, full, max_size, write_head, backend=None)

Compute whether the n-step return of a batch of transitions stored in a circular replay buffer is well-defined, as in nstep_valid().

Parameters:

• absorbing – the absorbing column of the buffer;

• last – the episode-boundary flags of the buffer;

• anchor_idxs – the buffer position of each transition;

• n_steps_return (int) – the number of steps summed in the return;

• size (int) – the number of entries currently stored;

• full (bool) – whether the buffer has wrapped around;

• max_size (int) – the buffer capacity;

• write_head (int) – the next write position; the newest stored transition is the one before it;

• backend (ArrayBackend, None) – the required array backend; when None the agent backend is used.

Returns:

The boolean valid flag aligned to anchor_idxs, as in nstep_valid().

classmethod default_streams(mdp_info, agent_info, history_length=None, action_history_length=None)

Build a manager wired up with the default reserved streams read from the MDP and action spaces. When neither stream is active the returned manager is the identity context (no stacking). The observation stream is registered as obs_history (offset 0) only when history_length is greater than 1; the previous-action stream is registered as action_history (offset 1) only when action_history_length is greater than 0.

Parameters:

• mdp_info (MDPInfo) – information about the MDP;

• agent_info (AgentInfo) – information about the agent;

• history_length (int, None) – number of observations stacked as policy input;

• action_history_length (int, None) – number of previous actions stacked as policy input.

Returns:

The HistoryManager instance.

property history_length

The number of observations stacked as policy input, or 1 when the observation stream is not active.

property uses_action

Whether the previous-action stream is active, i.e. an action_history window is assembled as policy input.

property action_history_length

The number of previous actions stacked as policy input, or 0 when the previous-action stream is not active.

property max_reach

The deepest backward reach across all streams, i.e. the maximum of offset + length - 1. A full circular buffer reserves this many of its oldest samples so that every window is rebuilt without crossing the write head.

Array Backend

class ArrayBackend

Bases: object

Interface for the array backends used across MushroomRL. A backend abstracts the array type used to store and manipulate data (states, actions, rewards, …) so that the same code can run on different array libraries. Three backends are provided: NumpyBackend, TorchBackend and ListBackend, selected by name through get_array_backend().

static get_backend_name()

Returns:

The name of the backend ('numpy', 'torch' or 'list').

static get_backend_serialization()

Returns:

The name of the MushroomObject save method (see _add_save_attr) to use for attributes stored in this backend’s array type, i.e. 'numpy' or 'torch'. Backends with no dedicated save method (e.g. ListBackend) return the name of another backend able to serialize their data instead ('numpy').

static get_array_backend(backend_name)

Parameters:

backend_name (str) – name of the backend, one of 'numpy', 'torch' or 'list'.

Returns:

The ArrayBackend subclass registered under backend_name.

static get_array_backend_from(array)

Parameters:

array (object) – an array produced by one of the supported backends (a NumPy ndarray, a PyTorch Tensor, or a list/deque).

Returns:

The ArrayBackend subclass handling the type of array.

classmethod check_device(device)

Parameters:

device – device requested by the caller.

Raises:

ValueError – if device is not None. Only backends that support devices (i.e. TorchBackend) override this check to accept a non-None device.

classmethod convert(*arrays, to=None, backend=None)

Convert one or more arrays from their current backend to another one.

Parameters:

• *arrays – one or more arrays to convert;

• to (str, None) – name of the destination backend. If None, the backend calling this method (cls) is used;

• backend (ArrayBackend, None) – backend of the input arrays. If None, it is autodetected from the first element of arrays.

Returns:

The converted array, or a tuple of converted arrays if more than one was passed in arrays.

static convert_to_backend(backend, array)

Convert a single array from another backend into this backend’s native array type. Unlike convert(), this is a static method called on the destination backend, taking the source backend as an explicit positional argument, e.g. TorchBackend.convert_to_backend(NumpyBackend, array) converts a NumPy array into a PyTorch Tensor.

Parameters:

• backend (ArrayBackend) – the backend array currently belongs to;

• array – the array to convert.

Returns:

array converted into this backend’s native format.

classmethod arrays_to_numpy(*arrays)

Parameters:

*arrays – one or more arrays in this backend’s format.

Returns:

A tuple with the arrays converted to NumPy ndarray.

classmethod arrays_to_torch(*arrays)

Parameters:

*arrays – one or more arrays in this backend’s format.

Returns:

A tuple with the arrays converted to PyTorch Tensor.

classmethod arrays_to_list(*arrays)

Parameters:

*arrays – one or more arrays in this backend’s format.

Returns:

A tuple with the arrays converted to plain Python list.

static to_numpy(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a NumPy ndarray.

static to_torch(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a PyTorch Tensor.

static to_list(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a plain Python list.

static as_array(array)

Cast array to this backend’s native array type without changing its backend, materializing it if needed (e.g. wrapping a plain Python list into a NumPy/PyTorch array).

Parameters:

array – an array-like object.

Returns:

array as a native object of this backend.

static from_list(array)

Build a backend array from a plain Python list (the inverse of to_list()).

Parameters:

array (list) – a plain Python list.

Returns:

array converted to this backend’s native array type.

static to_backend_dtype(dtype)

Parameters:

dtype – a dtype specification, either native to this backend or to another supported backend (e.g. a NumPy dtype or a PyTorch dtype).

Returns:

dtype converted to this backend’s native dtype representation.

static empty(shape, device=None)

Parameters:

• shape – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new, uninitialized array of the given shape.

static full(shape, value)

Parameters:

• shape – shape of the array to create;

• value – fill value.

Returns:

A new array of the given shape, filled with value.

static concatenate(list_of_arrays, dim)

Parameters:

• list_of_arrays – a list of arrays to concatenate;

• dim – dimension along which the arrays are concatenated.

Returns:

The arrays in list_of_arrays concatenated along dim.

static flatten(array)

Merge the first two axes of an array into a single leading axis, grouping the result by the second axis and ordering it by the first axis within each group (i.e. all entries with index 0 along the second axis, in order of their index along the first axis, then all entries with index 1 along the second axis, …). This differs from a plain row-major reshape, which would order elements by the first axis first.

Parameters:

array – an array of shape (A, B, ...).

Returns:

array reshaped to (A * B, ...).

static pack_padded_sequence(array, mask)

Select the entries of an array marked by a boolean mask over its first two axes, and concatenate them into a single leading axis, grouped by the second axis and ordered by the first axis within each group (i.e. all selected entries with index 0 along the second axis, in order of their index along the first axis, then all selected entries with index 1 along the second axis, …).

Parameters:

• array – an array of shape (A, B, ...);

• mask – a boolean array of shape (A, B) marking the entries of array to keep.

Returns:

The entries of array selected by mask, concatenated in the order described above.

classmethod zeros(*dims, dtype, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with zeros.

classmethod ones(*dims, dtype, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with ones.

classmethod zeros_like(array, dtype, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with zeros.

classmethod ones_like(array, dtype, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with ones.

static masked_init(mask, values)

Build an array of shape (len(mask), *values.shape[1:]) where the entries selected by mask are filled, in order, with values, and the remaining entries are left uninitialized.

Parameters:

• mask – a boolean array of shape (N,);

• values – an array of shape (M, ...), with M equal to the number of True entries in mask.

Returns:

An array of shape (N, ...) with values scattered at the positions where mask is True. The scattered entries are independent of values (NumpyBackend/ TorchBackend copy the underlying numeric buffer; ListBackend deep-copies each scattered element, since it can hold nested/ragged Python containers).

static shape(array)

Parameters:

array – an array.

Returns:

The shape of array, as a tuple.

static size(arr)

Parameters:

arr – an array.

Returns:

The total number of elements in arr.

static none()

Returns:

This backend’s representation of a missing value (nan for NumpyBackend and TorchBackend, None for ListBackend).

static inf()

Returns:

This backend’s representation of positive infinity.

static copy(array)

Parameters:

array – an array.

Returns:

An independent copy of array. For NumpyBackend/TorchBackend this is a shallow copy of the underlying numeric buffer, which is sufficient since they only ever hold regular numeric data. ListBackend performs a deep copy instead, since it can hold nested/ragged Python containers whose inner elements a shallow copy would still alias.

static squeeze(array, dim)

Parameters:

• array – an array;

• dim – dimension(s) to remove, must have size 1. If None, all size-1 dimensions are removed.

Returns:

array with the size-1 dimension(s) removed.

static expand_dims(array, dim)

Parameters:

• array – an array;

• dim – position where the new axis is inserted.

Returns:

array with a new size-1 axis inserted at position dim.

static atleast_2d(array)

Parameters:

array – an array.

Returns:

array reshaped so that it has at least two dimensions.

static repeat(array, repeats)

Parameters:

• array – an array;

• repeats – number of times each element is repeated.

Returns:

array with each element repeated repeats times.

static stack(lst, dim)

Parameters:

• lst – a list of arrays with the same shape;

• dim – dimension along which the arrays are stacked.

Returns:

The arrays in lst stacked along a new dimension dim.

static where(cond, x=None, y=None)

Parameters:

• cond – a boolean array/condition;

• x (None) – array of values to select where cond is True;

• y (None) – array of values to select where cond is False.

Returns:

If x and y are None, the indices where cond is True. Otherwise, an array with elements taken from x where cond is True and from y elsewhere.

static nonzero(array)

Parameters:

array – an array.

Returns:

The indices of the nonzero elements of array.

static abs(array)

Parameters:

array – an array.

Returns:

The element-wise absolute value of array.

static exp(array)

Parameters:

array – an array.

Returns:

The element-wise exponential of array.

static sqrt(array)

Parameters:

array – an array.

Returns:

The element-wise square root of array.

static clip(array, min, max)

Parameters:

• array – an array;

• min – lower bound;

• max – upper bound.

Returns:

array with values clipped to the [min, max] range.

static sum(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the sum is computed. If None, the sum over the whole array is returned.

Returns:

The sum of array along dim.

static median(array)

Parameters:

array – an array.

Returns:

The median of the elements of array.

static max(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the maximum is computed. If None, the maximum over the whole array is returned.

Returns:

The maximum value(s) of array along dim.

static min(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the minimum is computed. If None, the minimum over the whole array is returned.

Returns:

The minimum value(s) of array along dim.

static norm(array, ord=None, dim=None)

Parameters:

• array – an array;

• ord (None) – order of the norm (see numpy.linalg.norm/torch.linalg.norm for the accepted values). If None, the default order for the underlying library is used;

• dim (None) – dimension along which the norm is computed. If None, the norm of the flattened array is returned.

Returns:

The norm of array along dim.

static maximum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise maximum of x and y.

static minimum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise minimum of x and y.

static logical_and(x, y)

Parameters:

• x – a boolean array;

• y – a boolean array.

Returns:

The element-wise logical AND of x and y.

static rand(*dims, device=None)

Parameters:

• *dims – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape dims sampled uniformly in [0, 1).

static randint(low, high, size, device=None)

Parameters:

• low – lowest (inclusive) integer to be drawn;

• high – highest (exclusive) integer to be drawn;

• size – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape size with random integers in [low, high).

static multinomial(p)

Parameters:

p – a 1D array of (unnormalized) probabilities.

Returns:

A single index sampled according to the probabilities in p.

static uniform(low, high)

Parameters:

• low – lower bound(s) of the uniform distribution;

• high – upper bound(s) of the uniform distribution.

Returns:

An array sampled uniformly between low and high.

static arange(start, stop, step=1, dtype=None, device=None)

Parameters:

• start – start of the interval;

• stop – end of the interval (exclusive);

• step (1) – spacing between values;

• dtype (None) – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array with evenly spaced values within the given interval.

class NumpyBackend

Bases: ArrayBackend

Array backend storing data in NumPy ndarray objects. It is the default backend for CPU-based environments and agents.

static get_backend_name()

Returns:

The name of the backend ('numpy', 'torch' or 'list').

static get_backend_serialization()

Returns:

The name of the MushroomObject save method (see _add_save_attr) to use for attributes stored in this backend’s array type, i.e. 'numpy' or 'torch'. Backends with no dedicated save method (e.g. ListBackend) return the name of another backend able to serialize their data instead ('numpy').

static convert_to_backend(backend, array)

Convert a single array from another backend into this backend’s native array type. Unlike convert(), this is a static method called on the destination backend, taking the source backend as an explicit positional argument, e.g. TorchBackend.convert_to_backend(NumpyBackend, array) converts a NumPy array into a PyTorch Tensor.

Parameters:

• backend (ArrayBackend) – the backend array currently belongs to;

• array – the array to convert.

Returns:

array converted into this backend’s native format.

static to_numpy(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a NumPy ndarray.

static to_torch(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a PyTorch Tensor.

static to_list(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a plain Python list.

static as_array(array)

Cast array to this backend’s native array type without changing its backend, materializing it if needed (e.g. wrapping a plain Python list into a NumPy/PyTorch array).

Parameters:

array – an array-like object.

Returns:

array as a native object of this backend.

static from_list(array)

Build a backend array from a plain Python list (the inverse of to_list()).

Parameters:

array (list) – a plain Python list.

Returns:

array converted to this backend’s native array type.

classmethod to_backend_dtype(dtype)

Parameters:

dtype – a dtype specification, either native to this backend or to another supported backend (e.g. a NumPy dtype or a PyTorch dtype).

Returns:

dtype converted to this backend’s native dtype representation.

static empty(shape, device=None)

Parameters:

• shape – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new, uninitialized array of the given shape.

static full(shape, value)

Parameters:

• shape – shape of the array to create;

• value – fill value.

Returns:

A new array of the given shape, filled with value.

static concatenate(list_of_arrays, dim=0)

Parameters:

• list_of_arrays – a list of arrays to concatenate;

• dim – dimension along which the arrays are concatenated.

Returns:

The arrays in list_of_arrays concatenated along dim.

static flatten(array)

Merge the first two axes of an array into a single leading axis, grouping the result by the second axis and ordering it by the first axis within each group (i.e. all entries with index 0 along the second axis, in order of their index along the first axis, then all entries with index 1 along the second axis, …). This differs from a plain row-major reshape, which would order elements by the first axis first.

Parameters:

array – an array of shape (A, B, ...).

Returns:

array reshaped to (A * B, ...).

static pack_padded_sequence(array, mask)

Select the entries of an array marked by a boolean mask over its first two axes, and concatenate them into a single leading axis, grouped by the second axis and ordered by the first axis within each group (i.e. all selected entries with index 0 along the second axis, in order of their index along the first axis, then all selected entries with index 1 along the second axis, …).

Parameters:

• array – an array of shape (A, B, ...);

• mask – a boolean array of shape (A, B) marking the entries of array to keep.

Returns:

The entries of array selected by mask, concatenated in the order described above.

classmethod zeros(*dims, dtype=<class 'float'>, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with zeros.

classmethod ones(*dims, dtype=<class 'float'>, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with ones.

classmethod zeros_like(array, dtype=<class 'float'>, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with zeros.

classmethod ones_like(array, dtype=<class 'float'>, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with ones.

static masked_init(mask, values)

Build an array of shape (len(mask), *values.shape[1:]) where the entries selected by mask are filled, in order, with values, and the remaining entries are left uninitialized.

Parameters:

• mask – a boolean array of shape (N,);

• values – an array of shape (M, ...), with M equal to the number of True entries in mask.

Returns:

An array of shape (N, ...) with values scattered at the positions where mask is True. The scattered entries are independent of values (NumpyBackend/ TorchBackend copy the underlying numeric buffer; ListBackend deep-copies each scattered element, since it can hold nested/ragged Python containers).

static shape(array)

Parameters:

array – an array.

Returns:

The shape of array, as a tuple.

static size(arr)

Parameters:

arr – an array.

Returns:

The total number of elements in arr.

static none()

Returns:

This backend’s representation of a missing value (nan for NumpyBackend and TorchBackend, None for ListBackend).

static inf()

Returns:

This backend’s representation of positive infinity.

static copy(array)

Parameters:

array – an array.

Returns:

An independent copy of array. For NumpyBackend/TorchBackend this is a shallow copy of the underlying numeric buffer, which is sufficient since they only ever hold regular numeric data. ListBackend performs a deep copy instead, since it can hold nested/ragged Python containers whose inner elements a shallow copy would still alias.

static squeeze(array, dim=None)

Parameters:

• array – an array;

• dim – dimension(s) to remove, must have size 1. If None, all size-1 dimensions are removed.

Returns:

array with the size-1 dimension(s) removed.

static expand_dims(array, dim)

Parameters:

• array – an array;

• dim – position where the new axis is inserted.

Returns:

array with a new size-1 axis inserted at position dim.

static atleast_2d(array)

Parameters:

array – an array.

Returns:

array reshaped so that it has at least two dimensions.

static repeat(array, repeats)

Parameters:

• array – an array;

• repeats – number of times each element is repeated.

Returns:

array with each element repeated repeats times.

static stack(lst, dim)

Parameters:

• lst – a list of arrays with the same shape;

• dim – dimension along which the arrays are stacked.

Returns:

The arrays in lst stacked along a new dimension dim.

static where(cond, x=None, y=None)

Parameters:

• cond – a boolean array/condition;

• x (None) – array of values to select where cond is True;

• y (None) – array of values to select where cond is False.

Returns:

If x and y are None, the indices where cond is True. Otherwise, an array with elements taken from x where cond is True and from y elsewhere.

static nonzero(array)

Parameters:

array – an array.

Returns:

The indices of the nonzero elements of array.

static abs(array)

Parameters:

array – an array.

Returns:

The element-wise absolute value of array.

static exp(array)

Parameters:

array – an array.

Returns:

The element-wise exponential of array.

static sqrt(array)

Parameters:

array – an array.

Returns:

The element-wise square root of array.

static clip(array, min, max)

Parameters:

• array – an array;

• min – lower bound;

• max – upper bound.

Returns:

array with values clipped to the [min, max] range.

static sum(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the sum is computed. If None, the sum over the whole array is returned.

Returns:

The sum of array along dim.

static median(array)

Parameters:

array – an array.

Returns:

The median of the elements of array.

static max(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the maximum is computed. If None, the maximum over the whole array is returned.

Returns:

The maximum value(s) of array along dim.

static min(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the minimum is computed. If None, the minimum over the whole array is returned.

Returns:

The minimum value(s) of array along dim.

static norm(array, ord=None, dim=None)

Parameters:

• array – an array;

• ord (None) – order of the norm (see numpy.linalg.norm/torch.linalg.norm for the accepted values). If None, the default order for the underlying library is used;

• dim (None) – dimension along which the norm is computed. If None, the norm of the flattened array is returned.

Returns:

The norm of array along dim.

static maximum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise maximum of x and y.

static minimum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise minimum of x and y.

static logical_and(x, y)

Parameters:

• x – a boolean array;

• y – a boolean array.

Returns:

The element-wise logical AND of x and y.

classmethod rand(*dims, device=None)

Parameters:

• *dims – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape dims sampled uniformly in [0, 1).

classmethod randint(low, high, size, device=None)

Parameters:

• low – lowest (inclusive) integer to be drawn;

• high – highest (exclusive) integer to be drawn;

• size – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape size with random integers in [low, high).

static multinomial(p)

Parameters:

p – a 1D array of (unnormalized) probabilities.

Returns:

A single index sampled according to the probabilities in p.

static uniform(low, high)

Parameters:

• low – lower bound(s) of the uniform distribution;

• high – upper bound(s) of the uniform distribution.

Returns:

An array sampled uniformly between low and high.

classmethod arange(start, stop, step=1, dtype=None, device=None)

Parameters:

• start – start of the interval;

• stop – end of the interval (exclusive);

• step (1) – spacing between values;

• dtype (None) – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array with evenly spaced values within the given interval.

class TorchBackend

Bases: ArrayBackend

Array backend storing data in PyTorch Tensor objects. It supports GPU execution and is the backend of choice for neural-network based agents and vectorized environments running on device.

static get_backend_name()

Returns:

The name of the backend ('numpy', 'torch' or 'list').

static get_backend_serialization()

Returns:

The name of the MushroomObject save method (see _add_save_attr) to use for attributes stored in this backend’s array type, i.e. 'numpy' or 'torch'. Backends with no dedicated save method (e.g. ListBackend) return the name of another backend able to serialize their data instead ('numpy').

static convert_to_backend(backend, array)

Convert a single array from another backend into this backend’s native array type. Unlike convert(), this is a static method called on the destination backend, taking the source backend as an explicit positional argument, e.g. TorchBackend.convert_to_backend(NumpyBackend, array) converts a NumPy array into a PyTorch Tensor.

Parameters:

• backend (ArrayBackend) – the backend array currently belongs to;

• array – the array to convert.

Returns:

array converted into this backend’s native format.

static to_numpy(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a NumPy ndarray.

static to_torch(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a PyTorch Tensor.

static to_list(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a plain Python list.

static as_array(array)

Cast array to this backend’s native array type without changing its backend, materializing it if needed (e.g. wrapping a plain Python list into a NumPy/PyTorch array).

Parameters:

array – an array-like object.

Returns:

array as a native object of this backend.

static from_list(array)

Build a backend array from a plain Python list (the inverse of to_list()).

Parameters:

array (list) – a plain Python list.

Returns:

array converted to this backend’s native array type.

classmethod to_backend_dtype(dtype)

Parameters:

dtype – a dtype specification, either native to this backend or to another supported backend (e.g. a NumPy dtype or a PyTorch dtype).

Returns:

dtype converted to this backend’s native dtype representation.

static empty(shape, device=None)

Parameters:

• shape – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new, uninitialized array of the given shape.

static full(shape, value)

Parameters:

• shape – shape of the array to create;

• value – fill value.

Returns:

A new array of the given shape, filled with value.

static concatenate(list_of_arrays, dim=0)

Parameters:

• list_of_arrays – a list of arrays to concatenate;

• dim – dimension along which the arrays are concatenated.

Returns:

The arrays in list_of_arrays concatenated along dim.

static flatten(array)

Merge the first two axes of an array into a single leading axis, grouping the result by the second axis and ordering it by the first axis within each group (i.e. all entries with index 0 along the second axis, in order of their index along the first axis, then all entries with index 1 along the second axis, …). This differs from a plain row-major reshape, which would order elements by the first axis first.

Parameters:

array – an array of shape (A, B, ...).

Returns:

array reshaped to (A * B, ...).

static pack_padded_sequence(array, mask)

Select the entries of an array marked by a boolean mask over its first two axes, and concatenate them into a single leading axis, grouped by the second axis and ordered by the first axis within each group (i.e. all selected entries with index 0 along the second axis, in order of their index along the first axis, then all selected entries with index 1 along the second axis, …).

Parameters:

• array – an array of shape (A, B, ...);

• mask – a boolean array of shape (A, B) marking the entries of array to keep.

Returns:

The entries of array selected by mask, concatenated in the order described above.

classmethod zeros(*dims, dtype=torch.float32, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with zeros.

classmethod ones(*dims, dtype=torch.float32, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with ones.

classmethod zeros_like(array, dtype=torch.float32, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with zeros.

classmethod ones_like(array, dtype=torch.float32, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with ones.

static masked_init(mask, values)

Build an array of shape (len(mask), *values.shape[1:]) where the entries selected by mask are filled, in order, with values, and the remaining entries are left uninitialized.

Parameters:

• mask – a boolean array of shape (N,);

• values – an array of shape (M, ...), with M equal to the number of True entries in mask.

Returns:

An array of shape (N, ...) with values scattered at the positions where mask is True. The scattered entries are independent of values (NumpyBackend/ TorchBackend copy the underlying numeric buffer; ListBackend deep-copies each scattered element, since it can hold nested/ragged Python containers).

static shape(array)

Parameters:

array – an array.

Returns:

The shape of array, as a tuple.

static size(arr)

Parameters:

arr – an array.

Returns:

The total number of elements in arr.

static none()

Returns:

This backend’s representation of a missing value (nan for NumpyBackend and TorchBackend, None for ListBackend).

static inf()

Returns:

This backend’s representation of positive infinity.

static copy(array)

Parameters:

array – an array.

Returns:

An independent copy of array. For NumpyBackend/TorchBackend this is a shallow copy of the underlying numeric buffer, which is sufficient since they only ever hold regular numeric data. ListBackend performs a deep copy instead, since it can hold nested/ragged Python containers whose inner elements a shallow copy would still alias.

static squeeze(array, dim=None)

Parameters:

• array – an array;

• dim – dimension(s) to remove, must have size 1. If None, all size-1 dimensions are removed.

Returns:

array with the size-1 dimension(s) removed.

static expand_dims(array, dim)

Parameters:

• array – an array;

• dim – position where the new axis is inserted.

Returns:

array with a new size-1 axis inserted at position dim.

static atleast_2d(array)

Parameters:

array – an array.

Returns:

array reshaped so that it has at least two dimensions.

static repeat(array, repeats)

Parameters:

• array – an array;

• repeats – number of times each element is repeated.

Returns:

array with each element repeated repeats times.

static stack(lst, dim)

Parameters:

• lst – a list of arrays with the same shape;

• dim – dimension along which the arrays are stacked.

Returns:

The arrays in lst stacked along a new dimension dim.

static where(cond, x=None, y=None)

Parameters:

• cond – a boolean array/condition;

• x (None) – array of values to select where cond is True;

• y (None) – array of values to select where cond is False.

Returns:

If x and y are None, the indices where cond is True. Otherwise, an array with elements taken from x where cond is True and from y elsewhere.

static nonzero(array)

Parameters:

array – an array.

Returns:

The indices of the nonzero elements of array.

static abs(array)

Parameters:

array – an array.

Returns:

The element-wise absolute value of array.

static exp(array)

Parameters:

array – an array.

Returns:

The element-wise exponential of array.

static sqrt(array)

Parameters:

array – an array.

Returns:

The element-wise square root of array.

static clip(array, min, max)

Parameters:

• array – an array;

• min – lower bound;

• max – upper bound.

Returns:

array with values clipped to the [min, max] range.

static sum(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the sum is computed. If None, the sum over the whole array is returned.

Returns:

The sum of array along dim.

static median(array)

Parameters:

array – an array.

Returns:

The median of the elements of array.

static max(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the maximum is computed. If None, the maximum over the whole array is returned.

Returns:

The maximum value(s) of array along dim.

static min(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the minimum is computed. If None, the minimum over the whole array is returned.

Returns:

The minimum value(s) of array along dim.

static norm(array, ord=None, dim=None)

Parameters:

• array – an array;

• ord (None) – order of the norm (see numpy.linalg.norm/torch.linalg.norm for the accepted values). If None, the default order for the underlying library is used;

• dim (None) – dimension along which the norm is computed. If None, the norm of the flattened array is returned.

Returns:

The norm of array along dim.

static maximum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise maximum of x and y.

static minimum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise minimum of x and y.

static logical_and(x, y)

Parameters:

• x – a boolean array;

• y – a boolean array.

Returns:

The element-wise logical AND of x and y.

static rand(*dims, device=None)

Parameters:

• *dims – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape dims sampled uniformly in [0, 1).

static randint(low, high, size, device=None)

Parameters:

• low – lowest (inclusive) integer to be drawn;

• high – highest (exclusive) integer to be drawn;

• size – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape size with random integers in [low, high).

static multinomial(p)

Parameters:

p – a 1D array of (unnormalized) probabilities.

Returns:

A single index sampled according to the probabilities in p.

static uniform(low, high)

Parameters:

• low – lower bound(s) of the uniform distribution;

• high – upper bound(s) of the uniform distribution.

Returns:

An array sampled uniformly between low and high.

classmethod arange(start, stop, step=1, dtype=None, device=None)

Parameters:

• start – start of the interval;

• stop – end of the interval (exclusive);

• step (1) – spacing between values;

• dtype (None) – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array with evenly spaced values within the given interval.

class ListBackend

Bases: ArrayBackend

Storage backend that keeps data in plain Python lists. It grows without pre-allocation (which allows collecting episodes of unbounded/infinite horizon) and can hold ragged or non-array observations and actions. It is numpy-serialized: container reshaping (empty, full, concatenate, flatten, pack_padded_sequence) is list-native and ragged-safe, while any numeric computation materializes the (regular) data to numpy via as_array.

static get_backend_name()

Returns:

The name of the backend ('numpy', 'torch' or 'list').

static get_backend_serialization()

Returns:

The name of the MushroomObject save method (see _add_save_attr) to use for attributes stored in this backend’s array type, i.e. 'numpy' or 'torch'. Backends with no dedicated save method (e.g. ListBackend) return the name of another backend able to serialize their data instead ('numpy').

static convert_to_backend(backend, array)

Convert a single array from another backend into this backend’s native array type. Unlike convert(), this is a static method called on the destination backend, taking the source backend as an explicit positional argument, e.g. TorchBackend.convert_to_backend(NumpyBackend, array) converts a NumPy array into a PyTorch Tensor.

Parameters:

• backend (ArrayBackend) – the backend array currently belongs to;

• array – the array to convert.

Returns:

array converted into this backend’s native format.

static to_numpy(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a NumPy ndarray.

static to_torch(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a PyTorch Tensor.

static to_list(array)

Parameters:

array – an array in this backend’s format.

Returns:

array converted to a plain Python list.

static as_array(array)

Cast array to this backend’s native array type without changing its backend, materializing it if needed (e.g. wrapping a plain Python list into a NumPy/PyTorch array).

Parameters:

array – an array-like object.

Returns:

array as a native object of this backend.

static from_list(array)

Build a backend array from a plain Python list (the inverse of to_list()).

Parameters:

array (list) – a plain Python list.

Returns:

array converted to this backend’s native array type.

static to_backend_dtype(dtype)

Parameters:

dtype – a dtype specification, either native to this backend or to another supported backend (e.g. a NumPy dtype or a PyTorch dtype).

Returns:

dtype converted to this backend’s native dtype representation.

static empty(shape, device=None)

Parameters:

• shape – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new, uninitialized array of the given shape.

static full(shape, value)

Parameters:

• shape – shape of the array to create;

• value – fill value.

Returns:

A new array of the given shape, filled with value.

static concatenate(list_of_arrays, dim=0)

Parameters:

• list_of_arrays – a list of arrays to concatenate;

• dim – dimension along which the arrays are concatenated.

Returns:

The arrays in list_of_arrays concatenated along dim.

static flatten(array)

Merge the first two axes of an array into a single leading axis, grouping the result by the second axis and ordering it by the first axis within each group (i.e. all entries with index 0 along the second axis, in order of their index along the first axis, then all entries with index 1 along the second axis, …). This differs from a plain row-major reshape, which would order elements by the first axis first.

Parameters:

array – an array of shape (A, B, ...).

Returns:

array reshaped to (A * B, ...).

static pack_padded_sequence(array, mask)

Select the entries of an array marked by a boolean mask over its first two axes, and concatenate them into a single leading axis, grouped by the second axis and ordered by the first axis within each group (i.e. all selected entries with index 0 along the second axis, in order of their index along the first axis, then all selected entries with index 1 along the second axis, …).

Parameters:

• array – an array of shape (A, B, ...);

• mask – a boolean array of shape (A, B) marking the entries of array to keep.

Returns:

The entries of array selected by mask, concatenated in the order described above.

classmethod zeros(*dims, dtype=<class 'float'>, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with zeros.

classmethod ones(*dims, dtype=<class 'float'>, device=None)

Parameters:

• *dims – shape of the array to create;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array of shape dims filled with ones.

classmethod zeros_like(array, dtype=<class 'float'>, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with zeros.

classmethod ones_like(array, dtype=<class 'float'>, device=None)

Parameters:

• array – array whose shape is used for the new array;

• dtype – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

A new array with the same shape as array, filled with ones.

static masked_init(mask, values)

Build an array of shape (len(mask), *values.shape[1:]) where the entries selected by mask are filled, in order, with values, and the remaining entries are left uninitialized.

Parameters:

• mask – a boolean array of shape (N,);

• values – an array of shape (M, ...), with M equal to the number of True entries in mask.

Returns:

An array of shape (N, ...) with values scattered at the positions where mask is True. The scattered entries are independent of values (NumpyBackend/ TorchBackend copy the underlying numeric buffer; ListBackend deep-copies each scattered element, since it can hold nested/ragged Python containers).

static shape(array)

Parameters:

array – an array.

Returns:

The shape of array, as a tuple.

static size(arr)

Parameters:

arr – an array.

Returns:

The total number of elements in arr.

static none()

Returns:

This backend’s representation of a missing value (nan for NumpyBackend and TorchBackend, None for ListBackend).

static inf()

Returns:

This backend’s representation of positive infinity.

static copy(array)

Parameters:

array – an array.

Returns:

An independent copy of array. For NumpyBackend/TorchBackend this is a shallow copy of the underlying numeric buffer, which is sufficient since they only ever hold regular numeric data. ListBackend performs a deep copy instead, since it can hold nested/ragged Python containers whose inner elements a shallow copy would still alias.

static squeeze(array, dim=None)

Parameters:

• array – an array;

• dim – dimension(s) to remove, must have size 1. If None, all size-1 dimensions are removed.

Returns:

array with the size-1 dimension(s) removed.

static expand_dims(array, dim)

Parameters:

• array – an array;

• dim – position where the new axis is inserted.

Returns:

array with a new size-1 axis inserted at position dim.

static atleast_2d(array)

Parameters:

array – an array.

Returns:

array reshaped so that it has at least two dimensions.

static repeat(array, repeats)

Parameters:

• array – an array;

• repeats – number of times each element is repeated.

Returns:

array with each element repeated repeats times.

static stack(lst, dim)

Parameters:

• lst – a list of arrays with the same shape;

• dim – dimension along which the arrays are stacked.

Returns:

The arrays in lst stacked along a new dimension dim.

static where(cond, x=None, y=None)

Parameters:

• cond – a boolean array/condition;

• x (None) – array of values to select where cond is True;

• y (None) – array of values to select where cond is False.

Returns:

If x and y are None, the indices where cond is True. Otherwise, an array with elements taken from x where cond is True and from y elsewhere.

static nonzero(array)

Parameters:

array – an array.

Returns:

The indices of the nonzero elements of array.

static abs(array)

Parameters:

array – an array.

Returns:

The element-wise absolute value of array.

static exp(array)

Parameters:

array – an array.

Returns:

The element-wise exponential of array.

static sqrt(array)

Parameters:

array – an array.

Returns:

The element-wise square root of array.

static clip(array, min, max)

Parameters:

• array – an array;

• min – lower bound;

• max – upper bound.

Returns:

array with values clipped to the [min, max] range.

static sum(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the sum is computed. If None, the sum over the whole array is returned.

Returns:

The sum of array along dim.

static median(array)

Parameters:

array – an array.

Returns:

The median of the elements of array.

static max(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the maximum is computed. If None, the maximum over the whole array is returned.

Returns:

The maximum value(s) of array along dim.

static min(array, dim=None)

Parameters:

• array – an array;

• dim (None) – dimension along which the minimum is computed. If None, the minimum over the whole array is returned.

Returns:

The minimum value(s) of array along dim.

static norm(array, ord=None, dim=None)

Parameters:

• array – an array;

• ord (None) – order of the norm (see numpy.linalg.norm/torch.linalg.norm for the accepted values). If None, the default order for the underlying library is used;

• dim (None) – dimension along which the norm is computed. If None, the norm of the flattened array is returned.

Returns:

The norm of array along dim.

static maximum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise maximum of x and y.

static minimum(x, y)

Parameters:

• x – an array;

• y – an array.

Returns:

The element-wise minimum of x and y.

static logical_and(x, y)

Parameters:

• x – a boolean array;

• y – a boolean array.

Returns:

The element-wise logical AND of x and y.

classmethod rand(*dims, device=None)

Parameters:

• *dims – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape dims sampled uniformly in [0, 1).

classmethod randint(low, high, size, device=None)

Parameters:

• low – lowest (inclusive) integer to be drawn;

• high – highest (exclusive) integer to be drawn;

• size – shape of the array to create;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array of shape size with random integers in [low, high).

static multinomial(p)

Parameters:

p – a 1D array of (unnormalized) probabilities.

Returns:

A single index sampled according to the probabilities in p.

static uniform(low, high)

Parameters:

• low – lower bound(s) of the uniform distribution;

• high – upper bound(s) of the uniform distribution.

Returns:

An array sampled uniformly between low and high.

classmethod arange(start, stop, step=1, dtype=None, device=None)

Parameters:

• start – start of the interval;

• stop – end of the interval (exclusive);

• step (1) – spacing between values;

• dtype (None) – data type of the array;

• device (None) – device the array should be allocated on. Only meaningful for TorchBackend.

Returns:

An array with evenly spaced values within the given interval.

Serialization

class MushroomObject(*args, **kwargs)

Bases: object

Interface to implement serialization and logging of a MushroomRL object.

It provides save and load functionality to store the object in a zip file: subclasses declare which attributes to persist with _add_save_attr, and full_save selects how much of the state is saved.

It also provides logging functionality: a logger is attached with set_logger and forwarded to the loggable children declared with _add_logger_attr, so that the relevant quantities of an object and of its sub-objects are logged under a hierarchy of metric names.

save(path, full_save=False)

Serialize and save the object to the given path on disk.

Parameters:

• path (Path, str) – Relative or absolute path to the object save location;

• full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.

save_zip(zip_file, full_save, folder='')

Serialize and save the agent to the given path on disk.

Parameters:

• zip_file (ZipFile) – ZipFile where te object needs to be saved;

• full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;

• folder (string, '') – subfolder to be used by the save method.

classmethod load(path)

Load and deserialize the agent from the given location on disk.

Parameters:

path (Path, string) – Relative or absolute path to the agents save location.

Returns:

The loaded agent.

set_logger(logger, prefix=None, label=None)

Attach a logger to the object so that its relevant quantities are logged. The prefix groups the logged metrics (e.g. critic produces critic/loss); the label overrides the default metric name of a single-value object. The logger is then forwarded to every loggable child registered with _add_logger_attr, with the child group joined to prefix.

Parameters:

• logger (Logger) – the logger to be used by the object;

• prefix (str, None) – optional group prepended to the logged metric names;

• label (str, None) – optional metric name override for single-value objects.

copy()

Returns:

A deepcopy of the agent.

_add_save_attr(**attr_dict)

Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the MushroomObject interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.

Parameters:

**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.

_add_logger_attr(*attrs, group=None, **labels)

Register loggable child attributes so that set_logger forwards the logger to each of them, all grouped under the optional group prefix. Attributes passed positionally use their default metric name (loss for an approximator, value for a parameter), while attributes passed as keywords map to an explicit metric name. For example _add_logger_attr('_V', group='critic') logs the approximator under critic/loss, and _add_logger_attr(_epsilon='epsilon', group='policy') logs the parameter under policy/epsilon. The registry is saved, so the forwarding keeps working on a loaded object.

Parameters:

• *attrs – attribute names that use their child default metric name;

• group (str, None) – optional group prefix shared by all the registered children;

• **labels – mapping from attribute name to its explicit metric label.

_post_load()

This method can be overwritten to implement logic that is executed after the loading of the agent.

Logger

class Logger(log_name='', results_dir='./logs', log_console=False, use_timestamp=False, append=False, seed=None, wandb_kwargs=None, force_numpy=False, recorder_class=None, fps=None, recorder_kwargs=None, **kwargs)

Bases: DataLogger, ConsoleLogger, VideoLogger, WandbLogger

This class implements the logging functionality. It can be used to create automatically a log directory, save numpy data array and the current agent. It optionally logs to Weights & Biases (wandb), if the wandb package is installed and a set of init arguments is provided.

__init__(log_name='', results_dir='./logs', log_console=False, use_timestamp=False, append=False, seed=None, wandb_kwargs=None, force_numpy=False, recorder_class=None, fps=None, recorder_kwargs=None, **kwargs)

Constructor.

Parameters:

• log_name (string, '') – name of the current experiment directory if not specified, the current timestamp is used.

• results_dir (string, './logs') – name of the base logging directory. If set to None, no directory is created;

• log_console (bool, False) – whether to log or not the console output;

• use_timestamp (bool, False) – If true, adds the current timestamp to the folder name;

• append (bool, False) – If true, the logger will append the new data logged to the one already existing in the directory;

• seed (int, None) – seed for the current run. It can be optionally specified to add a seed suffix for each data file logged. When wandb logging is active, the seed is added to the wandb config and, if name is not set, to the run name;

• wandb_kwargs (dict, None) – dictionary of arguments forwarded to wandb.init to enable wandb logging. If None, or if the wandb package is not installed, wandb logging is disabled. Use Logger.default_wandb_kwargs to build a default dictionary. If group is not set, it defaults to log_name so that all runs from the same experiment are grouped together;

• force_numpy (bool, False) – if True, the values logged through the log method are also stored on disk as numpy arrays (only if a results directory is set);

• recorder_class (class, None) – the class used to record video. By default, the VideoRecorder class is used. The class must implement the __call__ and stop methods;

• fps (int, None) – frames per second for video recording. If None, the value is set automatically by Core.set_logger from the environment;

• recorder_kwargs (dict, None) – additional keyword arguments forwarded to the recorder class constructor;

• **kwargs – other parameters for ConsoleLogger class.

log_training(prefix=None, **kwargs)

Log a set of named training metrics. The values are logged to wandb under the training/ group (if active), to the console with the debug level (so they are not shown by default), and to disk as numpy arrays inside the training subfolder only if the logger was constructed with force_numpy=True.

An optional prefix groups the metrics (e.g. prefix='critic', loss=... becomes critic/loss); a '/' in the resulting name groups the metric in wandb and is replaced by '_' for the numpy file name (e.g. critic_loss.npy).

Parameters:

• prefix (str, None) – optional group prepended to each metric name;

• **kwargs – set of named values to be logged.

log_evaluation(epoch, **kwargs)

Log a set of named evaluation metrics. The values are logged to wandb under the eval/ group using the epoch as x-axis (if active), to the console through epoch_info, and to disk as numpy arrays in the logging directory.

Parameters:

• epoch (int) – the current epoch;

• **kwargs – set of named values to be logged.

log_video(epoch, video=None, wandb_name='evaluation')

If wandb logging is active, upload a video to wandb under the video/ group using the epoch as x-axis. The recording itself is stopped by Core; this method only handles the wandb upload. The video is uploaded as is, without any re-encoding. The same wandb_name should be used across epochs so that wandb shows a slider to browse them.

Parameters:

• epoch (int) – the current epoch, used as x-axis for the video;

• video (str, Path, None) – path of the video file to upload. If None, the last recorded video is used;

• wandb_name (str, 'evaluation') – the wandb key name for the video. Must be consistent across epochs for the slider to work.

class ConsoleLogger(log_name, log_dir=None, suffix='', log_file_name=None, console_log_level=20, file_log_level=10)

Bases: object

This class implements the console logging functionality. It can be used to log text into the console and optionally save a log file.

__init__(log_name, log_dir=None, suffix='', log_file_name=None, console_log_level=20, file_log_level=10)

Constructor.

Parameters:

• log_name (str, None) – Name of the current logger.

• log_dir (Path, None) – path of the logging directory. If None, no the console output is not logged into a file;

• suffix (int, None) – optional string to add a suffix to the logger id and to the data file logged;

• log_file_name (str, None) – optional specifier for log file name, id is used by default;

• console_log_level (int, logging.INFO) – logging level for console;

• file_log_level (int, logging.DEBUG) – logging level for file.

debug(msg)

Log a message with DEBUG level

info(msg)

Log a message with INFO level

warning(msg)

Log a message with WARNING level

error(msg)

Log a message with ERROR level

critical(msg)

Log a message with CRITICAL level

exception(msg)

Log a message with ERROR level. To be called only from an exception handler

strong_line()

Log a line of #

weak_line()

Log a line of -

epoch_info(epoch, **kwargs)

Log the epoch info with the format: Epoch <epoch number> | <label 1>: <data 1> <label 2> <data 2> …

Parameters:

• epoch (int) – epoch number;

• **kwargs – the labels and the data to be displayed.

class DataLogger(results_dir, suffix='', append=False)

Bases: object

This class implements the data logging functionality. It can be used to create automatically a log directory, save numpy data array and the current agent.

__init__(results_dir, suffix='', append=False)

Constructor.

Parameters:

• results_dir (Path) – path of the logging directory;

• suffix (string) – optional string to add a suffix to each data file logged;

• append (bool, False) – If true, the logger will append the new data logged to the one already existing in the directory.

log_numpy(folder='', **kwargs)

Log scalars into numpy arrays.

Parameters:

• folder (str, '') – optional subfolder of the logging directory where the arrays are stored. If empty, they are stored in the logging directory;

• **kwargs – set of named scalar values to be saved. The argument name will be used to identify the given quantity and as base file name.

_get_folder(folder='')

Return the path of the given subfolder of the logging directory, creating it if it does not exist yet. If folder is empty, the logging directory itself is returned.

Parameters:

folder (str, '') – name of the subfolder.

Returns:

The path of the (sub)folder.

log_numpy_array(**kwargs)

Log numpy arrays.

Parameters:

**kwargs – set of named arrays to be saved. The argument name will be used to identify the given quantity and as base file name.

log_agent(agent, epoch=None, full_save=False)

Log agent into the log folder.

Parameters:

• agent (Agent) – The agent to be saved;

• epoch (int, None) – optional epoch number to be added to the agent file currently saved;

• full_save (bool, False) – whether to save the full data from the agent or not.

log_best_agent(agent, J, full_save=False)

Log the best agent so far into the log folder. The agent is logged only if the current performance is better than the performance of the previously stored agent.

Parameters:

• agent (Agent) – The agent to be saved;

• J (float) – The performance metric of the current agent;

• full_save (bool, False) – whether to save the full data from the agent or not.

property path

Property to return the path to the current logging directory

class VideoLogger(recorder_class=None, fps=None, video_path=None, append=False, **recorder_kwargs)

Bases: object

This class implements the video recording functionality for the Logger. The recorder is created lazily on the first call to record_frame.

__init__(recorder_class=None, fps=None, video_path=None, append=False, **recorder_kwargs)

Constructor.

Parameters:

• recorder_class (class, None) – the class used to record the video. By default, the VideoRecorder class is used. The class must implement the __call__ and stop methods;

• fps (int, None) – frames per second for the video. If None, the default of the recorder class is used;

• video_path (Path, None) – path where videos are stored. If None, videos go to the default location of the recorder class;

• append (bool, False) – if True, the videos already present in video_path are loaded into the recorded videos list at construction;

• **recorder_kwargs – additional keyword arguments forwarded to the recorder class constructor.

record_frame(frame)

Record a single frame. The recorder is created lazily on the first call.

Parameters:

frame – the frame to record (np.ndarray, H x W x RGB).

stop_recording()

Stop the current recording. The next call to record_frame will start a new recording.

Returns:

The path of the recorded video file, or None if nothing was recorded.

set_video_fps(fps)

Set the frames per second for video recording, only if not already configured.

Parameters:

fps (int) – frames per second.

property video_recorder

Access to the underlying recorder instance. Returns None if no frame has been recorded yet.

property recorded_videos

Returns: The list of paths of the videos recorded (and loaded) so far.

class WandbLogger(wandb_kwargs=None, results_dir=None, log_dir=None, append=False)

Bases: object

This class implements the wandb logging functionality. It is enabled only if the wandb package is installed and a set of init arguments is provided, otherwise every method is a no-op.

__init__(wandb_kwargs=None, results_dir=None, log_dir=None, append=False)

Constructor.

Parameters:

• wandb_kwargs (dict, None) – dictionary of arguments forwarded to wandb.init. If None, or if the wandb package is not installed, wandb logging is disabled and all methods are no-ops;

• results_dir (Path, None) – logging directory created by the Logger. If provided, and dir is not already set in wandb_kwargs, wandb stores its files inside this directory;

• log_dir (Path, None) – experiment-specific directory where the wandb run id is persisted for resume on append;

• append (bool, False) – if True and a previous run id is found in log_dir, the wandb run is resumed with the same run id and the fit counter is restored from the wandb run summary.

static default_wandb_kwargs(project, config=None, **overrides)

Build a default dictionary of arguments for wandb.init. The returned dictionary can be freely edited and is meant to be passed to the Logger constructor through the wandb_kwargs argument.

Parameters:

• project (str) – name of the wandb project;

• config (dict, None) – dictionary of hyperparameters to log;

• **overrides – any additional key overrides the defaults.

Returns:

The dictionary of arguments for wandb.init.

log_wandb_training(prefix=None, **kwargs)

Log a set of named training metrics to wandb, grouped under the training/ prefix and using the number of fits as x-axis. An optional prefix adds an intermediate group, so that a value loss logged with prefix='critic' becomes training/critic/loss.

Parameters:

• prefix (str, None) – optional group prepended to each metric name;

• **kwargs – set of named values to be logged.

log_wandb_eval(epoch, **kwargs)

Log a set of named evaluation metrics to wandb, grouped under the eval/ prefix and using the epoch as x-axis.

Parameters:

• epoch (int) – the current epoch, used as x-axis for the logged values;

• **kwargs – set of named values to be logged.

log_wandb_video(name, path, epoch)

Log a video file to wandb, grouped under the video/ prefix and using the epoch as x-axis. The video is uploaded as is, without any re-encoding.

Parameters:

• name (str) – the name of the video;

• path (str, Path) – the path to the video file to upload;

• epoch (int) – the current epoch, used as x-axis for the video.

advance_step()

Advance the number of fits counter by one. To be called once per fit, so that all the training values logged during a fit share the same n_fit x-axis value.

finish()

Finish the current wandb run, flushing the data to disk. No-op if wandb logging is not active.

property wandb_active

Returns: True if wandb logging is enabled, False otherwise.