Environments

In mushroom_rl we distinguish between two different types of environment classes:

• proper environments
• generators

While environments directly implement the Environment interface, generators are a set of methods used to generate finite markov chains that represent a specific environment e.g., grid worlds.

Environments

Atari

class mushroom_rl.environments.atari.MaxAndSkip(env, skip, max_pooling=True)

Bases: gym.core.Wrapper

__init__(env, skip, max_pooling=True)

Initialize self. See help(type(self)) for accurate signature.

step(action)

Run one timestep of the environment’s dynamics. When end of episode is reached, you are responsible for calling reset() to reset this environment’s state.

Accepts an action and returns a tuple (observation, reward, done, info).

Parameters:action (object) – an action provided by the agent Returns:agent’s observation of the current environment reward (float) : amount of reward returned after previous action done (bool): whether the episode has ended, in which case further step() calls will return undefined results info (dict): contains auxiliary diagnostic information (helpful for debugging, and sometimes learning) Return type:observation (object)

reset(**kwargs)

Resets the state of the environment and returns an initial observation.

Returns:the initial observation. Return type:observation (object)

close()

Override close in your subclass to perform any necessary cleanup.

Environments will automatically close() themselves when garbage collected or when the program exits.

render(mode='human', **kwargs)

Renders the environment.

The set of supported modes varies per environment. (And some environments do not support rendering at all.) By convention, if mode is:

• human: render to the current display or terminal and return nothing. Usually for human consumption.
• rgb_array: Return an numpy.ndarray with shape (x, y, 3), representing RGB values for an x-by-y pixel image, suitable for turning into a video.
• ansi: Return a string (str) or StringIO.StringIO containing a terminal-style text representation. The text can include newlines and ANSI escape sequences (e.g. for colors).

Note

Make sure that your class’s metadata ‘render.modes’ key includes

the list of supported modes. It’s recommended to call super() in implementations to use the functionality of this method.

Parameters:mode (str) – the mode to render with

Example:

class MyEnv(Env):

metadata = {‘render.modes’: [‘human’, ‘rgb_array’]}

def render(self, mode=’human’):

if mode == ‘rgb_array’:

return np.array(…) # return RGB frame suitable for video

elif mode == ‘human’:

… # pop up a window and render

else:

super(MyEnv, self).render(mode=mode) # just raise an exception

seed(seed=None)

Sets the seed for this env’s random number generator(s).

Note

Some environments use multiple pseudorandom number generators. We want to capture all such seeds used in order to ensure that there aren’t accidental correlations between multiple generators.

Returns:

Returns the list of seeds used in this env’s random

number generators. The first value in the list should be the “main” seed, or the value which a reproducer should pass to ‘seed’. Often, the main seed equals the provided ‘seed’, but this won’t be true if seed=None, for example.

Return type:list<bigint>

unwrapped

Completely unwrap this env.

Returns:The base non-wrapped gym.Env instance Return type:gym.Env

class mushroom_rl.environments.atari.LazyFrames(frames, history_length)

Bases: object

From OpenAI Baseline. https://github.com/openai/baselines/blob/master/baselines/common/atari_wrappers.py

__init__(frames, history_length)

Initialize self. See help(type(self)) for accurate signature.

class mushroom_rl.environments.atari.Atari(name, width=84, height=84, ends_at_life=False, max_pooling=True, history_length=4, max_no_op_actions=30)

Bases: mushroom_rl.environments.environment.Environment

The Atari environment as presented in: “Human-level control through deep reinforcement learning”. Mnih et. al.. 2015.

__init__(name, width=84, height=84, ends_at_life=False, max_pooling=True, history_length=4, max_no_op_actions=30)

Constructor.

Parameters:

• name (str) – id name of the Atari game in Gym;
• width (int, 84) – width of the screen;
• height (int, 84) – height of the screen;
• ends_at_life (bool, False) – whether the episode ends when a life is lost or not;
• max_pooling (bool, True) – whether to do max-pooling or average-pooling of the last two frames when using NoFrameskip;
• history_length (int, 4) – number of frames to form a state;
• max_no_op_actions (int, 30) – maximum number of no-op action to execute at the beginning of an episode.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

set_episode_end(ends_at_life)

Setter.

Parameters:ends_at_life (bool) – whether the episode ends when a life is lost or not.

Car on hill

class mushroom_rl.environments.car_on_hill.CarOnHill(horizon=100, gamma=0.95)

Bases: mushroom_rl.environments.environment.Environment

The Car On Hill environment as presented in: “Tree-Based Batch Mode Reinforcement Learning”. Ernst D. et al.. 2005.

__init__(horizon=100, gamma=0.95)

Constructor.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

DeepMind Control Suite

class mushroom_rl.environments.dm_control_env.DMControl(domain_name, task_name, horizon, gamma, task_kwargs=None, dt=0.01, width_screen=480, height_screen=480, camera_id=0)

Bases: mushroom_rl.environments.environment.Environment

Interface for dm_control suite Mujoco environments. It makes it possible to use every dm_control suite Mujoco environment just providing the necessary information.

__init__(domain_name, task_name, horizon, gamma, task_kwargs=None, dt=0.01, width_screen=480, height_screen=480, camera_id=0)

Constructor.

Parameters:

• domain_name (str) – name of the environment;
• task_name (str) – name of the task of the environment;
• horizon (int) – the horizon;
• gamma (float) – the discount factor;
• task_kwargs (dict, None) – parameters of the task;
• dt (float, 01) – duration of a control step;
• width_screen (int, 480) – width of the screen;
• height_screen (int, 480) – height of the screen;
• camera_id (int, 0) – position of camera to render the environment;

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

Finite MDP

class mushroom_rl.environments.finite_mdp.FiniteMDP(p, rew, mu=None, gamma=0.9, horizon=inf)

Bases: mushroom_rl.environments.environment.Environment

Finite Markov Decision Process.

__init__(p, rew, mu=None, gamma=0.9, horizon=inf)

Constructor.

Parameters:

• p (np.ndarray) – transition probability matrix;
• rew (np.ndarray) – reward matrix;
• mu (np.ndarray, None) – initial state probability distribution;
• gamma (float, 9) – discount factor;
• horizon (int, np.inf) – the horizon.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

Grid World

class mushroom_rl.environments.grid_world.AbstractGridWorld(mdp_info, height, width, start, goal)

Bases: mushroom_rl.environments.environment.Environment

Abstract class to build a grid world.

__init__(mdp_info, height, width, start, goal)

Constructor.

Parameters:

• height (int) – height of the grid;
• width (int) – width of the grid;
• start (tuple) – x-y coordinates of the goal;
• goal (tuple) – x-y coordinates of the goal.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

class mushroom_rl.environments.grid_world.GridWorld(height, width, goal, start=(0, 0))

Bases: mushroom_rl.environments.grid_world.AbstractGridWorld

Standard grid world.

__init__(height, width, goal, start=(0, 0))

Constructor.

Parameters:

• height (int) – height of the grid;
• width (int) – width of the grid;
• start (tuple) – x-y coordinates of the goal;
• goal (tuple) – x-y coordinates of the goal.

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

class mushroom_rl.environments.grid_world.GridWorldVanHasselt(height=3, width=3, goal=(0, 2), start=(2, 0))

Bases: mushroom_rl.environments.grid_world.AbstractGridWorld

A variant of the grid world as presented in: “Double Q-Learning”. Hasselt H. V.. 2010.

__init__(height=3, width=3, goal=(0, 2), start=(2, 0))

Constructor.

Parameters:

• height (int) – height of the grid;
• width (int) – width of the grid;
• start (tuple) – x-y coordinates of the goal;
• goal (tuple) – x-y coordinates of the goal.

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

Gym

class mushroom_rl.environments.gym_env.Gym(name, horizon, gamma)

Bases: mushroom_rl.environments.environment.Environment

Interface for OpenAI Gym environments. It makes it possible to use every Gym environment just providing the id, except for the Atari games that are managed in a separate class.

__init__(name, horizon, gamma)

Constructor.

Parameters:

• name (str) – gym id of the environment;
• horizon (int) – the horizon;
• gamma (float) – the discount factor.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

Inverted pendulum

class mushroom_rl.environments.inverted_pendulum.InvertedPendulum(random_start=False, m=1.0, l=1.0, g=9.8, mu=0.01, max_u=5.0, horizon=5000, gamma=0.99)

Bases: mushroom_rl.environments.environment.Environment

The Inverted Pendulum environment (continuous version) as presented in: “Reinforcement Learning In Continuous Time and Space”. Doya K.. 2000. “Off-Policy Actor-Critic”. Degris T. et al.. 2012. “Deterministic Policy Gradient Algorithms”. Silver D. et al. 2014.

__init__(random_start=False, m=1.0, l=1.0, g=9.8, mu=0.01, max_u=5.0, horizon=5000, gamma=0.99)

Constructor.

Parameters:

• random_start (bool, False) – whether to start from a random position or from the horizontal one;
• m (float, 1.0) – mass of the pendulum;
• l (float, 1.0) – length of the pendulum;
• g (float, 9.8) – gravity acceleration constant;
• mu (float, 1e-2) – friction constant of the pendulum;
• max_u (float, 5.0) – maximum allowed input torque;
• horizon (int, 5000) – horizon of the problem;
• gamma (int, 99) – discount factor.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

Cart Pole

class mushroom_rl.environments.cart_pole.CartPole(m=2.0, M=8.0, l=0.5, g=9.8, mu=0.01, max_u=50.0, noise_u=10.0, horizon=3000, gamma=0.95)

Bases: mushroom_rl.environments.environment.Environment

The Inverted Pendulum on a Cart environment as presented in: “Least-Squares Policy Iteration”. Lagoudakis M. G. and Parr R.. 2003.

__init__(m=2.0, M=8.0, l=0.5, g=9.8, mu=0.01, max_u=50.0, noise_u=10.0, horizon=3000, gamma=0.95)

Constructor.

Parameters:

• m (float, 2.0) – mass of the pendulum;
• M (float, 8.0) – mass of the cart;
• l (float, 5) – length of the pendulum;
• g (float, 9.8) – gravity acceleration constant;
• mu (float, 1e-2) – friction constant of the pendulum;
• max_u (float, 50.) – maximum allowed input torque;
• noise_u (float, 10.) – maximum noise on the action;
• horizon (int, 3000) – horizon of the problem;
• gamma (int, 95) – discount factor.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

LQR

class mushroom_rl.environments.lqr.LQR(A, B, Q, R, max_pos=inf, max_action=inf, random_init=False, episodic=False, gamma=0.9, horizon=50)

Bases: mushroom_rl.environments.environment.Environment

This class implements a Linear-Quadratic Regulator. This task aims to minimize the undesired deviations from nominal values of some controller settings in control problems. The system equations in this task are:

\[x_{t+1} = Ax_t + Bu_t\]

where x is the state and u is the control signal.

The reward function is given by:

\[r_t = -\left( x_t^TQx_t + u_t^TRu_t \right)\]

“Policy gradient approaches for multi-objective sequential decision making”. Parisi S., Pirotta M., Smacchia N., Bascetta L., Restelli M.. 2014

__init__(A, B, Q, R, max_pos=inf, max_action=inf, random_init=False, episodic=False, gamma=0.9, horizon=50)

Constructor.

Args:

A (np.ndarray): the state dynamics matrix; B (np.ndarray): the action dynamics matrix; Q (np.ndarray): reward weight matrix for state; R (np.ndarray): reward weight matrix for action; max_pos (float, np.inf): maximum value of the state; max_action (float, np.inf): maximum value of the action; random_init (bool, False): start from a random state; episodic (bool, False): end the episode when the state goes over the threshold; gamma (float, 0.9): discount factor; horizon (int, 50): horizon of the mdp.

static generate(dimensions, max_pos=inf, max_action=inf, eps=0.1, index=0, random_init=False, episodic=False, gamma=0.9, horizon=50)

Factory method that generates an lqr with identity dynamics and symmetric reward matrices.

Parameters:

• dimensions (int) – number of state-action dimensions;
• max_pos (float, np.inf) – maximum value of the state;
• max_action (float, np.inf) – maximum value of the action;
• eps (double, 1) – reward matrix weights specifier;
• index (int, 0) – selector for the principal state;
• random_init (bool, False) – start from a random state;
• episodic (bool, False) – end the episode when the state goes over the threshold;
• gamma (float, 9) – discount factor;
• horizon (int, 50) – horizon of the mdp.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

Mujoco

class mushroom_rl.environments.mujoco.ObservationType

Bases: enum.Enum

An enum indicating the type of data that should be added to the observation of the environment, can be Joint-/Body-/Site- positions and velocities.

class mushroom_rl.environments.mujoco.MuJoCo(file_name, actuation_spec, observation_spec, gamma, horizon, n_substeps=1, n_intermediate_steps=1, additional_data_spec=None, collision_groups=None)

Bases: mushroom_rl.environments.environment.Environment

Class to create a Mushroom environment using the MuJoCo simulator.

__init__(file_name, actuation_spec, observation_spec, gamma, horizon, n_substeps=1, n_intermediate_steps=1, additional_data_spec=None, collision_groups=None)

Constructor.

Parameters:

• file_name (string) – The path to the XML file with which the environment should be created;
• actuation_spec (list) – A list specifying the names of the joints which should be controllable by the agent. Can be left empty when all actuators should be used;
• observation_spec (list) – A list containing the names of data that should be made available to the agent as an observation and their type (ObservationType). An entry in the list is given by: (name, type);
• gamma (float) – The discounting factor of the environment;
• horizon (int) – The maximum horizon for the environment;
• n_substeps (int) – The number of substeps to use by the MuJoCo simulator. An action given by the agent will be applied for n_substeps before the agent receives the next observation and can act accordingly;
• n_intermediate_steps (int) – The number of steps between every action taken by the agent. Similar to n_substeps but allows the user to modify, control and access intermediate states.
• additional_data_spec (list) – A list containing the data fields of interest, which should be read from or written to during simulation. The entries are given as the following tuples: (key, name, type) key is a string for later referencing in the “read_data” and “write_data” methods. The name is the name of the object in the XML specification and the type is the ObservationType;
• collision_groups (list) – A list containing groups of geoms for which collisions should be checked during simulation via check_collision. The entries are given as: (key, geom_names), where key is a string for later referencing in the “check_collision” method, and geom_names is a list of geom names in the XML specification.

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

_preprocess_action(action)

Compute a transformation of the action provided to the environment.

Parameters:action (np.ndarray) – numpy array with the actions provided to the environment. Returns:The action to be used for the current step

_step_init(state, action)

Allows information to be initialized at the start of a step.

_compute_action(action)

Compute a transformation of the action at every intermediate step. Useful to add control signals simulated directly in python.

Parameters:action (np.ndarray) – numpy array with the actions provided at every step. Returns:The action to be set in the actual mujoco simulation.

_simulation_pre_step()

Allows information to be accesed and changed at every intermediate step

before taking a step in the mujoco simulation. Can be usefull to apply an external force/torque to the specified bodies.

ex: apply a force over X to the torso:

force = [200, 0, 0] torque = [0, 0, 0] self.sim.data.xfrc_applied[self.sim.model._body_name2id[“torso”],:] = force + torque

_simulation_post_step()

Allows information to be accesed at every intermediate step

after taking a step in the mujoco simulation. Can be usefull to average forces over all intermediate steps.

_step_finalize()

Allows information to be accesed at the end of a step.

read_data(name)

Read data form the MuJoCo data structure.

Parameters:name (string) – A name referring to an entry contained the additional_data_spec list handed to the constructor. Returns:The desired data as a one-dimensional numpy array.

write_data(name, value)

Write data to the MuJoCo data structure.

Parameters:

• name (string) – A name referring to an entry contained in the additional_data_spec list handed to the constructor;
• value (ndarray) – The data that should be written.

check_collision(group1, group2)

Check for collision between the specified groups.

Parameters:

• group1 (string) – A name referring to an entry contained in the collision_groups list handed to the constructor;
• group2 (string) – A name referring to an entry contained in the collision_groups list handed to the constructor.

Returns:

A flag indicating whether a collision occurred between the given groups or not.

get_collision_force(group1, group2)

Returns the collision force and torques between the specified groups.

Parameters:

• group1 (string) – A name referring to an entry contained in the collision_groups list handed to the constructor;
• group2 (string) – A name referring to an entry contained in the collision_groups list handed to the constructor.

Returns:

A 6D vector specifying the collision forces/torques[3D force + 3D torque] between the given groups. Vector of 0’s in case there was no collision. http://mujoco.org/book/programming.html#siContact

reward(state, action, next_state)

Compute the reward based on the given transition.

Parameters:

• state (np.array) – the current state of the system;
• action (np.array) – the action that is applied in the current state;
• next_state (np.array) – the state reached after applying the given action.

Returns:

The reward as a floating point scalar value.

is_absorbing(state)

Check whether the given state is an absorbing state or not.

Parameters:state (np.array) – the state of the system. Returns:A boolean flag indicating whether this state is absorbing or not.

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

setup()

A function that allows to execute setup code after an environment reset.

Puddle World

class mushroom_rl.environments.puddle_world.PuddleWorld(start=None, goal=None, goal_threshold=0.1, noise_step=0.025, noise_reward=0, reward_goal=0.0, thrust=0.05, puddle_center=None, puddle_width=None, gamma=0.99, horizon=5000)

Bases: mushroom_rl.environments.environment.Environment

Puddle world as presented in: “Off-Policy Actor-Critic”. Degris T. et al.. 2012.

__init__(start=None, goal=None, goal_threshold=0.1, noise_step=0.025, noise_reward=0, reward_goal=0.0, thrust=0.05, puddle_center=None, puddle_width=None, gamma=0.99, horizon=5000)

Constructor.

Parameters:

• start (np.array, None) – starting position of the agent;
• goal (np.array, None) – goal position;
• goal_threshold (float, 1) – distance threshold of the agent from the goal to consider it reached;
• noise_step (float, 025) – noise in actions;
• noise_reward (float, 0) – standard deviation of gaussian noise in reward;
• reward_goal (float, 0) – reward obtained reaching goal state;
• thrust (float, 05) – distance walked during each action;
• puddle_center (np.array, None) – center of the puddle;
• puddle_width (np.array, None) – width of the puddle;

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

Segway

class mushroom_rl.environments.segway.Segway(random_start=False)

Bases: mushroom_rl.environments.environment.Environment

The Segway environment (continuous version) as presented in: “Deep Learning for Actor-Critic Reinforcement Learning”. Xueli Jia. 2015.

__init__(random_start=False)

Constructor.

Parameters:random_start (bool, False) – whether to start from a random position or from the horizontal one.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

Ship steering

class mushroom_rl.environments.ship_steering.ShipSteering(small=True, n_steps_action=3)

Bases: mushroom_rl.environments.environment.Environment

The Ship Steering environment as presented in: “Hierarchical Policy Gradient Algorithms”. Ghavamzadeh M. and Mahadevan S.. 2013.

__init__(small=True, n_steps_action=3)

Constructor.

Parameters:

• small (bool, True) – whether to use a small state space or not.
• n_steps_action (int, 3) – number of integration intervals for each step of the mdp.

reset(state=None)

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state. Returns:The current state.

step(action)

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute. Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).

stop()

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

static _bound(x, min_value, max_value)

Method used to bound state and action variables.

Parameters:

• x – the variable to bound;
• min_value – the minimum value;
• max_value – the maximum value;

Returns:

The bounded variable.

info

An object containing the info of the environment.

Type:Returns

seed(seed)

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.

Generators

Grid world

mushroom_rl.environments.generators.grid_world.generate_grid_world(grid, prob, pos_rew, neg_rew, gamma=0.9, horizon=100)

This Grid World generator requires a .txt file to specify the shape of the grid world and the cells. There are five types of cells: ‘S’ is the starting position where the agent is; ‘G’ is the goal state; ‘.’ is a normal cell; ‘*’ is a hole, when the agent steps on a hole, it receives a negative reward and the episode ends; ‘#’ is a wall, when the agent is supposed to step on a wall, it actually remains in its current state. The initial states distribution is uniform among all the initial states provided.

The grid is expected to be rectangular.

Parameters:

• grid (str) – the path of the file containing the grid structure;
• prob (float) – probability of success of an action;
• pos_rew (float) – reward obtained in goal states;
• neg_rew (float) – reward obtained in “hole” states;
• gamma (float, 9) – discount factor;
• horizon (int, 100) – the horizon.

Returns:

A FiniteMDP object built with the provided parameters.

mushroom_rl.environments.generators.grid_world.parse_grid(grid)

Parse the grid file:

Parameters:grid (str) – the path of the file containing the grid structure; Returns:A list containing the grid structure.

mushroom_rl.environments.generators.grid_world.compute_probabilities(grid_map, cell_list, prob)

Compute the transition probability matrix.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells;
• prob (float) – probability of success of an action.

Returns:

The transition probability matrix;

mushroom_rl.environments.generators.grid_world.compute_reward(grid_map, cell_list, pos_rew, neg_rew)

Compute the reward matrix.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells;
• pos_rew (float) – reward obtained in goal states;
• neg_rew (float) – reward obtained in “hole” states;

Returns:

The reward matrix.

mushroom_rl.environments.generators.grid_world.compute_mu(grid_map, cell_list)

Compute the initial states distribution.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells.

Returns:

The initial states distribution.

Simple chain

mushroom_rl.environments.generators.simple_chain.generate_simple_chain(state_n, goal_states, prob, rew, mu=None, gamma=0.9, horizon=100)

Simple chain generator.

Parameters:

• state_n (int) – number of states;
• goal_states (list) – list of goal states;
• prob (float) – probability of success of an action;
• rew (float) – reward obtained in goal states;
• mu (np.ndarray) – initial state probability distribution;
• gamma (float, 9) – discount factor;
• horizon (int, 100) – the horizon.

Returns:

A FiniteMDP object built with the provided parameters.

mushroom_rl.environments.generators.simple_chain.compute_probabilities(state_n, prob)

Compute the transition probability matrix.

Parameters:

• state_n (int) – number of states;
• prob (float) – probability of success of an action.

Returns:

The transition probability matrix;

mushroom_rl.environments.generators.simple_chain.compute_reward(state_n, goal_states, rew)

Compute the reward matrix.

Parameters:

• state_n (int) – number of states;
• goal_states (list) – list of goal states;
• rew (float) – reward obtained in goal states.

Returns:

The reward matrix.

Taxi

mushroom_rl.environments.generators.taxi.generate_taxi(grid, prob=0.9, rew=(0, 1, 3, 15), gamma=0.99, horizon=inf)

This Taxi generator requires a .txt file to specify the shape of the grid world and the cells. There are five types of cells: ‘S’ is the starting where the agent is; ‘G’ is the goal state; ‘.’ is a normal cell; ‘F’ is a passenger, when the agent steps on a hole, it picks up it. ‘#’ is a wall, when the agent is supposed to step on a wall, it actually remains in its current state. The initial states distribution is uniform among all the initial states provided. The episode terminates when the agent reaches the goal state. The reward is always 0, except for the goal state where it depends on the number of collected passengers. Each action has a certain probability of success and, if it fails, the agent goes in a perpendicular direction from the supposed one.

The grid is expected to be rectangular.

This problem is inspired from: “Bayesian Q-Learning”. Dearden R. et al.. 1998.

Parameters:

• grid (str) – the path of the file containing the grid structure;
• prob (float, 9) – probability of success of an action;
• rew (tuple, (0, 1, 3, 15)) – rewards obtained in goal states;
• gamma (float, 99) – discount factor;
• horizon (int, np.inf) – the horizon.

Returns:

A FiniteMDP object built with the provided parameters.

mushroom_rl.environments.generators.taxi.parse_grid(grid)

Parse the grid file:

Parameters:grid (str) – the path of the file containing the grid structure. Returns:A list containing the grid structure.

mushroom_rl.environments.generators.taxi.compute_probabilities(grid_map, cell_list, passenger_list, prob)

Compute the transition probability matrix.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells;
• passenger_list (list) – list of passenger cells;
• prob (float) – probability of success of an action.

Returns:

The transition probability matrix;

mushroom_rl.environments.generators.taxi.compute_reward(grid_map, cell_list, passenger_list, rew)

Compute the reward matrix.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells;
• passenger_list (list) – list of passenger cells;
• rew (tuple) – rewards obtained in goal states.

Returns:

The reward matrix.

mushroom_rl.environments.generators.taxi.compute_mu(grid_map, cell_list, passenger_list)

Compute the initial states distribution.

Parameters:

• grid_map (list) – list containing the grid structure;
• cell_list (list) – list of non-wall cells;
• passenger_list (list) – list of passenger cells.

Returns:

The initial states distribution.