Agent-Environment Interface

The three basic interface of mushroom are the Agent, the Environment and the Core interface.

  • The Agent is the basic interface for any Reinforcement Learning algorithm.
  • The Environment is the basic interface for every problem/task that the agent should solve.
  • The Core is a class used to control the interaction between an agent and an environment.

Agent

Mushroom provides the implementations of several algorithms belonging to all categories of RL:

  • value-based;
  • policy-search;
  • actor-critic.

One can easily implement customized algorithms following the structure of the already available ones, by extending the following interface:

class mushroom.algorithms.agent.Agent(policy, mdp_info, features=None)[source]

Bases: object

This class implements the functions to manage the agent (e.g. move the agent following its policy).

__init__(policy, mdp_info, features=None)[source]

Constructor.

Parameters:
  • policy (Policy) – the policy followed by the agent;
  • mdp_info (MDPInfo) – information about the MDP;
  • features (object, None) – features to extract from the state.
fit(dataset)[source]

Fit step.

Parameters:dataset (list) – the dataset.
draw_action(state)[source]

Return the action to execute in the given state. It is the action returned by the policy or the action set by the algorithm (e.g. in the case of SARSA).

Parameters:state (np.ndarray) – the state where the agent is.
Returns:The action to be executed.
episode_start()[source]

Called by the agent when a new episode starts.

stop()[source]

Method used to stop an agent. Useful when dealing with real world environments, simulators, or to cleanup environments internals after a core learn/evaluate to enforce consistency.

Environment

Mushroom provides several implementation of well known benchmarks with both continuous and discrete action spaces.

To implement a new environment, it is mandatory to use the following interface:

class mushroom.environments.environment.MDPInfo(observation_space, action_space, gamma, horizon)[source]

Bases: object

This class is used to store the information of the environment.

__init__(observation_space, action_space, gamma, horizon)[source]

Constructor.

Parameters:
  • observation_space ([Box, Discrete]) – the state space;
  • action_space ([Box, Discrete]) – the action space;
  • gamma (float) – the discount factor;
  • horizon (int) – the horizon.
size

The sum of the number of discrete states and discrete actions. Only works for discrete spaces.

Type:Returns
shape

The concatenation of the shape tuple of the state and action spaces.

Type:Returns
class mushroom.environments.environment.Environment(mdp_info)[source]

Bases: object

Basic interface used by any mushroom environment.

__init__(mdp_info)[source]

Constructor.

Parameters:mdp_info (MDPInfo) – an object containing the info of the environment.
seed(seed)[source]

Set the seed of the environment.

Parameters:seed (float) – the value of the seed.
reset(state=None)[source]

Reset the current state.

Parameters:state (np.ndarray, None) – the state to set to the current state.
Returns:The current state.
step(action)[source]

Move the agent from its current state according to the action.

Parameters:action (np.ndarray) – the action to execute.
Returns:The state reached by the agent executing action in its current state, the reward obtained in the transition and a flag to signal if the next state is absorbing. Also an additional dictionary is returned (possibly empty).
stop()[source]

Method used to stop an mdp. Useful when dealing with real world environments, simulators, or when using openai-gym rendering

info

An object containing the info of the environment.

Type:Returns
static _bound(x, min_value, max_value)[source]

Method used to bound state and action variables.

Parameters:
  • x – the variable to bound;
  • min_value – the minimum value;
  • max_value – the maximum value;
Returns:

The bounded variable.

Core

class mushroom.core.core.Core(agent, mdp, callbacks=None)[source]

Bases: object

Implements the functions to run a generic algorithm.

__init__(agent, mdp, callbacks=None)[source]

Constructor.

Parameters:
  • agent (Agent) – the agent moving according to a policy;
  • mdp (Environment) – the environment in which the agent moves;
  • callbacks (list) – list of callbacks to execute at the end of each learn iteration.
learn(n_steps=None, n_episodes=None, n_steps_per_fit=None, n_episodes_per_fit=None, render=False, quiet=False)[source]

This function moves the agent in the environment and fits the policy using the collected samples. The agent can be moved for a given number of steps or a given number of episodes and, independently from this choice, the policy can be fitted after a given number of steps or a given number of episodes. By default, the environment is reset.

Parameters:
  • n_steps (int, None) – number of steps to move the agent;
  • n_episodes (int, None) – number of episodes to move the agent;
  • n_steps_per_fit (int, None) – number of steps between each fit of the policy;
  • n_episodes_per_fit (int, None) – number of episodes between each fit of the policy;
  • render (bool, False) – whether to render the environment or not;
  • quiet (bool, False) – whether to show the progress bar or not.
evaluate(initial_states=None, n_steps=None, n_episodes=None, render=False, quiet=False)[source]

This function moves the agent in the environment using its policy. The agent is moved for a provided number of steps, episodes, or from a set of initial states for the whole episode. By default, the environment is reset.

Parameters:
  • initial_states (np.ndarray, None) – the starting states of each episode;
  • n_steps (int, None) – number of steps to move the agent;
  • n_episodes (int, None) – number of episodes to move the agent;
  • render (bool, False) – whether to render the environment or not;
  • quiet (bool, False) – whether to show the progress bar or not.
_step(render)[source]

Single step.

Parameters:render (bool) – whether to render or not.
Returns:A tuple containing the previous state, the action sampled by the agent, the reward obtained, the reached state, the absorbing flag of the reached state and the last step flag.
reset(initial_states=None)[source]

Reset the state of the agent.