MushroomRL

What is MushroomRL

MushroomRL is a Reinforcement Learning (RL) library developed to be a simple, yet powerful way to make RL and deep RL experiments. The idea behind MushroomRL is to offer the majority of RL algorithms providing a common interface in order to run them without excessive effort. Moreover, it is designed in such a way that new algorithms and other stuff can be added transparently, without the need of editing other parts of the code. MushroomRL is compatible with RL libraries like Gymnasium, DeepMind Control Suite, Pybullet, and MuJoCo, and the PyTorch library for tensor computation.

With MushroomRL you can:

  • solve RL problems simply writing a single small script;

  • use classic RL algorithms and deep RL ones from the same library, behind the same interface;

  • add custom algorithms, policies, and so on, transparently;

  • use all RL environments offered by well-known libraries and build customized environments as well;

  • run experiments on MuJoCo, PyBullet, Isaac Sim, Gymnasium and the DeepMind Control Suite;

  • exploit regression models offered by third-party libraries (e.g., scikit-learn) or build a customized one with PyTorch;

  • collect samples from parallel and vectorized environments;

  • seamlessly run experiments on CPU or GPU.

Basic run example

Solve a discrete MDP in few a lines. Firstly, create a MDP:

from mushroom_rl.environments import GridWorld

mdp = GridWorld.from_size(width=3, height=3, goal=(2, 2), start=(0, 0))

Then, an epsilon-greedy policy with:

from mushroom_rl.policy import EpsGreedy
from mushroom_rl.rl_utils.parameters import Parameter

epsilon = Parameter(value=1.)
policy = EpsGreedy(epsilon=epsilon)

Eventually, the agent is:

from mushroom_rl.algorithms.value import QLearning

learning_rate = Parameter(value=.6)
agent = QLearning(mdp.info, policy, learning_rate)

Learn:

from mushroom_rl.core import Core

core = Core(agent, mdp)
core.learn(n_steps=10000, n_steps_per_fit=1)

Print final Q-table:

import numpy as np

shape = agent.Q.shape
q = np.zeros(shape)
for i in range(shape[0]):
    for j in range(shape[1]):
        state = np.array([i])
        action = np.array([j])
        q[i, j] = agent.Q.predict(state, action)
print(q)

Results in:

[[0.6561 0.729  0.6561 0.729 ]
 [0.729  0.81   0.6561 0.81  ]
 [0.81   0.9    0.729  0.81  ]
 [0.6561 0.81   0.729  0.81  ]
 [0.729  0.9    0.729  0.9   ]
 [0.81   1.     0.81   0.9   ]
 [0.729  0.81   0.81   0.9   ]
 [0.81   0.9    0.81   1.    ]
 [0.     0.     0.     0.    ]]

where the Q-values of each action of the MDP are stored for each rows representing a state of the MDP.