MushroomRL

Introduction

What is MushroomRL

MushroomRL is a Reinforcement Learning (RL) library developed to be a simple, yet powerful way to make RL and deep RL experiments. The idea behind MushroomRL is to offer the majority of RL algorithms providing a common interface in order to run them without excessive effort. Moreover, it is designed in such a way that new algorithms and other stuff can be added transparently, without the need of editing other parts of the code. MushroomRL is compatible with RL libraries like OpenAI Gym, DeepMind Control Suite, Pybullet, and MuJoCo, and the PyTorch library for tensor computation.

With MushroomRL you can:

  • solve RL problems simply writing a single small script;

  • add custom algorithms, policies, and so on, transparently;

  • use all RL environments offered by well-known libraries and build customized environments as well;

  • exploit regression models offered by third-party libraries (e.g., scikit-learn) or build a customized one with PyTorch;

  • seamlessly run experiments on CPU or GPU.

Basic run example

Solve a discrete MDP in few a lines. Firstly, create a MDP:

from mushroom_rl.environments import GridWorld

mdp = GridWorld(width=3, height=3, goal=(2, 2), start=(0, 0))

Then, an epsilon-greedy policy with:

from mushroom_rl.policy import EpsGreedy
from mushroom_rl.utils.parameters import Parameter

epsilon = Parameter(value=1.)
policy = EpsGreedy(epsilon=epsilon)

Eventually, the agent is:

from mushroom_rl.algorithms.value import QLearning

learning_rate = Parameter(value=.6)
agent = QLearning(mdp.info, policy, learning_rate)

Learn:

from mushroom_rl.core import Core

core = Core(agent, mdp)
core.learn(n_steps=10000, n_steps_per_fit=1)

Print final Q-table:

import numpy as np

shape = agent.Q.shape
q = np.zeros(shape)
for i in range(shape[0]):
    for j in range(shape[1]):
        state = np.array([i])
        action = np.array([j])
        q[i, j] = agent.Q.predict(state, action)
print(q)

Results in:

[[  6.561   7.29    6.561   7.29 ]
 [  7.29    8.1     6.561   8.1  ]
 [  8.1     9.      7.29    8.1  ]
 [  6.561   8.1     7.29    8.1  ]
 [  7.29    9.      7.29    9.   ]
 [  8.1    10.      8.1     9.   ]
 [  7.29    8.1     8.1     9.   ]
 [  8.1     9.      8.1    10.   ]
 [  0.      0.      0.      0.   ]]

where the Q-values of each action of the MDP are stored for each rows representing a state of the MDP.

Download and installation

MushroomRL can be downloaded from the GitHub repository. Installation can be done running

pip3 install mushroom_rl

To compile the documentation:

cd mushroom_rl/docs
make html

or to compile the pdf version:

cd mushroom_rl/docs
make latexpdf

To launch MushroomRL test suite:

pytest

Installation troubleshooting

Common problems with the installation of MushroomRL arise in case some of its dependency are broken or not installed. In general, we recommend installing MushroomRL with the option all to install all the Python dependencies. The installation time mostly depends on the time to install the dependencies. A simple installation takes approximately 1 minute with a fast internet connection. Installing with all the dependencies takes approximately 5 minutes using a fast internet connection. A slower internet connection may increase the installation time significantly.

If installing all the dependencies, ensure that the swig library is installed, as it is used by some Gym environments and the installation may fail otherwise. For Atari, you might need to install the ROM separately, otherwise the creation of Atari environments may fail. Opencv should be installed too. For MuJoCo, ensure that the path of your MuJoCo folder is included in the environment variable LD_LIBRARY_PATH and that mujoco_py is correctly installed. Installing MushroomRL in a Conda environment is generally safe. However, we are aware that when installing with the option plots, some errors may arise due to incompatibility issues between pyqtgraph and Conda. We recommend not using Conda when installing using plots. Finally, ensure that C/C++ compilers and Cython are working as expected.

To check if the installation has been successful, you can try to run the basic example above.

MushroomRL is well-tested on Linux. If you are using another OS, you may incur in issues that we are still not aware of. In that case, please do not hesitate sending us an email at mushroom4rl@gmail.com.

MushroomRL vs other libraries

MushroomRL offers the majority of classical and deep RL algorithms, while keeping a modular and flexible architecture. It is compatible with Pytorch, and most machine learning and RL libraries.

Features

MushroomRL

Stable Baselines

RLLib

Keras RL

Chainer RL

Tensorforce

Classic RL algorithms

Deep RL algorithms

Updated documentation

Modular

Easy to extend

PEP8 compliant

Compatible with RL benchmarks

Benchmarking suite

MujoCo integration

Pybullet integration

Torch integration

Tensorflow integration

Chainer integration

Parallel environments

API Documentation

Tutorials