MushroomRL
Introduction
What is MushroomRL
MushroomRL is a Reinforcement Learning (RL) library developed to be a simple, yet powerful way to make RL and deep RL experiments. The idea behind MushroomRL is to offer the majority of RL algorithms providing a common interface in order to run them without excessive effort. Moreover, it is designed in such a way that new algorithms and other stuff can be added transparently, without the need of editing other parts of the code. MushroomRL is compatible with RL libraries like OpenAI Gym, DeepMind Control Suite, Pybullet, and MuJoCo, and the PyTorch library for tensor computation.
With MushroomRL you can:
solve RL problems simply writing a single small script;
add custom algorithms, policies, and so on, transparently;
use all RL environments offered by well-known libraries and build customized environments as well;
exploit regression models offered by third-party libraries (e.g., scikit-learn) or build a customized one with PyTorch;
seamlessly run experiments on CPU or GPU.
Basic run example
Solve a discrete MDP in few a lines. Firstly, create a MDP:
from mushroom_rl.environments import GridWorld
mdp = GridWorld(width=3, height=3, goal=(2, 2), start=(0, 0))
Then, an epsilon-greedy policy with:
from mushroom_rl.policy import EpsGreedy
from mushroom_rl.utils.parameters import Parameter
epsilon = Parameter(value=1.)
policy = EpsGreedy(epsilon=epsilon)
Eventually, the agent is:
from mushroom_rl.algorithms.value import QLearning
learning_rate = Parameter(value=.6)
agent = QLearning(mdp.info, policy, learning_rate)
Learn:
from mushroom_rl.core import Core
core = Core(agent, mdp)
core.learn(n_steps=10000, n_steps_per_fit=1)
Print final Q-table:
import numpy as np
shape = agent.Q.shape
q = np.zeros(shape)
for i in range(shape[0]):
for j in range(shape[1]):
state = np.array([i])
action = np.array([j])
q[i, j] = agent.Q.predict(state, action)
print(q)
Results in:
[[ 6.561 7.29 6.561 7.29 ]
[ 7.29 8.1 6.561 8.1 ]
[ 8.1 9. 7.29 8.1 ]
[ 6.561 8.1 7.29 8.1 ]
[ 7.29 9. 7.29 9. ]
[ 8.1 10. 8.1 9. ]
[ 7.29 8.1 8.1 9. ]
[ 8.1 9. 8.1 10. ]
[ 0. 0. 0. 0. ]]
where the Q-values of each action of the MDP are stored for each rows representing a state of the MDP.
Download and installation
MushroomRL can be downloaded from the GitHub repository. Installation can be done running
pip3 install mushroom_rl
To compile the documentation:
cd mushroom_rl/docs
make html
or to compile the pdf version:
cd mushroom_rl/docs
make latexpdf
To launch MushroomRL test suite:
pytest
Installation troubleshooting
Common problems with the installation of MushroomRL arise in case some of its dependency are
broken or not installed. In general, we recommend installing MushroomRL with the option all
to install all the Python
dependencies. The installation time mostly depends on the time to install the dependencies.
A simple installation takes approximately 1 minute with a fast internet connection.
Installing with all the dependencies takes approximately 5 minutes using a fast internet connection. A slower
internet connection may increase the installation time significantly.
If installing all the dependencies, ensure that the swig library is installed, as it is used
by some Gym environments and the installation may fail otherwise. For Atari, you might need to install the ROM separately, otherwise
the creation of Atari environments may fail. Opencv should be installed too. For MuJoCo, ensure that the path of your MuJoCo folder is included
in the environment variable LD_LIBRARY_PATH
and that mujoco_py
is correctly installed.
Installing MushroomRL in a Conda environment is generally
safe. However, we are aware that when installing with the option
plots
, some errors may arise due to incompatibility issues between
pyqtgraph
and Conda. We recommend not using Conda when installing using plots
.
Finally, ensure that C/C++ compilers and Cython are working as expected.
To check if the installation has been successful, you can try to run the basic example above.
MushroomRL is well-tested on Linux. If you are using another OS, you may incur in issues that we are still not aware of. In that case, please do not hesitate sending us an email at mushroom4rl@gmail.com.
MushroomRL vs other libraries
MushroomRL offers the majority of classical and deep RL algorithms, while keeping a modular and flexible architecture. It is compatible with Pytorch, and most machine learning and RL libraries.
Features |
MushroomRL |
Stable Baselines |
RLLib |
Keras RL |
Chainer RL |
Tensorforce |
---|---|---|---|---|---|---|
Classic RL algorithms |
✅ |
❌ |
❌ |
❌ |
❌ |
❌ |
Deep RL algorithms |
✅ |
✅ |
✅ |
❌ |
✅ |
❌ |
Updated documentation |
✅ |
✅ |
✅ |
❌ |
✅ |
✅ |
Modular |
✅ |
❌ |
❌ |
❌ |
✅ |
✅ |
Easy to extend |
✅ |
❌ |
❌ |
❌ |
❌ |
❌ |
PEP8 compliant |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
Compatible with RL benchmarks |
✅ |
✅ |
✅ |
❌ |
✅ |
✅ |
Benchmarking suite |
✅ |
✅ |
✅ |
✅ |
✅ |
✅ |
MujoCo integration |
✅ |
❌ |
❌ |
❌ |
❌ |
❌ |
Pybullet integration |
✅ |
❌ |
❌ |
❌ |
❌ |
❌ |
Torch integration |
✅ |
❌ |
✅ |
✅ |
❌ |
❌ |
Tensorflow integration |
❌ |
✅ |
✅ |
✅ |
❌ |
✅ |
Chainer integration |
❌ |
❌ |
❌ |
❌ |
✅ |
❌ |
Parallel environments |
❌ |
✅ |
✅ |
❌ |
✅ |
✅ |