Reinforcement Learning utils
Eligibility trace
- EligibilityTrace(shape, name='replacing')[source]
Factory method to create an eligibility trace of the provided type.
- Parameters:
shape (list) – shape of the eligibility trace table;
name (str, 'replacing') – type of the eligibility trace.
- Returns:
The eligibility trace table of the provided shape and type.
- class ReplacingTrace(shape, initial_value=0.0, dtype=None)[source]
Bases:
Table
Replacing trace.
- __init__(shape, initial_value=0.0, dtype=None)
Constructor.
- Parameters:
shape (tuple) – the shape of the tabular regressor.
initial_value (float, 0.) – the initial value for each entry of the tabular regressor.
dtype ([int, float], None) – the dtype of the table array.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- fit(x, y)
- Parameters:
x (int) – index of the table to be filled;
y (float) – value to fill in the table.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- property n_actions
Returns: The number of actions considered by the table.
- predict(*z)
Predict the output of the table given an input.
- Parameters:
*z (list) – list of input of the model. If the table is a Q-table,
depending (this list may contain states or states and actions) – on whether the call requires to predict all q-values or only one q-value corresponding to the provided action;
- Returns:
The table prediction.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table.
- class AccumulatingTrace(shape, initial_value=0.0, dtype=None)[source]
Bases:
Table
Accumulating trace.
- __init__(shape, initial_value=0.0, dtype=None)
Constructor.
- Parameters:
shape (tuple) – the shape of the tabular regressor.
initial_value (float, 0.) – the initial value for each entry of the tabular regressor.
dtype ([int, float], None) – the dtype of the table array.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- fit(x, y)
- Parameters:
x (int) – index of the table to be filled;
y (float) – value to fill in the table.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- property n_actions
Returns: The number of actions considered by the table.
- predict(*z)
Predict the output of the table given an input.
- Parameters:
*z (list) – list of input of the model. If the table is a Q-table,
depending (this list may contain states or states and actions) – on whether the call requires to predict all q-values or only one q-value corresponding to the provided action;
- Returns:
The table prediction.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table.
Optimizers
- class Optimizer(lr=0.001, maximize=True, *params)[source]
Bases:
Serializable
Base class for gradient optimizers. These objects take the current parameters and the gradient estimate to compute the new parameters.
- __init__(lr=0.001, maximize=True, *params)[source]
Constructor
- Parameters:
lr ([float, Parameter]) – the learning rate;
maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class AdaptiveOptimizer(eps, maximize=True)[source]
Bases:
Optimizer
This class implements an adaptive gradient step optimizer. Instead of moving of a step proportional to the gradient, takes a step limited by a given metric M. To specify the metric, the natural gradient has to be provided. If natural gradient is not provided, the identity matrix is used.
The step rule is:
\[ \begin{align}\begin{aligned}\Delta\theta=\underset{\Delta\vartheta}{argmax}\Delta\vartheta^{t}\nabla_{\theta}J\\s.t.:\Delta\vartheta^{T}M\Delta\vartheta\leq\varepsilon\end{aligned}\end{align} \]Lecture notes, Neumann G. http://www.ias.informatik.tu-darmstadt.de/uploads/Geri/lecture-notes-constraint.pdf
- __init__(eps, maximize=True)[source]
Constructor.
- Parameters:
eps (float) – the maximum step defined by the metric;
maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class SGDOptimizer(lr=0.001, maximize=True)[source]
Bases:
Optimizer
This class implements the SGD optimizer.
- __init__(lr=0.001, maximize=True)[source]
Constructor.
- Parameters:
lr ([float, Parameter], 0.001) – the learning rate;
maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class AdamOptimizer(lr=0.001, beta1=0.9, beta2=0.999, eps=1e-07, maximize=True)[source]
Bases:
Optimizer
This class implements the Adam optimizer.
- __init__(lr=0.001, beta1=0.9, beta2=0.999, eps=1e-07, maximize=True)[source]
Constructor.
- Parameters:
lr ([float, Parameter], 0.001) – the learning rate;
beta1 (float, 0.9) – Adam beta1 parameter;
beta2 (float, 0.999) – Adam beta2 parameter;
maximize (bool, True) – by default Optimizers do a gradient ascent step. Set to False for gradient descent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- static _append_folder(folder, name)
- static _get_serialization_method(class_name)
- static _load_json(zip_file, name)
- classmethod _load_list(zip_file, folder, length)
- static _load_mushroom(zip_file, name)
- static _load_numpy(zip_file, name)
- static _load_pickle(zip_file, name)
- static _load_torch(zip_file, name)
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- static _save_json(zip_file, name, obj, folder, **_)
- static _save_mushroom(zip_file, name, obj, folder, full_save)
- static _save_numpy(zip_file, name, obj, folder, **_)
- static _save_pickle(zip_file, name, obj, folder, **_)
- static _save_torch(zip_file, name, obj, folder, **_)
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- classmethod load_zip(zip_file, folder='')
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
Parameters
- class Parameter(value, min_value=None, max_value=None, size=(1,))[source]
Bases:
Serializable
This class implements function to manage parameters, such as learning rate. It also allows to have a single parameter for each state of state-action tuple.
- __init__(value, min_value=None, max_value=None, size=(1,))[source]
Constructor.
- Parameters:
value (float) – initial value of the parameter;
min_value (float, None) – minimum value that the parameter can reach when decreasing;
max_value (float, None) – maximum value that the parameter can reach when increasing;
size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.
- __call__(*idx, **kwargs)[source]
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- get_value(*idx, **kwargs)[source]
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- update(*idx, **kwargs)[source]
Updates the number of visit of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
- property shape
Returns: The shape of the table of parameters.
- property initial_value
Returns: The initial value of the parameters.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class LinearParameter(value, threshold_value, n, size=(1,))[source]
Bases:
Parameter
This class implements a linearly changing parameter according to the number of times it has been used. The parameter changes following the formula:
\[v_n = \textrm{clip}(v_0 + \dfrac{v_{th} - v_0}{n}, v_{th})\]where \(v_0\) is the initial value of the parameter, \(n\) is the number of steps and \(v_{th}\) is the upper or lower threshold for the parameter.
- __init__(value, threshold_value, n, size=(1,))[source]
Constructor.
- Parameters:
value (float) – initial value of the parameter;
threshold_value (float, None) – minimum or maximum value that the parameter can reach;
n (int) – number of time steps needed to reach the threshold value;
size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- update(*idx, **kwargs)
Updates the number of visit of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
- class DecayParameter(value, exp=1.0, min_value=None, max_value=None, size=(1,))[source]
Bases:
Parameter
This class implements a decaying parameter. The decay follows the formula:
\[v_n = \dfrac{v_0}{n^p}\]where \(v_0\) is the initial value of the parameter, \(n\) is the number of steps and \(p\) is an arbitrary exponent.
- __init__(value, exp=1.0, min_value=None, max_value=None, size=(1,))[source]
Constructor.
- Parameters:
value (float) – initial value of the parameter;
exp (float, 1.) – exponent for the step decay;
min_value (float, None) – minimum value that the parameter can reach when decreasing;
max_value (float, None) – maximum value that the parameter can reach when increasing;
size (tuple, (1,)) – shape of the matrix of parameters; this shape can be used to have a single parameter for each state or state-action tuple.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- update(*idx, **kwargs)
Updates the number of visit of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
Preprocessors
- class Preprocessor[source]
Bases:
Serializable
Abstract preprocessor class.
- __call__(obs)[source]
Preprocess the observations.
- Parameters:
obs (Array) – observations to be preprocessed.
- Returns:
Preprocessed observations.
- update(obs)[source]
Update internal state of the preprocessor using the current observations.
- Parameters:
obs (Array) – observations to be preprocessed.
- __init__()
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class StandardizationPreprocessor(mdp_info, backend, clip_obs=10.0, alpha=1e-32)[source]
Bases:
Preprocessor
Preprocess observations from the environment using a running standardization.
- __init__(mdp_info, backend, clip_obs=10.0, alpha=1e-32)[source]
Constructor.
- Parameters:
mdp_info (MDPInfo) – information of the MDP;
backend (str) – name of the backend to be used;
clip_obs (float, 10.) – values to clip the normalized observations;
alpha (float, 1e-32) – moving average catchup parameter for the normalization.
- __call__(obs)[source]
Preprocess the observations.
- Parameters:
obs (Array) – observations to be preprocessed.
- Returns:
Preprocessed observations.
- update(obs)[source]
Update internal state of the preprocessor using the current observations.
- Parameters:
obs (Array) – observations to be preprocessed.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class MinMaxPreprocessor(mdp_info, backend, clip_obs=10.0, alpha=1e-32)[source]
Bases:
StandardizationPreprocessor
Preprocess observations from the environment using the bounds of the observation space of the environment. For observations that are not limited falls back to using running mean standardization.
- __init__(mdp_info, backend, clip_obs=10.0, alpha=1e-32)[source]
Constructor.
- Parameters:
mdp_info (MDPInfo) – information of the MDP;
backend (str) – name of the backend to be used;
clip_obs (float, 10.) – values to clip the normalized observations;
alpha (float, 1e-32) – moving average catchup parameter for the normalization.
- __call__(obs)[source]
Preprocess the observations.
- Parameters:
obs (Array) – observations to be preprocessed.
- Returns:
Preprocessed observations.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- update(obs)
Update internal state of the preprocessor using the current observations.
- Parameters:
obs (Array) – observations to be preprocessed.
Replay memory
- class ReplayMemory(mdp_info, agent_info, initial_size, max_size)[source]
Bases:
Serializable
This class implements function to manage a replay memory as the one used in “Human-Level Control Through Deep Reinforcement Learning” by Mnih V. et al..
- add(dataset, n_steps_return=1, gamma=1.0)[source]
Add elements to the replay memory.
- Parameters:
dataset (Dataset) – dataset class elements to add to the replay memory;
n_steps_return (int, 1) – number of steps to consider for computing n-step return;
gamma (float, 1.) – discount factor for n-step return.
- get(n_samples)[source]
Returns the provided number of states from the replay memory.
- Parameters:
n_samples (int) – the number of samples to return.
- Returns:
The requested number of samples.
- property initialized
Returns: Whether the replay memory has reached the number of elements that allows it to be used.
- property size
Returns: The number of elements contained in the replay memory.
- _post_load()[source]
This method can be overwritten to implement logic that is executed after the loading of the agent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class SequenceReplayMemory(mdp_info, agent_info, initial_size, max_size, truncation_length)[source]
Bases:
ReplayMemory
This class extend the base replay memory to allow sampling sequences of a certain length. This is useful for training recurrent agents or agents operating on a window of states etc.
- get(n_samples)[source]
Returns the provided number of states from the replay memory. :param n_samples: the number of samples to return. :type n_samples: int
- Returns:
The requested number of samples.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- add(dataset, n_steps_return=1, gamma=1.0)
Add elements to the replay memory.
- Parameters:
dataset (Dataset) – dataset class elements to add to the replay memory;
n_steps_return (int, 1) – number of steps to consider for computing n-step return;
gamma (float, 1.) – discount factor for n-step return.
- copy()
- Returns:
A deepcopy of the agent.
- property initialized
Returns: Whether the replay memory has reached the number of elements that allows it to be used.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- reset()
Reset the replay memory.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property size
Returns: The number of elements contained in the replay memory.
- class SumTree(mdp_info, agent_info, max_size)[source]
Bases:
Serializable
This class implements a sum tree data structure. This is used, for instance, by
PrioritizedReplayMemory
.- add(dataset, priority, n_steps_return, gamma)[source]
Add elements to the tree.
- Parameters:
dataset (Dataset) – dataset class elements to add to the replay memory;
priority (np.ndarray) – priority of each sample in the dataset;
n_steps_return (int) – number of steps to consider for computing n-step return;
gamma (float) – discount factor for n-step return.
- get(s)[source]
Returns the provided number of states from the replay memory.
- Parameters:
s (float) – the value of the samples to return.
- Returns:
The requested sample.
- get_ind(s)[source]
Returns the provided number of states from the replay memory.
- Parameters:
s (float) – the value of the samples to return.
- Returns:
The requested sample.
- update(idx, priorities)[source]
Update the priority of the sample at the provided index in the dataset.
- Parameters:
idx (np.ndarray) – indexes of the transitions in the dataset;
priorities (np.ndarray) – priorities of the transitions.
- property size
Returns: The current size of the tree.
- property max_p
Returns: The maximum priority among the ones in the tree.
- property total_p
Returns: The sum of the priorities in the tree, i.e. the value of the root node.
- _post_load()[source]
This method can be overwritten to implement logic that is executed after the loading of the agent.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class PrioritizedReplayMemory(mdp_info, agent_info, initial_size, max_size, alpha, beta, epsilon=0.01)[source]
Bases:
Serializable
This class implements function to manage a prioritized replay memory as the one used in “Prioritized Experience Replay” by Schaul et al., 2015.
- __init__(mdp_info, agent_info, initial_size, max_size, alpha, beta, epsilon=0.01)[source]
Constructor.
- Parameters:
mdp_info (MDPInfo) – information about the MDP;
agent_info (AgentInfo) – information about the agent;
initial_size (int) – initial number of elements in the replay memory;
max_size (int) – maximum number of elements that the replay memory can contain;
alpha (float) – prioritization coefficient;
beta ([float, Parameter]) – importance sampling coefficient;
epsilon (float, .01) – small value to avoid zero probabilities.
- add(dataset, p, n_steps_return=1, gamma=1.0)[source]
Add elements to the replay memory.
- Parameters:
dataset (Dataset) – list of elements to add to the replay memory;
p (np.ndarray) – priority of each sample in the dataset.
n_steps_return (int, 1) – number of steps to consider for computing n-step return;
gamma (float, 1.) – discount factor for n-step return.
- get(n_samples)[source]
Returns the provided number of states from the replay memory.
- Parameters:
n_samples (int) – the number of samples to return.
- Returns:
The requested number of samples.
- update(error, idx)[source]
Update the priority of the sample at the provided index in the dataset.
- Parameters:
error (np.ndarray) – errors to consider to compute the priorities;
idx (np.ndarray) – indexes of the transitions in the dataset.
- property initialized
Returns: Whether the replay memory has reached the number of elements that allows it to be used.
- property max_priority
Returns: The maximum value of priority inside the replay memory.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
Running Statistics
- class RunningStandardization(shape, backend, alpha=1e-32)[source]
Bases:
Serializable
Compute a running standardization of values according to Welford’s online algorithm: https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford’s_online_algorithm
- __init__(shape, backend, alpha=1e-32)[source]
Constructor.
- Parameters:
shape (tuple) – shape of the data to standardize;
backend (str) – name of the backend to be used;
alpha (float, 1e-32) – minimum learning rate.
- update_stats(value)[source]
Update the statistics with the current data value.
- Parameters:
value (Array) – current data value to use for the update.
- property mean
Returns: The estimated mean value.
- property std
Returns: The estimated standard deviation value.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class RunningExpWeightedAverage(shape, alpha, backend, init_value=None)[source]
Bases:
Serializable
Compute an exponentially weighted moving average.
- __init__(shape, alpha, backend, init_value=None)[source]
Constructor.
- Parameters:
shape (tuple) – shape of the data to standardize;
alpha (float) – learning rate;
backend (str) – name of the backend to be used;
init_value (np.ndarray) – initial value of the filter.
- reset(init_value=None)[source]
Reset the mean and standard deviation.
- Parameters:
init_value (Array) – initial value of the filter.
- update_stats(value)[source]
Update the statistics with the current data value.
- Parameters:
value (Array) – current data value to use for the update.
- property mean
Returns: The estimated mean value.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- class RunningAveragedWindow(shape, window_size, backend, init_value=None)[source]
Bases:
Serializable
Compute the running average using a window of fixed size.
- __init__(shape, window_size, backend, init_value=None)[source]
Constructor.
- Parameters:
shape (tuple) – shape of the data to standardize;
window_size (int) – size of the windows;
backend (str) – name of the backend to be used;
init_value (np.ndarray) – initial value of the filter.
- reset(init_value=None)[source]
Reset the window.
- Parameters:
init_value (np.ndarray) – initial value of the filter.
- update_stats(value)[source]
Update the statistics with the current data value.
- Parameters:
value (np.ndarray) – current data value to use for the update.
- property mean
Returns: The estimated mean value.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
Spaces
- class Box(low, high, shape=None, data_type=<class 'float'>)[source]
Bases:
Serializable
This class implements functions to manage continuous states and action spaces. It is similar to the
Box
class ingym.spaces.box
.- __init__(low, high, shape=None, data_type=<class 'float'>)[source]
Constructor.
- Parameters:
low ([float, np.ndarray]) – the minimum value of each dimension of the space. If a scalar value is provided, this value is considered as the minimum one for each dimension. If a np.ndarray is provided, each i-th element is considered the minimum value of the i-th dimension;
high ([float, np.ndarray]) – the maximum value of dimensions of the space. If a scalar value is provided, this value is considered as the maximum one for each dimension. If a np.ndarray is provided, each i-th element is considered the maximum value of the i-th dimension;
shape (np.ndarray, None) – the dimension of the space. Must match the shape of
low
andhigh
, if they are np.ndarray.data_type (class, float) – the data type to be used.
- property low
Returns: The minimum value of each dimension of the space.
- property high
Returns: The maximum value of each dimension of the space.
- property shape
Returns: The dimensions of the space.
- class Discrete(n)[source]
Bases:
Serializable
This class implements functions to manage discrete states and action spaces. It is similar to the
Discrete
class ingym.spaces.discrete
.- property size
Returns: The number of elements of the space.
- property shape
Returns: The shape of the space that is always (1,).
Value Functions
- compute_advantage_montecarlo(V, s, ss, r, absorbing, gamma)[source]
Function to estimate the advantage and new value function target over a dataset. The value function is estimated using rollouts (monte carlo estimation).
- Parameters:
V (Regressor) – the current value function regressor;
s (torch.tensor) – the set of states in which we want to evaluate the advantage;
ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;
r (torch.tensor) – the reward obtained in each transition from state s to state ss;
absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;
gamma (float) – the discount factor of the considered problem.
- Returns:
The new estimate for the value function of the next state and the advantage function.
- compute_advantage(V, s, ss, r, absorbing, gamma)[source]
Function to estimate the advantage and new value function target over a dataset. The value function is estimated using bootstrapping.
- Parameters:
V (Regressor) – the current value function regressor;
s (torch.tensor) – the set of states in which we want to evaluate the advantage;
ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;
r (torch.tensor) – the reward obtained in each transition from state s to state ss;
absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;
gamma (float) – the discount factor of the considered problem.
- Returns:
The new estimate for the value function of the next state and the advantage function.
- compute_gae(V, s, ss, r, absorbing, last, gamma, lam)[source]
Function to compute Generalized Advantage Estimation (GAE) and new value function target over a dataset.
“High-Dimensional Continuous Control Using Generalized Advantage Estimation”. Schulman J. et al.. 2016.
- Parameters:
V (Regressor) – the current value function regressor;
s (torch.tensor) – the set of states in which we want to evaluate the advantage;
ss (torch.tensor) – the set of next states in which we want to evaluate the advantage;
r (torch.tensor) – the reward obtained in each transition from state s to state ss;
absorbing (torch.tensor) – an array of boolean flags indicating if the reached state is absorbing;
last (torch.tensor) – an array of boolean flags indicating if the reached state is the last of the trajectory;
gamma (float) – the discount factor of the considered problem;
lam (float) – the value for the lamba coefficient used by GEA algorithm.
- Returns:
The new estimate for the value function of the next state and the estimated generalized advantage.
Variance parameters
- class VarianceParameter(value, exponential=False, min_value=None, tol=1.0, size=(1,))[source]
Bases:
Parameter
Abstract class to implement variance-dependent parameters. A
target
parameter is expected.- __init__(value, exponential=False, min_value=None, tol=1.0, size=(1,))[source]
Constructor.
- Parameters:
tol (float) – value of the variance of the target variable such that The parameter value is 0.5.
- update(*idx, **kwargs)[source]
Updates the value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
target (float) – Value of the target variable;
factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- class VarianceIncreasingParameter(value, exponential=False, min_value=None, tol=1.0, size=(1,))[source]
Bases:
VarianceParameter
Class implementing a parameter that increases with the target variance.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- __init__(value, exponential=False, min_value=None, tol=1.0, size=(1,))
Constructor.
- Parameters:
tol (float) – value of the variance of the target variable such that The parameter value is 0.5.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _compute(*idx, **kwargs)
- Returns:
The value of the parameter in the provided index.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- update(*idx, **kwargs)
Updates the value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
target (float) – Value of the target variable;
factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.
- class VarianceDecreasingParameter(value, exponential=False, min_value=None, tol=1.0, size=(1,))[source]
Bases:
VarianceParameter
Class implementing a parameter that decreases with the target variance.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- __init__(value, exponential=False, min_value=None, tol=1.0, size=(1,))
Constructor.
- Parameters:
tol (float) – value of the variance of the target variable such that The parameter value is 0.5.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _compute(*idx, **kwargs)
- Returns:
The value of the parameter in the provided index.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- update(*idx, **kwargs)
Updates the value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
target (float) – Value of the target variable;
factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.
- class WindowedVarianceParameter(value, exponential=False, min_value=None, tol=1.0, window=100, size=(1,))[source]
Bases:
Parameter
Abstract class to implement variance-dependent parameters. A
target
parameter is expected. differently from the “Variance Parameter” class the variance is computed in a window interval.- __init__(value, exponential=False, min_value=None, tol=1.0, window=100, size=(1,))[source]
Constructor.
- Parameters:
tol (float) – value of the variance of the target variable such that the parameter value is 0.5.
window (int) –
- update(*idx, **kwargs)[source]
Updates the value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
target (float) – Value of the target variable;
factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- class WindowedVarianceIncreasingParameter(value, exponential=False, min_value=None, tol=1.0, window=100, size=(1,))[source]
Bases:
WindowedVarianceParameter
Class implementing a parameter that decreases with the target variance, where the variance is computed in a fixed length window.
- __call__(*idx, **kwargs)
Update and return the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The updated parameter in the provided index.
- __init__(value, exponential=False, min_value=None, tol=1.0, window=100, size=(1,))
Constructor.
- Parameters:
tol (float) – value of the variance of the target variable such that the parameter value is 0.5.
window (int) –
- _add_save_attr(**attr_dict)
Add attributes that should be saved for an agent. For every attribute, it is necessary to specify the method to be used to save and load. Available methods are: numpy, mushroom, torch, json, pickle, primitive and none. The primitive method can be used to store primitive attributes, while the none method always skip the attribute, but ensure that it is initialized to None after the load. The mushroom method can be used with classes that implement the Serializable interface. All the other methods use the library named. If a “!” character is added at the end of the method, the field will be saved only if full_save is set to True.
- Parameters:
**attr_dict – dictionary of attributes mapped to the method that should be used to save and load them.
- _compute(*idx, **kwargs)
- Returns:
The value of the parameter in the provided index.
- _post_load()
This method can be overwritten to implement logic that is executed after the loading of the agent.
- copy()
- Returns:
A deepcopy of the agent.
- get_value(*idx, **kwargs)
Return the current value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter to return.
- Returns:
The current value of the parameter in the provided index.
- property initial_value
Returns: The initial value of the parameters.
- classmethod load(path)
Load and deserialize the agent from the given location on disk.
- Parameters:
path (Path, string) – Relative or absolute path to the agents save location.
- Returns:
The loaded agent.
- save(path, full_save=False)
Serialize and save the object to the given path on disk.
- Parameters:
path (Path, str) – Relative or absolute path to the object save location;
full_save (bool) – Flag to specify the amount of data to save for MushroomRL data structures.
- save_zip(zip_file, full_save, folder='')
Serialize and save the agent to the given path on disk.
- Parameters:
zip_file (ZipFile) – ZipFile where te object needs to be saved;
full_save (bool) – flag to specify the amount of data to save for MushroomRL data structures;
folder (string, '') – subfolder to be used by the save method.
- property shape
Returns: The shape of the table of parameters.
- update(*idx, **kwargs)
Updates the value of the parameter in the provided index.
- Parameters:
*idx (list) – index of the parameter whose number of visits has to be updated.
target (float) – Value of the target variable;
factor (float) – Multiplicative factor for the parameter value, useful when the parameter depend on another parameter value.