This repository allows to run Rail Environment multi agent training with the RLLib Library.

## Installation:
```sh
pip install ray
pip install gin-config
```

To start a training with different parameters, you can create a folder containing a config.gin file (see example in `experiment_configs/config_example/config.gin`.

Then, you can modify the config.gin file path at the end of the `train_experiment.py` file.

The results will be stored inside the folder, and the learning curves can be visualized in 
tensorboard:

```
tensorboard --logdir=/path/to/folder_containing_config_gin_file
```

## Gin config files

In each config.gin files, all the parameters of the `run_experiment` functions have to be specified.
For example, to indicate the number of agents that have to be initialized at the beginning of each simulation, the following line should be added:

```
run_experiment.n_agents = 2
```

If several number of agents have to be explored during the experiment, one can pass the following value to the `n_agents` parameter:

```
run_experiment.n_agents = {"grid_search": [2,5]}
```

which is the way to indicate to the tune library to experiment several values for a parameter.

To reference a class or an object within gin, you should first register it from the `train_experiment.py` script adding the following line:

```
gin.external_configurable(TreeObsForRailEnv)
```

and then a `TreeObsForRailEnv` object can be referenced in the `config.gin` file:

```
run_experiment.obs_builder = {"grid_search": [@TreeObsForRailEnv(), @GlobalObsForRailEnv()]}
TreeObsForRailEnv.max_depth = 2
```

Note that `@TreeObsForRailEnv` references the class, while `@TreeObsForRailEnv()` references instantiates an object of this class.




More documentation on how to use gin-config can be found on the github repository: https://github.com/google/gin-config

## Run an example:
To start a training on a 20X20 map, with different numbers of agents initialized at each episode, on can run the train_experiment.py script:
```
python baselines/RLLib_training/train_experiment.py
```
This will load the gin config file in the folder `experiment_configs/config_examples`.

To visualize the result of a training, one can load a training checkpoint and use the policy learned.
This is done in the `render_training_result.py` script. One has to modify the `CHECKPOINT_PATH` at the beginning of this script:

```
CHECKPOINT_PATH = os.path.join(__file_dirname__, 'experiment_configs', 'config_example', 'ppo_policy_two_obs_with_predictions_n_agents_4_map_size_20q58l5_f7',
                               'checkpoint_101', 'checkpoint-101')
```
and load the corresponding gin config file:

```
gin.parse_config_file(os.path.join(__file_dirname__, 'experiment_configs', 'config_example', 'config.gin'))
```