README.md 1.97 KB
Newer Older
MasterScrat's avatar
MasterScrat committed
1
# 🚂 Flatland Baselines
metataro's avatar
metataro committed
2

MasterScrat's avatar
MasterScrat committed
3
This repository contains reinforcement learning baselines for the [NeurIPS 2020 Flatland Challenge](https://www.aicrowd.com/challenges/neurips-2020-flatland-challenge/).
metataro's avatar
metataro committed
4

MasterScrat's avatar
MasterScrat committed
5 6
## Provided baselines

MasterScrat's avatar
MasterScrat committed
7 8 9 10
```{note}
**Note:** looking for something simpler to start? We also provide a simpler Dueling Double DQN method implemented using PyTorch without relying on RLlib: **https://gitlab.aicrowd.com/flatland/flatland-examples** 
```

MasterScrat's avatar
MasterScrat committed
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
### RL Methods

- Ape-X
- PPO
- CCPPO
- Pure Imitation Learning: MARWIL
- Mixed IL/RL: DQfD 

### Custom observations

- Density observations
- Local conflict observations

### Tricks

- Action skipping
- Action masking

MasterScrat's avatar
MasterScrat committed
29 30 31



MasterScrat's avatar
MasterScrat committed
32
## Organisation
MasterScrat's avatar
Typo  
MasterScrat committed
33 34 35 36 37 38 39 40

Experiments consist of one or many RLlib YAML config files alongside a MARKDOWN file containing results, plots and a detailed description of the methodology.

All files are stored in a experiment folder under `experiments/<env-name>/<experiment-name>`.

- [Tree observations w/ fully connected network](experiments/flatland_random_sparse_small/tree_obs_fc_net)
- [Global observations w/ convnet](experiments/flatland_random_sparse_small/global_obs_conv_net)

MasterScrat's avatar
MasterScrat committed
41
## Setup
MasterScrat's avatar
MasterScrat committed
42

MasterScrat's avatar
MasterScrat committed
43
The setup uses conda, [install it](https://www.anaconda.com/products/individual) if necessary.
MasterScrat's avatar
MasterScrat committed
44 45

```
46 47
# with GPU support:
conda env create -f environment-gpu.yml
MasterScrat's avatar
MasterScrat committed
48
conda activate flatland-baseline-gpu-env
49 50

# or, without GPU support:
MasterScrat's avatar
MasterScrat committed
51 52
conda env create -f environment-cpu.yml
conda activate flatland-baseline-cpu-env
metataro's avatar
metataro committed
53 54 55
```

## Usage
56

metataro's avatar
metataro committed
57
Training example:
MasterScrat's avatar
MasterScrat committed
58 59

`python ./train.py -f experiments/flatland_random_sparse_small/global_obs_conv_net/ppo.yaml`
metataro's avatar
metataro committed
60

61
Evaluation example:
MasterScrat's avatar
MasterScrat committed
62 63 64

`python ./rollout.py /tmp/ray/checkpoint_dir/checkpoint-0 --run PPO --no-render
        --config '{"env_config": {"test": true}}' --episodes 1000 --out rollouts.pkl`
metataro's avatar
metataro committed
65

MasterScrat's avatar
MasterScrat committed
66
Note that `-f` overrides all other trial-specific command-line options.
metataro's avatar
metataro committed
67

MasterScrat's avatar
MasterScrat committed
68
## Notes
metataro's avatar
metataro committed
69

MasterScrat's avatar
MasterScrat committed
70
- The basic structure of this repository is adapted from [https://github.com/spMohanty/rl-experiments/](https://github.com/spMohanty/rl-experiments/)