README.md 8.21 KB
Newer Older
spmohanty's avatar
spmohanty committed
1
2
3
4
5
6
![AIcrowd-Logo](https://raw.githubusercontent.com/AIcrowd/AIcrowd/master/app/assets/images/misc/aicrowd-horizontal.png)

# Flatland Challenge Starter Kit

[![gitter-badge](https://badges.gitter.im/AIcrowd-HQ/flatland-rl.png)](https://gitter.im/AIcrowd-HQ/flatland-rl)

spmohanty's avatar
spmohanty committed
7
8
![](https://i.imgur.com/9cNtWjs.gif)

spmohanty's avatar
spmohanty committed
9
10
11
12
Instructions to make submissions to the [SBB CFF Flatland Challenge](https://www.aicrowd.com/challenges/flatland-challenge).

Participants will have to submit their code, with packaging specifications, and the evaluator will automatically build a docker image and execute their agent against an arbitrary number of pre-generated flatland environments.

13
### Dependencies 
spmohanty's avatar
spmohanty committed
14
15
16

- **Anaconda** (By following instructions [here](https://www.anaconda.com/download)) At least version `4.5.11` is required to correctly populate `environment.yml`.
- **flatland-rl** (By following instructions [here](http://flatland-rl-docs.s3-website.eu-central-1.amazonaws.com/readme.html))
17
  **IMPORTANT** : Please note that you will need `flatland-rl` version `>=2.1.7` to be able to submit.
18
19
20
21
22


### Setup

* Clone the repository 
spmohanty's avatar
spmohanty committed
23
24

```
25
26
git clone git@github.com:AIcrowd/flatland-challenge-starter-kit.git
cd flatland-challenge-starter-kit
spmohanty's avatar
spmohanty committed
27
28
```

29
* Create a conda environment from the provided `environment.yml`
spmohanty's avatar
spmohanty committed
30
31

```sh
32
conda env create -f environment.yml
spmohanty's avatar
spmohanty committed
33
34
```

35
* Activate the conda environment and install your code specific dependencies
spmohanty's avatar
spmohanty committed
36

37
38
39
40
41
42
43
44
45
```sh
conda activate flatland-rl
# If say you want to install PyTorch
# conda install pytorch torchvision -c pytorch
#
# or you can even use pip to install any additional packages
# for example : 
# pip install -U flatland-rl
# which updates the flatland-rl package to the latest version
spmohanty's avatar
spmohanty committed
46
```
47

spmohanty's avatar
spmohanty committed
48
49
50

### Test Submission Locally

spmohanty's avatar
spmohanty committed
51
52
53
54
* First lets begin by downloading a small set of test envs, and put them at a location of your choice.
In this exercise, we assume that you will download the test-envs provided at : [https://www.aicrowd.com/challenges/flatland-challenge/dataset_files](https://www.aicrowd.com/challenges/flatland-challenge/dataset_files), and will untar them inside `./scratch/test-envs`, so that you have a directory structure similar to : 

```
spmohanty's avatar
spmohanty committed
55
./scratch
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
└── test-envs
    ├── Test_0
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_1
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_2
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_3
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_4
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_5
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_6
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_7
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_8
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    └── Test_9
        ├── Level_0.pkl
        └── Level_1.pkl
spmohanty's avatar
spmohanty committed
87
88
89
```

* `redis-server` : **NOTE** : Please ensure that you have a redis-server running on localhost.
spmohanty's avatar
spmohanty committed
90
You can find more instructions on how to run redis [here](https://redis.io/topics/quickstart)
spmohanty's avatar
spmohanty committed
91
92
93

* **Run evaluator**

spmohanty's avatar
spmohanty committed
94
95
96
```

# In a separate tab : run local grader
spmohanty's avatar
spmohanty committed
97
flatland-evaluator --tests <path_to_your_tests_directory>
spmohanty's avatar
spmohanty committed
98

spmohanty's avatar
spmohanty committed
99
100
# If you downloaded the files to the location we specified above, then you should be running : 
flatland-evaluator --tests ./scratch/test-envs/
spmohanty's avatar
spmohanty committed
101
102
103
104
105
```

* **Run Agent(s)**

```
spmohanty's avatar
spmohanty committed
106
# In a separate tab :
spmohanty's avatar
spmohanty committed
107
export AICROWD_TESTS_FOLDER=<path_to_your_tests_directory>
spmohanty's avatar
spmohanty committed
108
# or on Windows :
spmohanty's avatar
spmohanty committed
109
110
111
# 
#  SET AICROWD_TESTS_FOLDER=<path_to_your_tests_directory>
python run.py
spmohanty's avatar
spmohanty committed
112
113
114
115
116
117
118
119
```

# How do I specify my software runtime ?

The software runtime is specified by exporting your `conda` env to the root
of your repository by doing :

```
120
121
122
# The included environment.yml is generated by the command below, and you do not need to run it again
# if you did not add any custom dependencies

spmohanty's avatar
spmohanty committed
123
conda env export --no-build > environment.yml
124
125

# Note the `--no-build` flag, which is important if you want your anaconda env to be replicable across all
spmohanty's avatar
spmohanty committed
126
127
128
129
130
```

This `environment.yml` file will be used to recreate the `conda environment` inside the Docker container.
This repository includes an example `environment.yml`

131
132
133
You can specify your software environment by using all the [available configuration options of repo2docker](https://repo2docker.readthedocs.io/en/latest/config_files.html). (But please remember to use [aicrowd-repo2docker](https://pypi.org/project/aicrowd-repo2docker/) to have GPU support)


spmohanty's avatar
spmohanty committed
134
135
136
137
138
139
140
141
142
143
144
145
146
147
# What should my code structure be like ?

Please follow the structure documented in the included [run.py](https://github.com/AIcrowd/flatland-challenge-starter-kit/blob/master/run.py) to adapt
your already existing code to the required structure for this round.

## Important Concepts

### Repository Structure

- `aicrowd.json`
  Each repository should have a `aicrowd.json` with the following content :

```json
{
spmohanty's avatar
spmohanty committed
148
149
  "challenge_id": "aicrowd_flatland_challenge_2019",
  "grader_id": "aicrowd_flatland_challenge_2019",
spmohanty's avatar
spmohanty committed
150
151
152
153
154
155
156
157
158
159
  "authors": ["your-aicrowd-username"],
  "description": "sample description about your awesome agent",
  "license": "MIT",
  "debug": false
}
```

This is used to map your submission to the said challenge, so please remember to use the correct `challenge_id` and `grader_id` as specified above.

If you set `debug` to `true`, then the evaluation will run on a separate set of 20 environments, and the logs from your submitted code (if it fails), will be made available to you to help you debug.
spmohanty's avatar
spmohanty committed
160
**NOTE** : **IMPORTANT** : By default we have set `debug:false`, so when you have done the basic integration testing of your code, and are ready to make a final submission, please do make sure to set `debug` to `true` in `aicrowd.json`.
spmohanty's avatar
spmohanty committed
161
162
163
164
165

### Code Entrypoint

The evaluator will use `/home/aicrowd/run.sh` as the entrypoint, so please remember to have a `run.sh` at the root, which can instantitate any necessary environment variables, and also start executing your actual code. This repository includes a sample `run.sh` file.
If you are using a Dockerfile to specify your software environment, please remember to create a `aicrowd` user, and place the entrypoint code at `run.sh`.
166
If you are unsure what this is all about, you can let `run.sh` be as is, and instead focus on the `run.py` which is being called from within `run.sh`.
spmohanty's avatar
spmohanty committed
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181

## Submission

To make a submission, you will have to create a private repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).

You will have to add your SSH Keys to your GitLab account by following the instructions [here](https://docs.gitlab.com/ee/gitlab-basics/create-your-ssh-keys.html).
If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).

Then you can create a submission by making a _tag push_ to your repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).
**Any tag push (where the tag name begins with "submission-") to your private repository is considered as a submission**  
Then you can add the correct git remote, and finally submit by doing :

```
cd flatland-challenge-starter-kit
# Add AIcrowd git remote endpoint
spmohanty's avatar
spmohanty committed
182
git remote add aicrowd git@gitlab.aicrowd.com:<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit.git
spmohanty's avatar
spmohanty committed
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
git push aicrowd master

# Create a tag for your submission and push
git tag -am "submission-v0.1" submission-v0.1
git push aicrowd master
git push aicrowd submission-v0.1

# Note : If the contents of your repository (latest commit hash) does not change,
# then pushing a new tag will **not** trigger a new evaluation.
```

You now should be able to see the details of your submission at :
[gitlab.aicrowd.com//<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit/issues](gitlab.aicrowd.com//<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit/issues)

**NOTE**: Remember to update your username in the link above :wink:

spmohanty's avatar
spmohanty committed
199
200
In the link above, you should start seeing something like this take shape (the whole evaluation can take a bit of time, so please be a bit patient too :wink: ) :
![](https://i.imgur.com/4HWf1jU.png)
spmohanty's avatar
spmohanty committed
201
202
203
204
205

**Best of Luck** :tada: :tada:

# Author

spmohanty's avatar
spmohanty committed
206
**[Sharada Mohanty](https://twitter.com/MeMohanty)**