Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
=====
Local Evaluation
=====
This document explains you how to locally evaluate your submissions before making
an official submission to the competition.
Requirements
--------------
* **flatland-rl** : We expect that you have `flatland-rl` installed by following the instructions in :doc:`installation`.
* **redis** : Additionally you will also need to have `redis installed <https://redis.io/topics/quickstart>`_ and **should have it running in the background.**
Test Data
--------------
* **test env data** : You can `download and untar the test-env-data <https://www.aicrowd.com/challenges/flatland-challenge/dataset_files>`_,
at a location of your choice, lets say `/path/to/test-env-data/`. After untarring the folder, the folder structure should look something like :
.. code-block:: console
.
└── test-env-data
├── Test_0
│ ├── Level_0.pkl
│ └── Level_1.pkl
├── Test_1
│ ├── Level_0.pkl
│ └── Level_1.pkl
├..................
├..................
├── Test_8
│ ├── Level_0.pkl
│ └── Level_1.pkl
└── Test_9
├── Level_0.pkl
└── Level_1.pkl
Evaluation Service
------------------
* **start evaluation service** : Then you can start the evaluator by running :
.. code-block:: console
flatland-evaluator --tests /path/to/test-env-data/
RemoteClient
------------------
* **run client** : Some `sample submission code can be found in the starter-kit <https://github.com/AIcrowd/flatland-challenge-starter-kit/>`_,
but before you can run your code locally using `FlatlandRemoteClient`, you will have to set the `AICROWD_TESTS_FOLDER` environment variable to the location where you
previous untarred the folder with `the test-env-data`:
.. code-block:: console
export AICROWD_TESTS_FOLDER="/path/to/test-env-data/"
# or on Windows :
#
# set AICROWD_TESTS_FOLDER "\path\to\test-env-data\"
# and then finally run your code
python run.py