update submission readmes

b8288472 · Dipam Chakraborty · af8cd9f3 · b8288472 · b8288472 · b8288472
Commit b8288472 authored Sep 03, 2021 by Dipam Chakraborty
--- a/README.md
+++ b/README.md
-![AIcrowd-Logo](https://raw.githubusercontent.com/AIcrowd/AIcrowd/master/app/assets/images/misc/aicrowd-horizontal.png)
+![](https://i.imgur.com/BznomH1.png)
+<h1 align="center"><a href="https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge">NeurIPS 2021 - AWS Deepracer Challenge</a> - Starter Kit</h1>

-<h1 align="center"><a href="https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge">NeurIPS 2021- AWS Deepracer AI Driving Olympics Challenge</a> - Starter Kit</h1>
+This is the starter kit for the [AWS Deepracer Challenge](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge), a part of AI Driving Olympics at NeurIPS 2021, hosted on [AIcrowd](https://www.aicrowd.com).

+In this competition, you will train a reinforcement learning agent (i.e. an autonomous car), to run on the deepracer simulator. This model will then be tested on a **real world track** with a miniature AWS Deepracer car.
+Your goal is to train a model that can complete a lap as fast as possible without going off track, while avoiding crashing into the objects placed on the track.  

+Clone the repository to compete now!

-[![Discord](https://img.shields.io/discord/565639094860775436.svg)](https://discord.com)
+This repository contains:

+- **Deepracer Gym Environment** which makes it easy to use the deepracer simulator.
+- **Documentation** on how to submit your models to the leaderboard.
+- Information on **best practices** to have hassle free submissions.
+- **Starter code** for you to get started!

-This repository is the AWS Deepracer AI Driving Olympics Challenge **Submission template and Starter kit**! 
+[IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/challenge_rules)

-AWS DeepRacer is an AWS Machine Learning service for exploring reinforcement learning that is focused on autonomous racing.
-In this competition, you will train a reinforcement learning agent (i.e. an autonomous car), that learns to drive by interacting with its environment, e.g., the track, by taking an action in a given state to maximize the expected reward. 
+# Table of contents
+
+- [📚 Competition procedure](#-competition-procedure)
+- [💪 Getting started](#-getting-started)
+- [🛠 Preparing your submission](#-preparing-your-submission)
+  * [Write your agents](#write-your-agents)
+  * [Evaluate your agents locally](#evaluate-your-agents-locally)
+- [📨 Submission](#-submission)
+  * [Repository Structure](#repository-structure)
+  * [Runtime configuration](#runtime-configuration)
+  * [🚀 Submitting to AIcrowd](#-submitting-to-aicrowd)
+    + [`aicrowd.json`](#aicrowdjson)
+    + [Configuring the submission repository](#configuring-the-submission-repository)
+    + [Pushing the code to AIcrowd](#pushing-the-code-to-aicrowd)
+- [📝 Submission checklist](#-submission-checklist)
+- [📎 Important links](#-important-links)
+- [✨ Contributors](#-contributors)
+
+# 📚 Competition procedure
+
+The AWS Deepracer Challenge is an opportunity for participants to test their agents for **simulation to real world transfer**, testing it on a **real world track** with a miniature AWS Deepracer car.
 Your goal is to train a model that can complete a lap as fast as possible without going off track, while avoiding crashing into the objects placed on the track.  

-Clone the repository to compete now!
+In this challenge, you will train your models locally and then upload them to AIcrowd (via git) to be evaluated.

-**This repository contains**:
-*  **Documentation** on how to submit your agent to the leaderboard
-*  **The procedure** for best practices and information on how we evaluate your agent, etc.
-*  **Starter code** for you to get started!
-*  **Baseline**: Baseline Models
+<!-- **The following is a high level description of how this process works.** -->

-[IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/challenge_rules)
+<!-- <div class="tenor-gif-embed" data-postid="22946542" data-share-method="host" data-width="100%" data-aspect-ratio="5.03030303030303"><a href="https://tenor.com/view/gif-22946542">GIF</a> </div><script type="text/javascript" async src="https://tenor.com/embed.js"></script> -->

+1. **Sign up** to join the competition [on the AIcrowd website](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge).
+2. **Clone** this repo and start developing your solution.
+3. **Design and build** your agents that can compete in Deepracer environment and implement an agent class as described in [writing your agents](#write-your-agents) section.
+4. [**Submit**](#-submission) your agents to [AIcrowd Gitlab](https://gitlab.aicrowd.com) for evaluation. [[Refer this for detailed instructions]](#-submission).

-# Table of Contents
+# 💪 Getting started

- [📚 Competition Procedure](#competition-procedure)
- [💪 Setup](#how-to-access-and-use-dataset)
- [🛠 Specify software runtime / dependencies?](#how-do-i-specify-my-software-runtime-dependencies-)
- [🚀 Making a submission](#how-to-make-submission)
- [🤔 Other concepts and FAQs](#other-concepts)
- [📎 Important links](#-important-links)
+> We recommend using `python 3.6` or higher. If you are using Miniconda/Anaconda, you can install it using `conda install python=3.6`

+Clone the starter kit repository and install the dependencies.

-## 📚  Competition Procedure
+```bash
+git clone http://gitlab.aicrowd.com/deepracer/neurips-2021-aws-deepracer-starter-kit.git

+cd neurips-2021-aws-deepracer-starter-kit

-**The following is a high level description of how this round works**
+#Optional: Install Deepracer Gym Environment
+pip install -e ./deepracer-gym
+```

-![](https://i.imgur.com/xzQkwKV.jpg)
+# 🏎 Deepracer Gym Environment

-1. **Sign up** to join the competition [on the AIcrowd website].(https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge)
-2. **Clone** this repo and start developing your solution.
-3. **Train** your models and writer code in `run.py`.
-4. [**Submit**](#how-to-submit-a-model) your trained models to [AIcrowd Gitlab](https://gitlab.aicrowd.com) for evaluation [(full instructions below)](#how-to-submit-a-model). The automated evaluation setup will evaluate the submissions against the test dataset to compute and report the metrics on the leaderboard of the competition.
+Originally, AWS Deepracer is a service hosted with AWS Robomaker platform. To make it easy for partcipants, we are releasing a gym environment for Deepracer. The simulator runs by starting a Docker container that runs the simulator and using a network connection with ZeroMQ server to provide a Gym interface. 

+Run these to quickly get started.

-## 💪 Setup
+```bash
+# Install docker if needed
+sudo snap install docker

+# Install the Deepracer Gym Environment
+pip install -e ./deepracer-gym

-2.  **Clone the repository**
+# Start the Deepracer docker container
+source deepracer-gym/start_deepracer_docker.sh
+# This might take a while to download and start

-    ```
-    git clone git@gitlab.aicrowd.com:neurips-2021-aws-deepracer-ai-driving-olympics-challenge/neurips-2021-aws-deepracer-ai-driving-olympics-challenge-starter-kit.git
-    ```
+# Wait until the terminal says "===Waiting for gym client==="

-3. **Install** competition specific dependencies!
-    ```
-    cd neurips-2021-aws-deepracer-ai-driving-olympics-challenge-starter-kit
-    pip3 install -r requirements.txt
-    ```
+# Open a new terminal
+# Run a random actions agent with Deepracer Gym
+python deepracer-gym/random_actions_example.py

-4. Try out the baseline model available in `run.py`.
+# Stop the docker container once done
+source deepracer-gym/stop_deepracer_docker.sh
+```

+For more instructions look at [`deepracer-gym/README.md`](`deepracer-gym/README.md`)

+# 🛠 Preparing your submission

+## Write your agents

+Your agents need to implement a subclass of [`DeepracerAgent`](agents/deepracer_base_agent.py#L1) class from [`agents/deepracer_base_agent.py`](agents/deepracer_base_agent.py). You can check the code in [`agents`](agents) directory for examples.

-## 🚀 Making a submission
+**Note:** If your agent doesn't inherit the `DeepracerAgent` class, the evaluation will fail.

-### Repository structure
+Once your agent class is ready, you can specify the class to use as the player agent in your [`submission_config.py`](submission_config.py). The starter kit comes with a random agent submission. The [`submission_config.py`](submission_config.py) in the starter kit points to this class. You should update it to use your class.

-Please follow the example structure as it is in the starter kit for the code structure.
-The different files and directories have following meaning:
+# 📨 Submission

-```
-.
-├── aicrowd.json           # Submission meta information - like your username
-├── apt.txt                # Packages to be installed inside docker image
-├── data                   # Your local dataset copy - you don't need to upload it (read DATASET.md)
-├── requirements.txt       # Python packages to be installed
-├── run.py                # IMPORTANT: Your testing/inference phase code, must be derived from AirbornePredictor (example in test.py)
-```
+## Repository Structure
+
+**File/Directory** | **Description**
+--- | ---
+[`agents`](agents) | Directory containing different scripted bots, baseline agent and bots performing random actions. We recommend that you add your agents to this directory.
+[`submission_config.py`](config.py) | File containing the configuration options for local evaluation. We will use the same player agent you specify here during the evaluation.
+[`utils/submit.sh`](utils/submit.sh) | Helper script to submit your repository to [AIcrowd GitLab](https://gitlab.aicrowd.com).
+[`Dockerfile`](Dockerfile) | (Optional) Docker config for your submission. Refer [runtime configuration](#runtime-configuration) for more information.
+[`requirements.txt`](requirements.txt) | File containing the list of python packages you want to install for the submission to run. Refer [runtime configuration](#runtime-configuration) for more information.
+[`apt.txt`](apt.txt) | File containing the list of packages you want to install for submission to run. Refer [runtime configuration](#runtime-configuration) for more information.

-### Specify runtime/dependencies
+## Runtime configuration

-We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.
+You can specify the list of python packages needed for your code to run in your [`requirements.txt`](requirements.txt) file. We will install the packages using `pip install` command.

-The configuration files typically include `requirements.txt` (pypi packages), `environment.yml` (conda environment), `apt.txt` (apt packages) or even your own `Dockerfile`.
+You can also specify the OS packages needed using [`apt.txt`](apt.txt) file. We install these packages using `apt-get install` command.

-You can check detailed information about the same in the 👉 [RUNTIME.md](/docs/RUNTIME.md) file.
+For more information on how you can configure the evaluation runtime, please refer [`RUNTIME.md`](docs/RUNTIME.md).

-### Submitting to aicrowd
+## 🚀 Submitting to AIcrowd

- **Add your SSH key** to AIcrowd GitLab
+### **Add your SSH key** to AIcrowd GitLab

 You can add your SSH Keys to your GitLab account by going to your profile settings [here](https://gitlab.aicrowd.com/profile/keys). If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).

- Your repository should have an aicrowd.json file with following fields:
+
+### `aicrowd.json`
+
+Your repository should have an `aicrowd.json` file with following fields:

 ```json
 {
-  "challenge_id": "evaluations-api-deepracer",
-  "grader_id": "evaluations-api-deepracer",
-  "authors": ["aicrowd-bot"],
-  "tags": "change-me",
-  "description": "Random agent for AWS Deep Racer",
+    "challenge_id" : "neurips-2021-aws-deepracer-ai-driving-olympics-challenge",
+    "authors" : ["Your Name"],
+    "description" : "Brief description for your submission"
 }
 ```

+This file is used to identify your submission as a part of the AWS Deepracer Challenge. You must use the `challenge_id` as specified above.

-This JSON is used to map your submission to the challenge - so please remember to use the correct `challenge_id` as specified above.
-
- Follow the instructions in [SUBMISSION.md](/docs/SUBMISSION.md) to get your submission evaluated.
+### Configuring the submission repository

+```bash
+git remote add aicrowd git@gitlab.aicrowd.com:<username>/neurips-2021-aws-deepracer-starter-kit.git
+```
+**Note**: The above step needs to be done only once. This configuration will be saved in your repository for future use.

-# 🤔 Other Concepts
-
-## Time constraints
-
-You need to make sure that your model finishes evaluation in 1500 seconds, otherwise your evaluation will be marked failed.
-
-## Local evaluation
+### Pushing the code to AIcrowd for evaluation

-You can also test end to end evaluation on your own systems, by executing `run.py`.
+```bash
+./utils/submit.sh "some description"
+```

-## Hardware used for evaluations
+If you want to submit without the helper script, please refer [`SUBMISSION.md`](docs/SUBMISSION.md).

-We use g4dn instances to run your evaluations.

+# 📝 Submission checklist

+- [x] **Accept the challenge rules**. You can do this by going to the [challenge overview page](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge) and clicking the "Participate" button. You only need to do this once.
+- [x] **Add your agent code** that implements the `DeepracerAgent` class from `evaluator/base_agent`. 
+- [x] **Add your model checkpoints** (if any) to the repo. The `utils/submit.sh` will automatically detect large files and add them to git LFS. If you are using the script, please refer to [this post explaining how to add your models](https://discourse.aicrowd.com/t/how-to-upload-large-files-size-to-your-submission/2304).
+- [x] **Update runtime configuration** using `requirements.txt`, `apt.txt` and/or `Dockerfile` as necessary. Please make sure that you specified the same package versions that you use locally on your machine.

 # 📎 Important links

+- 💪 Challenge information
+   * [Challenge page](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge)
+   * [Leaderboard](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/leaderboards)
+ - 🗣 Community
+    * [Challenge discussion forum](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/discussion)
+- 🎮 Deepracer resources
+    * [Deepracer documentation](https://docs.aws.amazon.com/deepracer/latest/developerguide/what-is-deepracer.html)
+    * [Deepracer homepage](https://aws.amazon.com/deepracer/)
+    
+
+# ✨ Contributors

-💪 &nbsp;Challenge Page: https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge
+- [Dipam Chakraborty](https://www.aicrowd.com/participants/dipam)
+- [Siddhartha Laghuvarapu](https://www.aicrowd.com/participants/siddhartha)
+- [Jyotish Poonganam](https://www.aicrowd.com/participants/jyotish)
+- Sahika Genc

-🗣️ &nbsp;Discussion Forum: https://www.aicrowd.com/challengesneurips-2021-aws-deepracer-ai-driving-olympics-challengee/discussion

-🏆 &nbsp;Leaderboard: https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/leaderboards
+**Best of Luck** 🎉 
--- a/aicrowd.json
+++ b/aicrowd.json
 {
  "challenge_id": "neurips-2021-aws-deepracer-ai-driving-olympics-challenge",
  "authors": ["aicrowd-bot"],
-  "tags": "change-me",
  "description": "Random agent for AWS Deep Racer"
 }

--- a/apt.txt
+++ b/apt.txt
+gcc
+g++
+git
\ No newline at end of file
--- a/docs/RUNTIME.md
+++ b/docs/RUNTIME.md
+# Adding your runtime
+
+## **Installation setup steps**
+
+### **How to specify your installation setup for the submission**
+The entrypoint to the installation is the Dockerfile
+The Dockerfile provided will have commands to install apt packages from apt.txt and pip packages from requirements.txt
+You are strongly advised to specify the version of the library that you use to use for your submission
+
+Examples
+
+For requirements.txt
+
+```torch==1.8.1```
+
+You can add git repositories to requirements.txt as well
+
+```git+https://github.com/tqdm/tqdm.git@devel#egg=tqdm```
+
+For apt.txt
+
+```firefox=45.0.2+build1-0ubuntu1```
+
+## (Optional) Check your installation setup on your own machine
+
+### Setup the environment 
+Install docker on your machine and run
+
+```bash
+docker build -t your-custom-tag . 
+```
+
+If you get installation errors during the above step, your submission is likely to fail, please review the errors and fix the installation
+
+### Run the submission locally
+
+Run the docker container This will create an environment that emulates how your submission environment will be.
+
+```bash
+docker run -it your-custom-tag /bin/bash
+```
+
+```bash
+python rollout.py 
+```
+
+If you get runtime errors during the above step, your submission is likely to fail. Please review the errors.
+A common error is not specifying one or multiple required libraries
+
+## Installation FAQ
+- How to install with an `environment.yml`?
+
+  - Add `environment.yml` to the base of your repo. You also need to add commands to add `environment.yml` to the `Dockerfile`. Afterwards, You’re encouraged to follow the above steps Check your installation setup on your own machine to check everything is properly installed.
+
+- How do I install with a setup.py
+  - You need to add the command to run it in the ```Dockerfile``` - ```RUN pip install .```
+
+- What’s the package versions I have installed on my machine?
+
+  - You can find the versions of the python package installations you currently have using pip freeze
+
+- What’s aicrowd_gym
+  - AIcrowd gym is a library AIcrowd uses to limit the functionality on the submission environment to prevent tampering with the RL Library. For all intents and purposes, aicrowd_gym should work the same as gym for you.
+- How do I use a custom script to install?
+  - You’ll need to call the custom script in the ```Dockerfile```, example you can add this line ```RUN ./custom_script.sh```
\ No newline at end of file
--- a/docs/SUBMISSION.md
+++ b/docs/SUBMISSION.md
+# Making submission
+
+This file will help you in making your first submission.
+
+
+## Submission Entrypoint (where you write your code!)
+
+The evaluator will import the `submission_agent` from `submission_config.py` for generating actions in the deepracer environment. 
+
+## IMPORTANT: Saving Models before submission!
+
+Before you submit make sure that you have saved your models, which are needed by your inference code.
+In case your files are larger in size you can use `git-lfs` to upload them. More details [here](https://discourse.aicrowd.com/t/how-to-upload-large-files-size-to-your-submission/2304).
+
+## How to submit a trained model!
+
+To make a submission, you will have to create a **private** repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).
+
+You will have to add your SSH Keys to your GitLab account by going to your profile settings [here](https://gitlab.aicrowd.com/profile/keys). If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).
+
+Then you can create a submission by making a _tag push_ to your repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).
+**Any tag push (where the tag name begins with "submission-") to your private repository is considered as a submission**  
+Then you can add the correct git remote, and finally submit by doing :
+
+```
+cd neurips-2021-aws-deepracer-starter-kit
+# Add AIcrowd git remote endpoint
+git remote add aicrowd git@gitlab.aicrowd.com:<username>/neurips-2021-aws-deepracer-starter-kit.git
+git push aicrowd master
+```
+
+```
+# Create a tag for your submission and push
+git tag -am "submission-v0.1" submission-v0.1
+git push aicrowd master
+git push aicrowd submission-v0.1
+
+# Note : If the contents of your repository (latest commit hash) does not change,
+# then pushing a new tag will **not** trigger a new evaluation.
+```
+
+You now should be able to see the details of your submission at :
+[gitlab.aicrowd.com/<YOUR_AICROWD_USER_NAME>/neurips-2021-aws-deepracer-starter-kit/issues](https://gitlab.aicrowd.com//<YOUR_AICROWD_USER_NAME>/neurips-2021-aws-deepracer-starter-kit/issues)
+
+**NOTE**: Remember to update your username instead of `<YOUR_AICROWD_USER_NAME>` above.
+
+### Other helpful files
+
+👉 [RUNTIME.md](/docs/RUNTIME.md)
--- a/requirements.txt
+++ b/requirements.txt
+numpy==1.21.2
\ No newline at end of file
--- a/utils/logging.sh
+++ b/utils/logging.sh
+#!/bin/bash
+
+log_info() {
+  echo -e "\033[0;36m$@\033[0m"
+}
+
+log_success() {
+  echo -e "\033[0;32m$@\033[0m"
+}
+
+log_normal() {
+  echo -e "$@"
+}
+
+log_error() {
+  echo -e "\033[0;31m$@\033[0m"
+}
+
--- a/utils/submit.sh
+++ b/utils/submit.sh
+#!/bin/bash
+
+set -e
+
+REPO_ROOT_DIR=$(git rev-parse --show-toplevel)
+SCRIPTS_DIR="${REPO_ROOT_DIR}/utils"
+
+source "${SCRIPTS_DIR}/logging.sh"
+
+
+print_usage() {
+cat << USAGE
+Usage: ./utils/submit.sh "impala-ppo-v0.1"
+USAGE
+}
+
+
+bad_remote_message() {
+  log_normal "AIcrowd remote not found"
+  log_error "Please run \`git remote add aicrowd git@gitlab.aicrowd.com:<username>/<repo>.git\` and rerun this command"
+  exit 1
+}
+
+get_submission_remote() {
+  bad_remotes=(
+    git@gitlab.aicrowd.com:deepracer/neurips-2021-aws-deepracer-starter-kit.git
+    http://gitlab.aicrowd.com/deepracer/neurips-2021-aws-deepracer-starter-kit.git
+  )
+  submission_remote=""
+
+  for remote in $(git remote); do
+    remote=$(git remote get-url $remote)
+    if [[ ! "$remote" =~ "$bad_remotes" ]] && echo $remote | grep "gitlab.aicrowd.com" > /dev/null; then
+      submission_remote=$remote
+    fi
+  done
+
+  if [[ "$submission_remote" == "" ]]; then
+    bad_remote_message
+  fi
+
+  echo $submission_remote
+}
+
+check_remote() {
+  log_info Checking git remote settings...
+  log_normal Using $(get_submission_remote) as the submission repository
+}
+
+
+setup_lfs() {
+  git lfs install
+  HTTPS_REMOTE=$(git remote -v | grep gitlab.aicrowd.com | head -1 | awk '{print $2}' | sed 's|git@gitlab.aicrowd.com:|https://gitlab.aicrowd.com|g')
+  git config lfs.$HTTPS_REMOTE/info/lfs.locksverify false
+  find . -type f -size +5M -exec git lfs track {} &> /dev/null \;
+  git add .gitattributes
+}
+
+
+setup_commits() {
+  REMOTE=$(get_submission_remote)
+  TAG=$(echo "$@" | sed 's/ /-/g')
+  git add --all
+  git commit -m "Changes for submission-$TAG" || true  # don't exit when no new commits are there
+  git tag -am "submission-$TAG" "submission-$TAG" || (log_error "There is another submission with the same description. Please give a different description." && exit 1)
+  git push -f $REMOTE master
+  git push -f $REMOTE "submission-$TAG"
+}
+
+
+submit() {
+  check_remote
+  setup_lfs "$@"
+  setup_commits "$@"
+}
+  
+
+
+if [[ $# -lt 1 ]]; then
+  print_usage
+  exit 1
+fi
+
+submit "$@"