Commit b8288472 authored by Dipam Chakraborty's avatar Dipam Chakraborty
Browse files

update submission readmes

parent af8cd9f3
![AIcrowd-Logo](https://raw.githubusercontent.com/AIcrowd/AIcrowd/master/app/assets/images/misc/aicrowd-horizontal.png)
![](https://i.imgur.com/BznomH1.png)
<h1 align="center"><a href="https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge">NeurIPS 2021 - AWS Deepracer Challenge</a> - Starter Kit</h1>
<h1 align="center"><a href="https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge">NeurIPS 2021- AWS Deepracer AI Driving Olympics Challenge</a> - Starter Kit</h1>
This is the starter kit for the [AWS Deepracer Challenge](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge), a part of AI Driving Olympics at NeurIPS 2021, hosted on [AIcrowd](https://www.aicrowd.com).
In this competition, you will train a reinforcement learning agent (i.e. an autonomous car), to run on the deepracer simulator. This model will then be tested on a **real world track** with a miniature AWS Deepracer car.
Your goal is to train a model that can complete a lap as fast as possible without going off track, while avoiding crashing into the objects placed on the track.
Clone the repository to compete now!
[![Discord](https://img.shields.io/discord/565639094860775436.svg)](https://discord.com)
This repository contains:
- **Deepracer Gym Environment** which makes it easy to use the deepracer simulator.
- **Documentation** on how to submit your models to the leaderboard.
- Information on **best practices** to have hassle free submissions.
- **Starter code** for you to get started!
This repository is the AWS Deepracer AI Driving Olympics Challenge **Submission template and Starter kit**!
[IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/challenge_rules)
AWS DeepRacer is an AWS Machine Learning service for exploring reinforcement learning that is focused on autonomous racing.
In this competition, you will train a reinforcement learning agent (i.e. an autonomous car), that learns to drive by interacting with its environment, e.g., the track, by taking an action in a given state to maximize the expected reward.
# Table of contents
- [📚 Competition procedure](#-competition-procedure)
- [💪 Getting started](#-getting-started)
- [🛠 Preparing your submission](#-preparing-your-submission)
* [Write your agents](#write-your-agents)
* [Evaluate your agents locally](#evaluate-your-agents-locally)
- [📨 Submission](#-submission)
* [Repository Structure](#repository-structure)
* [Runtime configuration](#runtime-configuration)
* [🚀 Submitting to AIcrowd](#-submitting-to-aicrowd)
+ [`aicrowd.json`](#aicrowdjson)
+ [Configuring the submission repository](#configuring-the-submission-repository)
+ [Pushing the code to AIcrowd](#pushing-the-code-to-aicrowd)
- [📝 Submission checklist](#-submission-checklist)
- [📎 Important links](#-important-links)
- [✨ Contributors](#-contributors)
# 📚 Competition procedure
The AWS Deepracer Challenge is an opportunity for participants to test their agents for **simulation to real world transfer**, testing it on a **real world track** with a miniature AWS Deepracer car.
Your goal is to train a model that can complete a lap as fast as possible without going off track, while avoiding crashing into the objects placed on the track.
Clone the repository to compete now!
In this challenge, you will train your models locally and then upload them to AIcrowd (via git) to be evaluated.
**This repository contains**:
* **Documentation** on how to submit your agent to the leaderboard
* **The procedure** for best practices and information on how we evaluate your agent, etc.
* **Starter code** for you to get started!
* **Baseline**: Baseline Models
<!-- **The following is a high level description of how this process works.** -->
[IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/challenge_rules)
<!-- <div class="tenor-gif-embed" data-postid="22946542" data-share-method="host" data-width="100%" data-aspect-ratio="5.03030303030303"><a href="https://tenor.com/view/gif-22946542">GIF</a> </div><script type="text/javascript" async src="https://tenor.com/embed.js"></script> -->
1. **Sign up** to join the competition [on the AIcrowd website](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge).
2. **Clone** this repo and start developing your solution.
3. **Design and build** your agents that can compete in Deepracer environment and implement an agent class as described in [writing your agents](#write-your-agents) section.
4. [**Submit**](#-submission) your agents to [AIcrowd Gitlab](https://gitlab.aicrowd.com) for evaluation. [[Refer this for detailed instructions]](#-submission).
# Table of Contents
# 💪 Getting started
- [📚 Competition Procedure](#competition-procedure)
- [💪 Setup](#how-to-access-and-use-dataset)
- [🛠 Specify software runtime / dependencies?](#how-do-i-specify-my-software-runtime-dependencies-)
- [🚀 Making a submission](#how-to-make-submission)
- [🤔 Other concepts and FAQs](#other-concepts)
- [📎 Important links](#-important-links)
> We recommend using `python 3.6` or higher. If you are using Miniconda/Anaconda, you can install it using `conda install python=3.6`
Clone the starter kit repository and install the dependencies.
## 📚 Competition Procedure
```bash
git clone http://gitlab.aicrowd.com/deepracer/neurips-2021-aws-deepracer-starter-kit.git
cd neurips-2021-aws-deepracer-starter-kit
**The following is a high level description of how this round works**
#Optional: Install Deepracer Gym Environment
pip install -e ./deepracer-gym
```
![](https://i.imgur.com/xzQkwKV.jpg)
# 🏎 Deepracer Gym Environment
1. **Sign up** to join the competition [on the AIcrowd website].(https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge)
2. **Clone** this repo and start developing your solution.
3. **Train** your models and writer code in `run.py`.
4. [**Submit**](#how-to-submit-a-model) your trained models to [AIcrowd Gitlab](https://gitlab.aicrowd.com) for evaluation [(full instructions below)](#how-to-submit-a-model). The automated evaluation setup will evaluate the submissions against the test dataset to compute and report the metrics on the leaderboard of the competition.
Originally, AWS Deepracer is a service hosted with AWS Robomaker platform. To make it easy for partcipants, we are releasing a gym environment for Deepracer. The simulator runs by starting a Docker container that runs the simulator and using a network connection with ZeroMQ server to provide a Gym interface.
Run these to quickly get started.
## 💪 Setup
```bash
# Install docker if needed
sudo snap install docker
# Install the Deepracer Gym Environment
pip install -e ./deepracer-gym
2. **Clone the repository**
# Start the Deepracer docker container
source deepracer-gym/start_deepracer_docker.sh
# This might take a while to download and start
```
git clone git@gitlab.aicrowd.com:neurips-2021-aws-deepracer-ai-driving-olympics-challenge/neurips-2021-aws-deepracer-ai-driving-olympics-challenge-starter-kit.git
```
# Wait until the terminal says "===Waiting for gym client==="
3. **Install** competition specific dependencies!
```
cd neurips-2021-aws-deepracer-ai-driving-olympics-challenge-starter-kit
pip3 install -r requirements.txt
```
# Open a new terminal
# Run a random actions agent with Deepracer Gym
python deepracer-gym/random_actions_example.py
4. Try out the baseline model available in `run.py`.
# Stop the docker container once done
source deepracer-gym/stop_deepracer_docker.sh
```
For more instructions look at [`deepracer-gym/README.md`](`deepracer-gym/README.md`)
# 🛠 Preparing your submission
## Write your agents
Your agents need to implement a subclass of [`DeepracerAgent`](agents/deepracer_base_agent.py#L1) class from [`agents/deepracer_base_agent.py`](agents/deepracer_base_agent.py). You can check the code in [`agents`](agents) directory for examples.
## 🚀 Making a submission
**Note:** If your agent doesn't inherit the `DeepracerAgent` class, the evaluation will fail.
### Repository structure
Once your agent class is ready, you can specify the class to use as the player agent in your [`submission_config.py`](submission_config.py). The starter kit comes with a random agent submission. The [`submission_config.py`](submission_config.py) in the starter kit points to this class. You should update it to use your class.
Please follow the example structure as it is in the starter kit for the code structure.
The different files and directories have following meaning:
# 📨 Submission
```
.
├── aicrowd.json # Submission meta information - like your username
├── apt.txt # Packages to be installed inside docker image
├── data # Your local dataset copy - you don't need to upload it (read DATASET.md)
├── requirements.txt # Python packages to be installed
├── run.py # IMPORTANT: Your testing/inference phase code, must be derived from AirbornePredictor (example in test.py)
```
## Repository Structure
**File/Directory** | **Description**
--- | ---
[`agents`](agents) | Directory containing different scripted bots, baseline agent and bots performing random actions. We recommend that you add your agents to this directory.
[`submission_config.py`](config.py) | File containing the configuration options for local evaluation. We will use the same player agent you specify here during the evaluation.
[`utils/submit.sh`](utils/submit.sh) | Helper script to submit your repository to [AIcrowd GitLab](https://gitlab.aicrowd.com).
[`Dockerfile`](Dockerfile) | (Optional) Docker config for your submission. Refer [runtime configuration](#runtime-configuration) for more information.
[`requirements.txt`](requirements.txt) | File containing the list of python packages you want to install for the submission to run. Refer [runtime configuration](#runtime-configuration) for more information.
[`apt.txt`](apt.txt) | File containing the list of packages you want to install for submission to run. Refer [runtime configuration](#runtime-configuration) for more information.
### Specify runtime/dependencies
## Runtime configuration
We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.
You can specify the list of python packages needed for your code to run in your [`requirements.txt`](requirements.txt) file. We will install the packages using `pip install` command.
The configuration files typically include `requirements.txt` (pypi packages), `environment.yml` (conda environment), `apt.txt` (apt packages) or even your own `Dockerfile`.
You can also specify the OS packages needed using [`apt.txt`](apt.txt) file. We install these packages using `apt-get install` command.
You can check detailed information about the same in the 👉 [RUNTIME.md](/docs/RUNTIME.md) file.
For more information on how you can configure the evaluation runtime, please refer [`RUNTIME.md`](docs/RUNTIME.md).
### Submitting to aicrowd
## 🚀 Submitting to AIcrowd
- **Add your SSH key** to AIcrowd GitLab
### **Add your SSH key** to AIcrowd GitLab
You can add your SSH Keys to your GitLab account by going to your profile settings [here](https://gitlab.aicrowd.com/profile/keys). If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).
- Your repository should have an aicrowd.json file with following fields:
### `aicrowd.json`
Your repository should have an `aicrowd.json` file with following fields:
```json
{
"challenge_id": "evaluations-api-deepracer",
"grader_id": "evaluations-api-deepracer",
"authors": ["aicrowd-bot"],
"tags": "change-me",
"description": "Random agent for AWS Deep Racer",
"challenge_id" : "neurips-2021-aws-deepracer-ai-driving-olympics-challenge",
"authors" : ["Your Name"],
"description" : "Brief description for your submission"
}
```
This file is used to identify your submission as a part of the AWS Deepracer Challenge. You must use the `challenge_id` as specified above.
This JSON is used to map your submission to the challenge - so please remember to use the correct `challenge_id` as specified above.
- Follow the instructions in [SUBMISSION.md](/docs/SUBMISSION.md) to get your submission evaluated.
### Configuring the submission repository
```bash
git remote add aicrowd git@gitlab.aicrowd.com:<username>/neurips-2021-aws-deepracer-starter-kit.git
```
**Note**: The above step needs to be done only once. This configuration will be saved in your repository for future use.
# 🤔 Other Concepts
## Time constraints
You need to make sure that your model finishes evaluation in 1500 seconds, otherwise your evaluation will be marked failed.
## Local evaluation
### Pushing the code to AIcrowd for evaluation
You can also test end to end evaluation on your own systems, by executing `run.py`.
```bash
./utils/submit.sh "some description"
```
## Hardware used for evaluations
If you want to submit without the helper script, please refer [`SUBMISSION.md`](docs/SUBMISSION.md).
We use g4dn instances to run your evaluations.
# 📝 Submission checklist
- [x] **Accept the challenge rules**. You can do this by going to the [challenge overview page](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge) and clicking the "Participate" button. You only need to do this once.
- [x] **Add your agent code** that implements the `DeepracerAgent` class from `evaluator/base_agent`.
- [x] **Add your model checkpoints** (if any) to the repo. The `utils/submit.sh` will automatically detect large files and add them to git LFS. If you are using the script, please refer to [this post explaining how to add your models](https://discourse.aicrowd.com/t/how-to-upload-large-files-size-to-your-submission/2304).
- [x] **Update runtime configuration** using `requirements.txt`, `apt.txt` and/or `Dockerfile` as necessary. Please make sure that you specified the same package versions that you use locally on your machine.
# 📎 Important links
- 💪 Challenge information
* [Challenge page](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge)
* [Leaderboard](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/leaderboards)
- 🗣 Community
* [Challenge discussion forum](https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/discussion)
- 🎮 Deepracer resources
* [Deepracer documentation](https://docs.aws.amazon.com/deepracer/latest/developerguide/what-is-deepracer.html)
* [Deepracer homepage](https://aws.amazon.com/deepracer/)
# ✨ Contributors
💪 &nbsp;Challenge Page: https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge
- [Dipam Chakraborty](https://www.aicrowd.com/participants/dipam)
- [Siddhartha Laghuvarapu](https://www.aicrowd.com/participants/siddhartha)
- [Jyotish Poonganam](https://www.aicrowd.com/participants/jyotish)
- Sahika Genc
🗣️ &nbsp;Discussion Forum: https://www.aicrowd.com/challengesneurips-2021-aws-deepracer-ai-driving-olympics-challengee/discussion
🏆 &nbsp;Leaderboard: https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge/leaderboards
**Best of Luck** 🎉
{
"challenge_id": "neurips-2021-aws-deepracer-ai-driving-olympics-challenge",
"authors": ["aicrowd-bot"],
"tags": "change-me",
"description": "Random agent for AWS Deep Racer"
}
gcc
g++
git
\ No newline at end of file
# Adding your runtime
## **Installation setup steps**
### **How to specify your installation setup for the submission**
The entrypoint to the installation is the Dockerfile
The Dockerfile provided will have commands to install apt packages from apt.txt and pip packages from requirements.txt
You are strongly advised to specify the version of the library that you use to use for your submission
Examples
For requirements.txt
```torch==1.8.1```
You can add git repositories to requirements.txt as well
```git+https://github.com/tqdm/tqdm.git@devel#egg=tqdm```
For apt.txt
```firefox=45.0.2+build1-0ubuntu1```
## (Optional) Check your installation setup on your own machine
### Setup the environment
Install docker on your machine and run
```bash
docker build -t your-custom-tag .
```
If you get installation errors during the above step, your submission is likely to fail, please review the errors and fix the installation
### Run the submission locally
Run the docker container This will create an environment that emulates how your submission environment will be.
```bash
docker run -it your-custom-tag /bin/bash
```
```bash
python rollout.py
```
If you get runtime errors during the above step, your submission is likely to fail. Please review the errors.
A common error is not specifying one or multiple required libraries
## Installation FAQ
- How to install with an `environment.yml`?
- Add `environment.yml` to the base of your repo. You also need to add commands to add `environment.yml` to the `Dockerfile`. Afterwards, You’re encouraged to follow the above steps Check your installation setup on your own machine to check everything is properly installed.
- How do I install with a setup.py
- You need to add the command to run it in the ```Dockerfile``` - ```RUN pip install .```
- What’s the package versions I have installed on my machine?
- You can find the versions of the python package installations you currently have using pip freeze
- What’s aicrowd_gym
- AIcrowd gym is a library AIcrowd uses to limit the functionality on the submission environment to prevent tampering with the RL Library. For all intents and purposes, aicrowd_gym should work the same as gym for you.
- How do I use a custom script to install?
- You’ll need to call the custom script in the ```Dockerfile```, example you can add this line ```RUN ./custom_script.sh```
\ No newline at end of file
# Making submission
This file will help you in making your first submission.
## Submission Entrypoint (where you write your code!)
The evaluator will import the `submission_agent` from `submission_config.py` for generating actions in the deepracer environment.
## IMPORTANT: Saving Models before submission!
Before you submit make sure that you have saved your models, which are needed by your inference code.
In case your files are larger in size you can use `git-lfs` to upload them. More details [here](https://discourse.aicrowd.com/t/how-to-upload-large-files-size-to-your-submission/2304).
## How to submit a trained model!
To make a submission, you will have to create a **private** repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).
You will have to add your SSH Keys to your GitLab account by going to your profile settings [here](https://gitlab.aicrowd.com/profile/keys). If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).
Then you can create a submission by making a _tag push_ to your repository on [https://gitlab.aicrowd.com/](https://gitlab.aicrowd.com/).
**Any tag push (where the tag name begins with "submission-") to your private repository is considered as a submission**
Then you can add the correct git remote, and finally submit by doing :
```
cd neurips-2021-aws-deepracer-starter-kit
# Add AIcrowd git remote endpoint
git remote add aicrowd git@gitlab.aicrowd.com:<username>/neurips-2021-aws-deepracer-starter-kit.git
git push aicrowd master
```
```
# Create a tag for your submission and push
git tag -am "submission-v0.1" submission-v0.1
git push aicrowd master
git push aicrowd submission-v0.1
# Note : If the contents of your repository (latest commit hash) does not change,
# then pushing a new tag will **not** trigger a new evaluation.
```
You now should be able to see the details of your submission at :
[gitlab.aicrowd.com/<YOUR_AICROWD_USER_NAME>/neurips-2021-aws-deepracer-starter-kit/issues](https://gitlab.aicrowd.com//<YOUR_AICROWD_USER_NAME>/neurips-2021-aws-deepracer-starter-kit/issues)
**NOTE**: Remember to update your username instead of `<YOUR_AICROWD_USER_NAME>` above.
### Other helpful files
👉 [RUNTIME.md](/docs/RUNTIME.md)
numpy==1.21.2
\ No newline at end of file
#!/bin/bash
log_info() {
echo -e "\033[0;36m$@\033[0m"
}
log_success() {
echo -e "\033[0;32m$@\033[0m"
}
log_normal() {
echo -e "$@"
}
log_error() {
echo -e "\033[0;31m$@\033[0m"
}
#!/bin/bash
set -e
REPO_ROOT_DIR=$(git rev-parse --show-toplevel)
SCRIPTS_DIR="${REPO_ROOT_DIR}/utils"
source "${SCRIPTS_DIR}/logging.sh"
print_usage() {
cat << USAGE
Usage: ./utils/submit.sh "impala-ppo-v0.1"
USAGE
}
bad_remote_message() {
log_normal "AIcrowd remote not found"
log_error "Please run \`git remote add aicrowd git@gitlab.aicrowd.com:<username>/<repo>.git\` and rerun this command"
exit 1
}
get_submission_remote() {
bad_remotes=(
git@gitlab.aicrowd.com:deepracer/neurips-2021-aws-deepracer-starter-kit.git
http://gitlab.aicrowd.com/deepracer/neurips-2021-aws-deepracer-starter-kit.git
)
submission_remote=""
for remote in $(git remote); do
remote=$(git remote get-url $remote)
if [[ ! "$remote" =~ "$bad_remotes" ]] && echo $remote | grep "gitlab.aicrowd.com" > /dev/null; then
submission_remote=$remote
fi
done
if [[ "$submission_remote" == "" ]]; then
bad_remote_message
fi
echo $submission_remote
}
check_remote() {
log_info Checking git remote settings...
log_normal Using $(get_submission_remote) as the submission repository
}
setup_lfs() {
git lfs install
HTTPS_REMOTE=$(git remote -v | grep gitlab.aicrowd.com | head -1 | awk '{print $2}' | sed 's|git@gitlab.aicrowd.com:|https://gitlab.aicrowd.com|g')
git config lfs.$HTTPS_REMOTE/info/lfs.locksverify false
find . -type f -size +5M -exec git lfs track {} &> /dev/null \;
git add .gitattributes
}
setup_commits() {
REMOTE=$(get_submission_remote)
TAG=$(echo "$@" | sed 's/ /-/g')
git add --all
git commit -m "Changes for submission-$TAG" || true # don't exit when no new commits are there
git tag -am "submission-$TAG" "submission-$TAG" || (log_error "There is another submission with the same description. Please give a different description." && exit 1)
git push -f $REMOTE master
git push -f $REMOTE "submission-$TAG"
}
submit() {
check_remote
setup_lfs "$@"
setup_commits "$@"
}
if [[ $# -lt 1 ]]; then
print_usage
exit 1
fi
submit "$@"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment