Skip to content
Snippets Groups Projects
Commit 07c2a067 authored by Dipam Chakraborty's avatar Dipam Chakraborty
Browse files

Merge branch 'dipam/updates' into 'master' 

Fix TOC, change images, other minor changes

See merge request !11
parents 7ec8d594 7cb833b1
No related branches found
No related tags found
2 merge requests!14Merge from Master,!11Fix TOC, change images, other minor changes
Hide whitespace changes
Inline Side-by-side
README.md
+ 38
− 231
View file @ 07c2a067
![Nethack Banner](https://raw.githubusercontent.com/facebookresearch/nle/master/dat/nle/logo.png) ![Nethack Banner](https://images.aicrowd.com/raw_images/challenges/banner_file/423/05645091231d5b0610f4.jpg)
# **NeurIPS 2021 - The NetHack Challenge** - Starter Kit # **NeurIPS 2021 - The NetHack Challenge** - Starter Kit
...@@ -11,7 +11,7 @@ This repository is the Nethack Challenge **Starter kit**! It contains: ...@@ -11,7 +11,7 @@ This repository is the Nethack Challenge **Starter kit**! It contains:
Quick Links: Quick Links:
* [The NetHack Challenge - Competition Page](https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge) * [The NetHack Challenge - Competition Page](https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge)
* [The NetHack Challenge - Discord Server](https://discord.gg/zkFWQmSWBA) * [The NetHack Challenge - Discord Server](https://discord.gg/zkFWQmSWBA)
* [The NetHack Challenge - Starter Kit](https://gitlab.aicrowd.com/nethack/neurips-2021-the-nethack-challenge) * [The NetHack Challenge - Starter Kit](https://gitlab.aicrowd.com/nethack/neurips-2021-the-nethack-challenge)
* [IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge/challenge_rules) * [IMPORTANT - Accept the rules before you submit](https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge/challenge_rules)
...@@ -19,10 +19,10 @@ Quick Links: ...@@ -19,10 +19,10 @@ Quick Links:
# Table of Contents # Table of Contents
1. [Intro to Nethack and the Nethack Challenge](#intro-to-nethack-and-the-nethack-challenge) 1. [Intro to Nethack and the Nethack Challenge](#intro-to-nethack-and-the-nethack-challenge)
2. [Setting up your codebase](setting-up-your-codebase) 2. [Setting up your codebase](#setting-up-your-codebase)
3. [Baselines](baselines) 3. [Baselines](#baselines)
4. [How to test and debug locally](how-to-test-and-debug-locally) 4. [How to test and debug locally](#how-to-test-and-debug-locally)
5. [How to submit](how-to-submit) 5. [How to submit](#how-to-submit)
# Intro to Nethack and the Nethack Challenge # Intro to Nethack and the Nethack Challenge
...@@ -43,7 +43,7 @@ will evaluate the submissions against the NLE environment for a fixed ...@@ -43,7 +43,7 @@ will evaluate the submissions against the NLE environment for a fixed
number of rollouts to compute and report the metrics on the leaderboard number of rollouts to compute and report the metrics on the leaderboard
of the competition. of the competition.
![](https://i.imgur.com/xzQkwKV.jpg) ![](https://images.aicrowd.com/raw_images/challenges/banner_file/423/5f69010437d25bf569c4.jpg)
# Setting Up Your Codebase # Setting Up Your Codebase
...@@ -79,7 +79,7 @@ We accept submissions with custom runtimes, so you can choose your ...@@ -79,7 +79,7 @@ We accept submissions with custom runtimes, so you can choose your
favorite! The configuration files typically include `requirements.txt` favorite! The configuration files typically include `requirements.txt`
(pypi packages), `apt.txt` (apt packages) or even your own `Dockerfile`. (pypi packages), `apt.txt` (apt packages) or even your own `Dockerfile`.
You can check detailed information about the same in the `RUNTIME.md` file. You can check detailed information about the same in the [RUNTIME.md](/docs/RUNTIME.md) file.
**What should my code structure look like?** **What should my code structure look like?**
...@@ -88,15 +88,30 @@ The different files and directories have following meaning: ...@@ -88,15 +88,30 @@ The different files and directories have following meaning:
``` ```
. .
├── aicrowd.json # Submission meta information - like your username ├── aicrowd.json # Submission meta information - add tags for tracks here
├── apt.txt # Packages to be installed inside docker image ├── apt.txt # Packages to be installed inside submission environment
├── requirements.txt # Python packages to be installed ├── requirements.txt # Python packages to be installed with pip
├── rollout.py # Your rollout code - can use a batched agent ├── rollout.py # This will run rollouts on a batched agent
├── run.sh # Submission entrypoint ├── test_submission.py # Run this on your machine to get an estimated score
└── utility # The utility scripts to provide smoother experience to you. ├── run.sh # Submission entrypoint
├── docker_build.sh ├── utilities # Helper scripts for setting up and submission
├── docker_run.sh ├── submit.sh # script for easy submission of your code
├── environ.sh ├── envs # Operations on the env like batching and wrappers
├── batched_env.py # Batching for multiple envs
├── wrappers.py # Add wrappers to your env here
├── agents # Baseline agents for submission
├── batched_agent.py # Abstraction reference batched agents
├── random_batched_agent.py # Batched agent that returns random actions
├── rllib_batched_agent.py # Batched agent that runs with the rllib baseline
├── torchbeast_agent.py # Batched agent that runs with the torchbeast baseline
├── nethack_baselines # Baseline agents for submission
├── other_examples
├── random_rollouts.py # Barebones random agent with no batching
├── rllib # Baseline agent trained with rllib
├── torchbeast # Baseline agent trained with IMPALA on Pytorch
├── notebooks
├── NetHackTutorial.ipynb # Tutorial on the Nethack Learning Environment
``` ```
Finally, **you must specify an AIcrowd submission JSON in `aicrowd.json` to be scored!** See "How do I actually make a submission" below for more details. Finally, **you must specify an AIcrowd submission JSON in `aicrowd.json` to be scored!** See "How do I actually make a submission" below for more details.
...@@ -129,10 +144,10 @@ The `aicrowd.json` of each submission should contain the following content: ...@@ -129,10 +144,10 @@ The `aicrowd.json` of each submission should contain the following content:
```json ```json
{ {
"challenge_id": "evaluations-api-neurips-nethack", "challenge_id": "neurips-2021-the-nethack-challenge",
"authors": ["your-aicrowd-username"], "authors": ["your-aicrowd-username"],
"description": "(optional) description about your awesome agent", "description": "(optional) description about your awesome agent",
"external_dataset_used": false "gpu": true
} }
``` ```
...@@ -234,220 +249,12 @@ More information on submissions can be found at our [SUBMISSION.md](/docs/SUBMIS ...@@ -234,220 +249,12 @@ More information on submissions can be found at our [SUBMISSION.md](/docs/SUBMIS
- [Danielle Rothermel](https://www.aicrowd.com/participants/danielle_rothermel) - [Danielle Rothermel](https://www.aicrowd.com/participants/danielle_rothermel)
<!-- # 📎 Important links
=================
# Table of Contents
1. [Competition Procedure](#competition-procedure)
# Competition Procedure
The NetHack Learning Environment (NLE) is a Reinforcement Learning environment presented at NeurIPS 2020. NLE is based on NetHack 3.6.6 and designed to provide a standard RL interface to the game, and comes with tasks that function as a first step to evaluate agents on this new environment. You can read more about NLE in the NeurIPS 2020 paper.
We are excited that this competition offers machine learning students, researchers and NetHack-bot builders the opportunity to participate in a grand challenge in AI without prohibitive computational costs—and we are eagerly looking forward to the wide variety of submissions.
**The following is a high level description of how this process works**
1. **Sign up** to join the competition [on the AIcrowd website](https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge).
2. **Clone** this repo and start developing your solution.
3. **Train** your models on NLE and write rollout code in `rollout.py`.
4. [**Submit**](#how-to-submit-a-model) your trained models to [AIcrowd Gitlab](https://gitlab.aicrowd.com) for evaluation [(full instructions below)](#how-to-submit-a-model). The automated evaluation setup will evaluate the submissions against the NLE environment for a fixed number of rollouts to compute and report the metrics on the leaderboard of the competition.
![](https://i.imgur.com/xzQkwKV.jpg)
# Installation - Nethack Learning Environment
NLE requires `python>=3.5`, `cmake>=3.14` to be installed and available both when building the
package, and at runtime.
On **MacOS**, one can use `Homebrew` as follows:
``` bash
$ brew install cmake
```
On a plain **Ubuntu 18.04** distribution, `cmake` and other dependencies
can be installed by doing:
```bash
# Python and most build deps
$ sudo apt-get install -y build-essential autoconf libtool pkg-config \
python3-dev python3-pip python3-numpy git flex bison libbz2-dev
# recent cmake version
$ wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -
$ sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
$ sudo apt-get update && apt-get --allow-unauthenticated install -y \
cmake \
kitware-archive-keyring
```
Afterwards it's a matter of setting up your environment. We advise using a conda
environment for this:
```bash
$ conda create -n nle python=3.8
$ conda activate nle
$ pip install git+https://github.com/facebookresearch/nle.git@eric/competition --no-binary:nle
```
Find more details on the [original nethack repository](https://github.com/facebookresearch/nle)
# How to start participating
## Setup
1. **Add your SSH key** to AIcrowd GitLab
You can add your SSH Keys to your GitLab account by going to your profile settings [here](https://gitlab.aicrowd.com/profile/keys). If you do not have SSH Keys, you will first need to [generate one](https://docs.gitlab.com/ee/ssh/README.html#generating-a-new-ssh-key-pair).
2. **Clone the repository** - TODO
```
git clone git@github.com:AIcrowd/neurips-2021-nethack-starter-kit.git
```
3. **Install** competition specific dependencies!
```
pip install aicrowd-api
pip install aicrowd-gym
## Install NLE according to the instructions above
```
4. Try out random rollout script in `rollout.py`.
## How do I specify my software runtime / dependencies ? - TODO
We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.
The configuration files typically include `requirements.txt` (pypi packages), `apt.txt` (apt packages) or even your own `Dockerfile`.
You can check detailed information about the same in the [RUNTIME.md](/docs/RUNTIME.md) file. 💪 Challenge Page: https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge
## What should my code structure be like ?
Please follow the example structure as it is in the starter kit for the code structure.
The different files and directories have following meaning:
```
.
├── aicrowd.json # Submission meta information - like your username
├── apt.txt # Packages to be installed inside docker image
├── requirements.txt # Python packages to be installed
├── rollout.py # Your rollout code - can use a batched agent
├── run.sh # Submission entrypoint
└── utility # The utility scripts to provide smoother experience to you.
├── docker_build.sh
├── docker_run.sh
├── environ.sh
```
Finally, **you must specify an AIcrowd submission JSON in `aicrowd.json` to be scored!**
The `aicrowd.json` of each submission should contain the following content:
```json
{
"challenge_id": "evaluations-api-neurips-nethack",
"authors": ["your-aicrowd-username"],
"description": "(optional) description about your awesome agent",
"external_dataset_used": false
}
```
This JSON is used to map your submission to the challenge - so please remember to use the correct `challenge_id` as specified above.
## Can I use some other language instead of python?
The submission entrypoint is a bash script `run.sh`, you can call any arbitrary code you like from here. However, the environment has to be setup using python as in `rollout.py`. Any other code will have to communicte with the envrironment created in python.
**Note**: You need to install your dependencies for running your code by following the `How do I specify my software runtime/dependencies` section above.
🗣️ Discussion Forum: https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge/discussion
## How to make submission 🏆 Leaderboard: https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge/leaderboards
👉 [SUBMISSION.md](/docs/SUBMISSION.md)
# Other Information
## Hardware and Time constraints
To be added.
## Local Run
To be added.
## Contributing? - TODO
To be added
## Contributors - TODO
- [Jyotish Poonganam](https://www.aicrowd.com/participants/jyotish)
- [Dipam chakraborty](https://www.aicrowd.com/participants/dipam)
- [Shivam Khandelwal](https://www.aicrowd.com/participants/shivam)
# 📎 Important links - TODO
💪 &nbsp;Challenge Page: https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge
🗣️ &nbsp;Discussion Forum: https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge/discussion
🏆 &nbsp;Leaderboard: https://www.aicrowd.com/challenges/neurips-2021-nethack-challenge/leaderboards
**Best of Luck** 🎉 🎉 **Best of Luck** 🎉 🎉
├── Dockerfile
├── LICENSE
├── README.md
├── __pycache__
│ ├── rollout.cpython-37.pyc
│ └── submission_config.cpython-37.pyc
├── agents
│ ├── __pycache__
│ ├── batched_agent.py
│ ├── random_batched_agent.py
│ ├── rllib_batched_agent.py
│ └── torchbeast_agent.py
├── aicrowd.json
├── apt.txt
├── docs
│ ├── RUNTIME.md
│ └── SUBMISSION.md
├── envs
│ ├── __init__.py
│ ├── __pycache__
│ ├── batched_env.py
│ └── wrappers.py
├── evaluation_utils
│ └── custom_wrappers.py
├── nethack_baselines
│ ├── other_examples
│ ├── rllib
│ └── torchbeast
├── notebooks
├── requirements.txt
├── rollout.py
├── run.sh
├── submission_config.py
├── test_submission.py
└── utility
├── docker_build.sh
├── docker_run.sh
├── environ.sh
├── parser.py
└── verify_or_download_data.py
<p style="text-align:center"><img style="text-align:center" src="https://raw.githubusercontent.com/facebookresearch/nle/master/dat/nle/example_run.gif"></p> -->
utility/docker_build.sh deleted 100755 → 0
+ 0
− 24
View file @ 7ec8d594
#!/bin/bash
if [ -e environ_secret.sh ]
then
source utility/environ_secret.sh
else
source utility/environ.sh
fi
if ! [ -x "$(command -v aicrowd-repo2docker)" ]; then
echo 'Error: aicrowd-repo2docker is not installed.' >&2
echo 'Please install it using requirements.txt or pip install -U aicrowd-repo2docker' >&2
exit 1
fi
# Expected Env variables : in environ.sh
REPO2DOCKER="$(which aicrowd-repo2docker)"
sudo ${REPO2DOCKER} --no-run \
--user-id 1001 \
--user-name aicrowd \
--image-name ${IMAGE_NAME}:${IMAGE_TAG} \
--debug .
utility/docker_run.sh deleted 100755 → 0
+ 0
− 29
View file @ 7ec8d594
#!/bin/bash
# This script run your submission inside a docker image, this is identical in termrs of
# how your code will be executed on AIcrowd platform
if [ -e environ_secret.sh ]
then
echo "Note: Gathering environment variables from environ_secret.sh"
source utility/environ_secret.sh
else
echo "Note: Gathering environment variables from environ.sh"
source utility/environ.sh
fi
# Skip building docker image on run, by default each run means new docker image build
if [[ " $@ " =~ " --no-build " ]]; then
echo "Skipping docker image build"
else
echo "Building docker image, for skipping docker image build use \"--no-build\""
./utility/docker_build.sh
fi
# Expected Env variables : in environ.sh
sudo docker run \
--net=host \
--user 0 \
-e AICROWD_IS_GRADING=True \
-e AICROWD_DEBUG_MODE=True \
-it ${IMAGE_NAME}:${IMAGE_TAG} \
/bin/bash
\ No newline at end of file
utility/environ.sh deleted 100644 → 0
+ 0
− 4
View file @ 7ec8d594
#!/bin/bash
export IMAGE_NAME="aicrowd/music-demixing-challenge"
export IMAGE_TAG="local"
utility/parser.py deleted 100644 → 0
+ 0
− 2
View file @ 7ec8d594
#!/usr/bin/env python3
print("Evaluator script to test predictions locally to be added here.")
utility/submit.sh 0 → 100644
+ 76
− 0
View file @ 07c2a067
#!/bin/bash
set -e
REPO_ROOT_DIR=$(git rev-parse --show-toplevel)
SCRIPTS_DIR="${REPO_ROOT_DIR}/utility"
source "${SCRIPTS_DIR}/logging.sh"
print_usage() {
cat << USAGE
Usage: ./utility/submit.sh "impala-ppo-v0.1"
USAGE
}
bad_remote_message() {
log_normal "AIcrowd remote not found"
log_error "Please run \`git remote add aicrowd git@gitlab.aicrowd.com:<username>/<repo>.git\` and rerun this command"
exit 1
}
check_remote() {
log_info Checking git remote settings...
bad_remotes=(
git@gitlab.aicrowd.com:nethack/neurips-2021-the-nethack-challenge.git
http://gitlab.aicrowd.com/nethack/neurips-2021-the-nethack-challenge.git
)
for bad_remote in $bad_remotes; do
if git remote -v | grep "$bad_remote" > /dev/null; then
bad_remote_message
fi
done
if ! git remote -v | grep "gitlab.aicrowd.com"; then
bad_remote_message
fi
}
setup_lfs() {
git lfs install
HTTPS_REMOTE=$(git remote -v | grep gitlab.aicrowd.com | head -1 | awk '{print $2}' | sed 's|git@gitlab.aicrowd.com:|https://gitlab.aicrowd.com|g')
git config lfs.$HTTPS_REMOTE/info/lfs.locksverify false
find . -type f -size +5M -exec git lfs track {} &> /dev/null \;
git add .gitattributes
}
setup_commits() {
REMOTE=$(git remote -v | grep gitlab.aicrowd.com | head -1 | awk '{print $1}')
TAG=$(echo "$@" | sed 's/ /-/g')
git add --all
git commit -m "Changes for submission-$TAG"
git tag -am "submission-$TAG" "submission-$TAG" || (log_error "There is another submission with the same description. Please give a different description." && exit 1)
git push -f $REMOTE master
git push -f $REMOTE "submission-$TAG"
}
submit() {
check_remote
setup_lfs
setup_commits
}
if [[ $# -lt 1 ]]; then
print_usage
exit 1
fi
submit "$@"
utility/verify_or_download_data.py deleted 100644 → 0
+ 0
− 57
View file @ 7ec8d594
import os, sys
import requests
import zipfile
sys.path.append(os.path.dirname(os.path.realpath(os.getcwd())))
sys.path.append(os.path.realpath(os.getcwd()))
DATASET_FILE_NAME = 'download.zip'
DATASET_DOWNLOAD_URL = 'https://zenodo.org/record/3270814/files/MUSDB18-7-WAV.zip?download=1'
DATASET_FULL_DOWNLOAD_URL = 'https://zenodo.org/record/3338373/files/musdb18hq.zip?download=1'
def download_dataset(full=False):
dn_url = DATASET_DOWNLOAD_URL
if full:
dn_url = DATASET_FULL_DOWNLOAD_URL
r = requests.get(dn_url, stream=True)
with open(DATASET_FILE_NAME, 'wb') as fd:
for chunk in r.iter_content(chunk_size=256):
fd.write(chunk)
def unzip_dataset():
with zipfile.ZipFile(DATASET_FILE_NAME, 'r') as zip_ref:
zip_ref.extractall('data/')
def cleanup():
os.remove(DATASET_FILE_NAME)
def verify_dataset():
assert os.path.isdir("data/train") and os.path.isdir("data/test"), "Dataset folder not found"
assert os.path.isdir("data/train/Hollow Ground - Left Blind"), "Random song check in training folder failed"
assert os.path.isdir("data/test/Louis Cressy Band - Good Time"), "Random song check in testing folder failed"
def move_to_git_root():
if not os.path.exists(os.path.join(os.getcwd(), ".git")):
os.chdir("..")
assert os.path.exists(os.path.join(os.getcwd(), ".git")), "Unable to reach to repository root"
if __name__ == "__main__":
move_to_git_root()
try:
verify_dataset()
except AssertionError:
print("Dataset not found...")
option = input("Download full dataset (y/Y) or 7s dataset (n/N)? ")
if option.lower() == 'y':
print("Downloading full dataset...")
download_dataset(full=True)
else:
print("Downloading 7s dataset...")
download_dataset()
print("Unzipping the dataset...")
unzip_dataset()
print("Cleaning up...")
#cleanup()
print("Verifying the dataset...")
verify_dataset()
print("Done.")
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment