Skip to content
Snippets Groups Projects
Select Git revision
  • master default
  • starterkit_changes
2 results
Compare
Name Last commit Last update
core
data
docs
evaluator
siam-mot
train
utility
.gitattributes
.gitignore
LICENSE
README.md
__init__.py
aicrowd.json
requirements.txt
run.sh
siam_mot_test.py
test.py
README.md

Airborne Banner

Airborne Object Tracking Challenge Starter Kit

👉 Challenge page

Discord

This repository is the main Airborne Object Tracking challenge Submission template and Starter kit!

Clone the repository to compete now!

This repository contains:

  • Documentation on how to submit your agent to the leaderboard
  • The procedure for best practices and information on how we evaluate your agent, etc.
  • Starter code for you to get started!
  • SiamMOT: Siamese Multi-Object Tracking baseline

Table of Contents

  1. Competition Procedure
  2. How to access and use dataset
  3. How to start participating
  4. How do I specify my software runtime / dependencies?
  5. What should my code structure be like ?
  6. How to make submission
  7. ⭐ SiamMOT baseline
  8. Other concepts and FAQs
  9. Important links

Competition Procedure

The main task of the competition is to detect a collision threat reliably. In this challenge, you will train your agents locally and then upload them to AIcrowd (via git) to be evaluated.

The following is a high level description of how this round works

  1. Sign up to join the competition [on the AIcrowd website].(https://www.aicrowd.com/challenges/airborne-object-tracking-challenge)
  2. Clone this repo and start developing your solution.
  3. Train your models to detect objects and write inference code in test.py.
  4. Submit your trained models to AIcrowd Gitlab for evaluation (full instructions below). The automated evaluation setup will evaluate the submissions against the test dataset to compute and report the metrics on the leaderboard of the competition.

How to access and use dataset

The starter kit contains dataset exploration notebooks and helper functions to access the dataset. You can check the instructions for the same here: 👉 DATASET.md.

How to start participating

Setup

  1. Add your SSH key to AIcrowd GitLab

You can add your SSH Keys to your GitLab account by going to your profile settings here. If you do not have SSH Keys, you will first need to generate one.

  1. Clone the repository

    git clone git@gitlab.aicrowd.com:amazon-prime-air/airborne-detection-starter-kit.git

  2. Install competition specific dependencies!

    cd airborne-detection-starter-kit
pip3 install -r requirements.txt

  3. Run local exploration notebook present in data/dataset-playground.ipynb using jupyter notebook command locally.

  4. Try out random prediction codebase present in test.py.

How do I specify my software runtime / dependencies ?

We accept submissions with custom runtime, so you don't need to worry about which libraries or framework to pick from.

The configuration files typically include requirements.txt (pypi packages), environment.yml (conda environment), apt.txt (apt packages) or even your own Dockerfile.

You can check detailed information about the same in the 👉 RUNTIME.md file.

What should my code structure be like ?

Please follow the example structure as it is in the starter kit for the code structure. The different files and directories have following meaning:

.
├── aicrowd.json           # Submission meta information - like your username
├── apt.txt                # Packages to be installed inside docker image
├── data                   # Your local dataset copy - you don't need to upload it (read DATASET.md)
├── requirements.txt       # Python packages to be installed
├── test.py                # IMPORTANT: Your testing/inference phase code, must be derived from AirbornePredictor (example in test.py)
└── utility                # The utility scripts to provide smoother experience to you.
    ├── docker_build.sh
    ├── docker_run.sh
    ├── environ.sh
    └── verify_or_download_data.sh

Finally, you must specify an AIcrowd submission JSON in aicrowd.json to be scored!

The aicrowd.json of each submission should contain the following content:

{
  "challenge_id": "evaluations-api-airborne",
  "grader_id": "evaluations-api-airborne",
  "authors": ["aicrowd-bot"],
  "tags": "change-me",
  "description": "Random prediction model for Airborne challenge",
  "gpu": false
}

This JSON is used to map your submission to the challenge - so please remember to use the correct challenge_id as specified above.

Please specify if your code will use a GPU or not for the evaluation of your model. If you specify true for the GPU, GPU will be provided and used for the evaluation.

How to make submission

👉 SUBMISSION.md

Best of Luck 🎉 🎉

SiamMOT baseline

SiamMOT is a region-based Siamese Multi-Object Tracking network that detects and associates object instances simultaneously.

This repository contains SiamMOT baseline interface which you can submit and improve upon.

⚠️ Submissions that make use of the the provided SIMA-MOT baseline will be considered for ranking only if use a different model (different weights) which improves EDR by at least 1.5% (that is EDR >= 0.685, AFDR >= 0.6415) and HFAR < 0.5/ FPPI< 0.0005 — improvement of 1.5% in EDR practically means 2 more encounters detected (out of 102) OR Keeps the same EDR = 0.6699 / AFDR = 0.6265 and reduces HFAR/ FPPI by at least 50% (e.g. HFAR <= 0.23, FPPI <= 0.0002)

Additional Steps

  1. Change your entrypoint i.e. run.sh from python test.py to python siam_mot_test.py.
  2. Copy the Dockerfile present in siam-mot/Dockerfile to repository root.
  3. Set gpu: true in your aicrowd.yaml.
  4. Follow common steps shared in SUBMISSION.md 
#> cp siam-mot/Dockerfile Dockerfile

Other Concepts

Time constraints

You need to make sure that your model can predict airborne objects for each flight within 800 seconds, otherwise the submission will be marked as failed.

Local evaluation

You can also test end to end evaluation on your own systems. The scripts are available in core/metrics folder. A working example is also available as Colab Notebook here.

Hardware used for evaluations

We use p3.2xlarge to run your evaluations i.e. 8 vCPU, 61 GB RAM, V100 GPU.

(please enable GPU by putting "gpu": true in your aicrowd.json file)

Frequently Asked Questions

We have curated frequently asked questions and common mistakes on Discourse, you can read them here: FAQ and Common mistakes

📎 Important links

💪  Challenge Page: https://www.aicrowd.com/challenges/airborne-object-tracking-challenge

🗣️  Discussion Forum: https://www.aicrowd.com/challenges/airborne-object-tracking-challenge/discussion

🏆  Leaderboard: https://www.aicrowd.com/challenges/airborne-object-tracking-challenge/leaderboards