Table of contents

• Competition procedure
• Quick start
  • Install
  • Make your first submission
• Submit your own agents
  • Repository structure
  • Custom runtime configuration
  • Implement submission.py
  • Submit
• Baseline
• Local Evaluation
• FAQ
• Important links
• Contributors

Competition procedure

The Neural MMO Challenge is an opportunity for researchers and machine learning enthusiasts to test their skills by designing and building agents that can survive and thrive together in a massively multi-agent environment full of potential adversaries.

In this challenge, you will train your models locally and then upload them to AIcrowd (via git) to be evaluated.

The following is a high level description of how this process works.

1. Sign up to join the competition on the AIcrowd website and click Participate on the homepage.
2. Clone this repo and start developing your solution.
3. Design and build your agents that can compete in Neural MMO environment and create your submission as described in Make your own submission section.
4. Submit your agents to AIcrowd Gitlab for evaluation.

Quick Start

Install

Clone this starter kit repository.

git clone http://gitlab.aicrowd.com/neural-mmo/neurips2022-nmmo-starter-kit.git
conda create -n neurips2022-nmmo python==3.9
conda activate neurips2022-nmmo
cd ./neurips2022-nmmo-starter-kit

Install necessary dependencies, git-lfs is for submitting large files by git (use brew for mac users) and neurips2022nmmo is a Neural MMO environment wrapper prepared for NeurIPS 2022 Competition.

apt install git-lfs
pip install git+http://gitlab.aicrowd.com/neural-mmo/neurips2022-nmmo.git
pip install -r requirements_tool.txt

This competition is based on NeuralMMO v1.6 version.

For more information about NeuralMMO v1.6, please refer to https://github.com/NeuralMMO/environment/tree/v1.6.

Make your first submission

python tool.py submit "my-first-submission"

See your submission on Submissions and check your rank on Leaderboard in a few minutes.

Submit your own agents

1. Repository structure

Must put all the files and models under the my-submission/ directory. Otherwise the evaluation will fail.

- my-submission/            # Directory containing your submission.
    | - other_files         # All others files needed by your submission.
    | - submission.py       # Entrypoint of your submission.
- submission-runtime        # Directory containing default Dockerfile and requirements.txt. 
- Dockerfile                # Dockerfile for your submission. 
- requirements.txt          # Python requirements for your submission.
- .submit.sh                # Helper script for submit.
- tool.py                   # Helper script for validate locally and submit.
- aicrowd.json              # Json for submit.

2. Custom runtime configuration

The default runtime is provided in submission-runtime/. We also accept submissions with custom runtime. The configuration files include requirements.txt and Dockerfile.

3. Implement submission.py

Here is an example of submission:

from nmmo.io import action
from neurips2022nmmo import Team

class YourTeam(Team):
    def __init__(self, team_id: str, config=None, **kwargs):
        super().__init__(team_id, config)

    def act(self, observations):
        actions = {}
        if "stat" in observations:
            observations.pop("stat")
        for player_idx, obs in observations.items():
            actions[player_idx] = {}
        return actions

class Submission:
    team_klass = YourTeam
    init_params = {}

Note that YourTeam class which must inherit Team class and act method should be implemented and we will call the function act for evaluation. An example with random agents is provided in my-submission/submission.py for your reference.

4. Validate locally and Submit to AICrowd

We provide tool.py to validate and submit your submission.

1. To locally validate your repo before submission.

python tool.py test

2. To submit your submission to AIcrowd gitlab.

(Caution) Add offline: true in aicrowd.json if you use Offline Learning, e.g.

{
    "challenge_id" : "neurips-2022-the-neural-mmo-challenge",
    "authors" : [],
    "offline": true,
    "description" : "Brief description for your submission"
}

python tool.py submit <unique-submission-name>

If you can see the following output, congratulations! Now you can check your submission.

          #///(            )///#
         ////      ///      ////
        /////   //////////   ////
        /////////////////////////
     /// /////////////////////// ///
   ///////////////////////////////////
  /////////////////////////////////////
    )////////////////////////////////(
     /////                      /////
   (///////   ///       ///    //////)
  ///////////    ///////     //////////
(///////////////////////////////////////)
          /////           /////
            /////////////////
               ///////////

However even after seeing the above output, one can still get "failed" status on online submission webpage. It is most likely that your local running environment is different from that of our competition server. To test your repo in the exact server environment, add --startby=docker option when testing.

python tool.py test --startby=docker
python tool.py submit <unique-submission-name> --startby=docker

Baselines

We provide a variety of baseline agents, please refer to neurips2022-nmmo-baselines repository.

Local evaluation

To do local evaluation, we provide Rollout for easier debug, here is an example. The environment is same with the online evaluation environment in PvE Stage 1.

from neurips2022nmmo import CompetitionConfig, scripted, submission, RollOut

config = CompetitionConfig()

my_team = submission.get_team_from_submission(
    submission_path="my-submission/",
    team_id="MyTeam",
    env_config=config,
)
# Or initialize the team directly
# my_team = MyTeam("Myteam", config, ...)

teams = [scripted.CombatTeam(f"Combat-{i}", config) for i in range(5)]
teams.extend([scripted.MixtureTeam(f"Mixture-{i}", config) for i in range(10)])
teams.append(my_team)

ro = RollOut(config, teams, parallel=True, show_progress=True)
ro.run(n_episode=1)

FAQ

1. Are there any hardware or time constraints?

During evaluation, your submission will be allocated with 1 CPU core and 1G memory. Each step, your Agent Team must give decision within 600ms.

2. Are there any file size limit for submission?

Your submission should not be larger than 500MB.

3. How can I speed up docker image pull?

For participants in China, you can pull image from tencentcloud

python tool.py submit <unique-submission-name> --startby=docker --registry=tencentcloud

4. How do windows users participate the competition?

For participants using windows, we strongly recommend you to install wsl.

5. How can I render/save replay locally?

Please refer to Unity viewer tutorial for an instruction about how to render locally.

6. Error: Pack Exceeds Maximum Allowed Size?

Try dividing your whole commit into multiple smaller commits.

Important links

• Challenge information
  • Challenge page
  • Leaderboard
• Community
  • Neural MMO discord server
  • Challenge discussion forum
  • 微信群: chaocanshustuff
  • 
• Neural MMO resources
  • Neural MMO documentation
  • Neural MMO GitHub repository
  • Neural MMO Environment Tutorial

Contributors

• Parametrix.ai
• Joseph Suarez
• AIcrowd