Meta KDD Cup '24 CRAG: Comprehensive RAG Benchmark Starter Kit

This repository is the CRAG: Comphrensive RAG Benchmark Submission template and Starter kit! Clone the repository to compete now!

This repository contains:

• Documentation on how to submit your models to the leaderboard
• The procedure for best practices and information on how we evaluate your model, etc.
• Starter code for you to get started!

Table of Contents

1. Competition Overview
2. Dataset
3. Tasks
4. Evaluation Metrics
5. Getting Started
   • How to write your own model?
   • How to start participating?
     • Setup
     • How to make a submission?
     • What hardware does my code run on?
     • How are my model responses parsed by the evaluators?
6. Frequently Asked Questions
7. Important Links

📖 Competition Overview

📊 Dataset

👨‍💻👩‍💻 Tasks

📏 Evaluation Metrics

Please refer to local_evaluation.py for more details on how we will evaluate your submissions.

🏁 Getting Started

1. Sign up to join the competition on the AIcrowd website.
2. Fork this starter kit repository. You can use this link to create a fork.
3. Clone your forked repo and start developing your model.
4. Develop your model(s) following the template in how to write your own model section.
5. Submit your trained models to AIcrowd Gitlab for evaluation (full instructions below). The automated evaluation will evaluate the submissions on the public test set and report the metrics on the leaderboard of the competition.

✍️ How to write your own model?

Please follow the instructions in models/README.md for instructions and examples on how to write your own models for this competition.

🚴 How to start participating?

Setup

1. Add your SSH key to AIcrowd GitLab

You can add your SSH Keys to your GitLab account by going to your profile settings here. If you do not have SSH Keys, you will first need to generate one.

2. Fork the repository. You can use this link to create a fork.

3. Clone the repository

   git clone git@gitlab.aicrowd.com:<YOUR-AICROWD-USERNAME>/meta-comphrehensive-rag-benchmark-starter-kit.git
   cd meta-comphrehensive-rag-benchmark-starter-kit

4. Install competition specific dependencies!

   cd meta-comphrehensive-rag-benchmark-starter-kit
   pip install -r requirements.txt

5. Write your own model as described in How to write your own model section.

6. Test your model locally using python local_evaluation.py.

7. Accept the Challenge Rules on the main challenge page by clicking on the Participate button. Also accept the Challenge Rules on the Task specific page (link on the challenge page) that you want to submit to.

8. Make a submission as described in How to make a submission section.

✍️ How to write your own model?

Please follow the instructions in models/README.md for instructions and examples on how to write your own models for this competition.

📮 How to make a submission?

Please follow the instructions in docs/submission.md to make your first submission. This also includes instructions on specifying your software runtime, code structure, submitting to different tracks.

Note: Remember to accept the Challenge Rules on the challenge page, and the task page before making your first submission.

💻 What hardware does my code run on ?

You can find more details about the hardware and system configuration in docs/hardware-and-system-config.md. In summary, we provide you 4 x [NVIDIA T4 GPUs].

❓ Frequently Asked Questions

Which track is this starter kit for ?

This starter kit can be used to submit to any of the tracks. You can find more information in docs/submission.md#submitting-to-different-tracks.

Where can I know more about the dataset schema ?

The dataset schema is described in docs/dataset.md.

Best of Luck 🎉 🎉

📎 Important links

• 💪 Challenge Page: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024
• 🗣️ Discussion Forum: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024/discussion
• 🏆 Leaderboard: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024/leaderboards