Skip to content
Snippets Groups Projects

角頭-大橋頭線上看(2024)完整版HD.1080P.高清电影

Open 角頭-大橋頭線上看(2024)完整版HD.1080P.高清电影
dune_kudune/mnist-code-evaluator:dune_kudune-master-patch-42773 into master
Open Dataset Movie BLACKPINK WORLD TOUR BORN PINk in CINEMAS requested to merge dune_kudune/mnist-code-evaluator:dune_kudune-master-patch-42773 into master
Compare
and
1 file
+ 8
− 233
Compare changes
  • Side-by-side
  • Inline
readme.md
+ 8
− 233
# MNIST evaluator
This repository uses `predictions-evaluator` template to evaluate code submissions.
![ Gatao: Like Father Like Son 2024 Baner](https://image.tmdb.org/t/p/original/SdQT0VMHTNZ78piTQgrkwYOtGV.jpg)
For more information of how the evaluator works, please refer the [template's reference page](https://gitlab.aicrowd.com/aicrowd/evaluator-templates/tree/master/predictions-evaluator).
The starter kit for this template is available at [https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit).
# Table of contents
- [How does this work?](#how-does-this-work)
- [How to write evaluators?](#how-to-write-evaluators)
* [Submission starter kit](#submission-starter-kit)
* [Evaluation code](#evaluation-code)
+ [Evaluation script](#evaluation-script)
+ [Launcher scripts](#launcher-scripts)
+ [Evaluator configuration](#evaluator-configuration)
+ [Dataset configuration](#dataset-configuration)
- [File structure](#file-structure)
- [Specifying dataset configuration](#specifying-dataset-configuration)
<h1 style="text-align: center;">&nbsp;角頭-大橋頭線上看(2024)完整版HD.1080P.高清电影&nbsp;</h1><p style="text-align: center;"><br /></p><p style="text-align: center;">**LAST UPDATED : AGUSTUS 18, 2024**</p><p style="text-align: center;">────────────────── •✧✧• ──────────────────</p><p style="text-align: center;"><br /></p><h3 style="text-align: center;"><a href="https://taiwan-movies-hd-full-2024.blogspot.com/">📺 立即播放 ▶️▶ 角頭-大橋頭 2024 在線流媒體</a></h3><h3 style="text-align: center;"><br /><a href="https://taiwan-movies-hd-full-2024.blogspot.com/">✅➤➤Sub tw zh ➫ ➫ 角頭-大橋頭 2024 在線流媒體 2024</a></h3><h3 style="text-align: center;"><br />✅➤➤Sub English ➫ ➫&nbsp;<a href="https://lawe.sensacinema.site/en/">角頭-大橋頭 2024 在線流媒體2024 HD</a></h3><p style="text-align: center;"><br /></p><p style="text-align: center;">────────────────── •✧✧• ──────────────────</p><p style="text-align: center;">'11 secs ago - Still Now Here 選擇下載或觀看 角頭-大橋頭 2024 流式傳輸整部電影 免費在線. 你喜歡電影嗎？ 如果是這樣，那麼你會愛上新的 恐怖, 科幻 電影: 角頭-大橋頭 2024. 這部電影是 同類中最好的之一 恐怖, 科幻 . 角頭-大橋頭 很快就可以在 Netflix 上在線觀看！</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 是一部即將上映的電影，由 20th Century Studios</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 2024 影片时间线设定在1979年第一部《异形》与1986年的续集《异形2》之间，围绕一群年轻而勇敢的太空殖民者展开。讲述他们为逃离外星采矿殖民地的沉闷生活，在冒险探索一座废弃的太空站时，意外遭遇了宇宙中最可怕的生命体——异形。狭窄幽暗、危机四伏的空间站中，大逃杀的序幕已经被无情拉开，人类再次成为异形生物捕猎的目标。在无尽的黑暗与死亡威胁的笼罩之下，太空探险队员们能否在每一次心跳的瞬间，察觉到未知生物潜伏的丝丝寒意？在这场你死我活的追逐战中，他们将直面怎样的信任挑战和道德挣扎？当一个又一个同伴被残忍吞噬，求救无门的他们究竟能否在这场太空杀戮中寻得一线生机？</p><p style="text-align: center;"><br /></p><p style="text-align: center;">這部電影計劃在迪士尼+上映 2024-08-13, 2024.</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">2024 如何觀看 有幾種觀看方式 角頭-大橋頭 將可以在 Netflix 上非常觀看在線的 很快！因此，無論您想在筆記本電腦、手機還是平板電腦上觀看角頭-大橋頭，您都可以享受 電影幾乎在任何地方。而角頭-大橋頭 是一個如此令人期待的版本！是的，我們找到了 正宗的流媒體選項/服務。有關如何觀看 角頭-大橋頭 2024 全程免費 年份如下所述。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 可以流式傳輸嗎？正在 Crunchyroll、Disney Plus、HBO Max、Netflix 或 Amazon 上觀看 角頭-大橋頭主要的？是的，我們找到了一個真實的流媒體選項。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">Showcase Cinema Warwick 在美國有幾種觀看 角頭-大橋頭 在線的方式。您可以使用 流媒體服務，例如 Netflix、Hulu 或 Amazon Prime Video。您也可以租借或購買電影 iTunes 或谷歌播放。您還可以點播或在電視上可用的流媒體應用程序上觀看，或 流媒體設備，如果你有電纜。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在 HBO Max 上不可用。這是一部電視電影，是漫畫海賊王的一部分。工作室 在它背後，遺憾的是，角頭-大橋頭 目前無法在任何流媒體服務上觀看。不過，粉絲 不要害怕，因為我們的計劃是讓角頭-大橋頭跟隨其他索尼電影的腳步，登陸 Starz—— 您可以通過 Amazon Prime Video 訂閱的流媒體頻道。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">因此，無論您是想在筆記本電腦、手機還是平板電腦上觀看角頭-大橋頭，您都可以欣賞這部電影 幾乎在任何地方。而角頭-大橋頭 是一個如此令人期待的版本。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 於 8 月 6 日發布 2024 在影院上映，現在，它將在 OTT 上正式上映 幾個月後的平台。該電影可在線觀看和下載全高清（1080P）， 高清 (720P)、480P、360P 畫質。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">如何免費觀看角頭-大橋頭</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 推遲一點的一線希望，讓新電影觀眾有更多機會體驗 原創 角頭-大橋頭 給自己——或者讓影迷看八百遍，不 判斷。</p><p style="text-align: center;"><br /></p><p style="text-align: center;">&nbsp;</p><p style="text-align: center;"><br /></p><p style="text-align: center;">目前，角頭-大橋頭 可通過 Netflix 訂閱進行流式傳輸。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在 Netflix 上嗎？</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">這家流媒體巨頭擁有海量的電視節目和電影目錄，但不包括 “角頭-大橋頭。”我們建議讀者觀看其他黑暗奇幻電影，例如《巫師：夢魘》 狼。”</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在 Amazon Prime 上嗎？</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">亞馬遜 Prime 的當前目錄不包括“角頭-大橋頭”。但是，這部電影最終可能會在 該平台將在未來幾個月內作為視頻點播。因此，人們必須定期尋找 Amazon Prime 官方網站上的黑暗奇幻電影。正在尋找類似內容的觀眾 可以看原版節目《多羅羅》。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">在美國有幾種觀看 角頭-大橋頭 在線的方法。您可以使用流媒體服務，例如 Netflix， Hulu，或亞馬遜 Prime 視頻。您還可以在 iTunes 或 Google Play 上租借或購買電影。你也可以 點播觀看，或在電視或流媒體設備上可用的流媒體應用（如果您有有線電視）上觀看。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">在哪裡觀看 角頭-大橋頭 在線的？</p><p style="text-align: center;"><br /></p><p style="text-align: center;">目前沒有平台有權觀看角頭-大橋頭在線的。 MAPPA決定播出 這部電影只在影院上映，因為它取得了巨大的成功。另一方面，工作室沒有 希望轉移收入。流媒體電影只會削減利潤，不會增加利潤。</p><p style="text-align: center;"><br /></p><p style="text-align: center;">&nbsp;</p><p style="text-align: center;"><br /></p><p style="text-align: center;">因此，沒有任何流媒體服務被授權免費提供角頭-大橋頭。然而，這部電影將非常 肯定會被 Funimation、Netflix 和 Crunchyroll 等服務收購。作為最後的考慮， 這些媒體中的哪一個可能會在全球發行這部電影？</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 可以在 HBO Max 上播放嗎？</p><p style="text-align: center;"><br /></p><p style="text-align: center;">HBO Max 是一項相對較新的流媒體服務，提供角頭-大橋頭 供觀看。您可以在 HBO 上觀看角頭-大橋頭如果您已經是會員，則 Max。如果您還不是會員，您可以免費註冊一個月 如果您不想繼續訂閱，請在當月結束前取消試用。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在 Disney Plus 上可用嗎？</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 是一部可以在 Disney Plus 上播放的電影。如果您是，您可以在 Disney Plus 上觀看角頭-大橋頭已經是會員。如果您在試用服務一個月後不想訂閱，您可以 月底前取消。在其他流媒體服務上，角頭-大橋頭 可以租用或購買。</p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 是關於什麼的？</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Hocus Pocus (1993) 事件發生 29 年後，三名高中生必須工作 一起阻止桑德森姐妹回到今天的塞勒姆</p><p style="text-align: center;">關鍵字Google：</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 電影2024-fULL HD-1080</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在線觀看完全免费</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 完整的在线电影</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影完全免费</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影完整版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在线电影全免费中文</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 免费在线完整电影 2024</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 下载全高清-1080P</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影中文字幕</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 影评</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影简介</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影完整在线台湾，香港版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 澳門上映</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭2024上映</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 HD線上看</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 線上看小鴨</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 电影完整版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 線上看下載</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 2024 下載</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 線上看完整版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 線上看完整版小鴨</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 電影2024-fULL HD-1080</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在線觀看完全免费</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 完整的在线电影</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影完全免费</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影完整版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">角頭-大橋頭 在线电影全免费中文</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 免费在线完整电影 2024</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 下载全高清-1080P</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影中文字幕</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 影评</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影简介</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影完整在线台湾，香港版</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 澳門上映</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 20242024上映</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 HD線上看</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 線上看小鴨</p><p style="text-align: center;"><br /></p><p style="text-align: center;">Gatao: Like Father Like Son 2024 电影完整版</p><div style="text-align: center;"><br /></div>
# How does this work?
1. Participant forks the [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit).
2. Participant adds their code to the [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit), commits it and creates a git tag (manually or using the helper scripts).
3. AIcrowd triggers evaluation pipeline.
4. AIcrowd sets up the software runtime needed to run the participant's code. For more information on how participants can specify their software runtime please refer to [this post](https://discourse.aicrowd.com/t/how-to-specify-runtime-environment-for-your-submission/2274).
5. AIcrowd runs the participant's code on a validation dataset. The logs during this phase are exposed to the participants so that they can debug any code errors.
a. The information on how the participant's code should run comes from the evaluator code.
6. AIcrowd runs the participant's code on the test dataset. The logs during this phase are generally not exposed to the participants to prevent data leaks. However, the logs can be exposed by settings `logs: true` in `aicrowd.yaml` (file inside evaluator code repo).
7. AIcrowd runs evaluator code (written by the organizer) that will aggregate the results generated by the participant's code and return a score.
# How to write evaluators?
The evaluation pipeline has two code components
## We have curated frequently asked questions and common mistakes on Discourse, you can read them here: [FAQ and Common mistakes](https://watching.nwsautodaily.com/en/)
1. Submission [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit) (submitted by the participant).
2. Evaluator code (setup by organizers).
## Submission starter kit
The [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit) should contain an `aicrowd.json` file with `challenge_id` attribute pointing to the challenge. For example, if the challenge page is https://www.aicrowd.com/challenges/my-challenge, then the contents of `aicrowd.json` should look similar to this.
```json
{
"challenge_id": "my-challenge"
}
```
For the remainder of the [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit), we recommend that it be structured similar to the example [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit) containing
1. An [`evaluation_utils`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/tree/master/evaluation_utils) directory with scripts for local evaluation. The files in the [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit) are only for participants' reference. These files can be ignored/replaced during evaluation. Explained in [writing launcher scripts](#launcher-scripts).
2. An optional `models` directory where participants upload their model weights.
3. A [`local_evaluation.py`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/blob/master/local_evaluation.py) that participants can use to run their code locally.
4. A [`run.py`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/blob/master/run.py) where participants can add their code in a well-defined class interface.
a. It is recommended that organizers add class interface in this file with as much inline documentation as possible for the participants.
b. It is recommended to avoid adding logical code from organizer's side to this file. Any evaluation related code or utility functions can remain as a part of [`evaluation_utils`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/tree/master/evaluation_utils) package.
5. Runtime setup files to run the starter kit code. For a list of supported runtime configuration, please refer to [this post](https://discourse.aicrowd.com/t/how-to-specify-runtime-environment-for-your-submission/2274).
Please use the [example starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit) as a starting point when creating your own starter kit.
## 📎 Important links
## Evaluation code
The repo can be split into three components.
1. Evaluation script (`AIcrowdEvaluator` class in [`evaluator.py`](evaluator.py#L15)).
2. Launcher scripts (scripts that will start the participant code; placed in the [`data`](data) directory).
3. Evaluator configuration ([`aicrowd.yaml`](aicrowd.yaml)).
💪 🔴👉 [CLICK HERE TO WATCH FULL VIDEO!](https://watching.nwsautodaily.com/en/)
### Evaluation script
💪 🔴👉 https://gitlab.aicrowd.com/cocot_ngasu/gatao-like-father-like-son-2024-hd-tw
`AIcrowdEvaluator` can be a simple wrapper around your existing evaluation scripts. The class needs to implement an `evaluate` method.
```python
class AIcrowdEvaluator:
def __init__(self, **kwargs):
pass
def render_current_status_as_markdown(self) -> str:
return ""
def live_evaluate(self) -> str:
# Not compulsory but encouraged to do so.
# For long running evaluations, this will help the participants understand the performance
# of their models before the completion of the evaluation.
# Refer example evaluator.py at https://gitlab.aicrowd.com/aicrowd/mnist-code-evaluator/-/tree/master/evaluator.py
return {}
def evaluate(self):
scores = {
"score": 0,
"score_secondary": 0,
}
return scores
```
The scores returned by `AIcrowdEvaluator(...).evaluate()` are updated on the leaderboard.
**Note:** Please refer [this article](https://wiki.aicrowd.com/share/efa47829-ef4e-452e-b75e-c5ac3fb24290) for how to use `scores` dictionary. You can add multiple metrics, private scores, media using the `scores` dictionary.
You can find more information on how the evaluation scripts are invoked at [[Evaluation Flow]](https://gitlab.aicrowd.com/aicrowd/evaluator-templates/-/tree/master/predictions-evaluator#evaluation-flow).
The `AIcrowdEvaluator(...).render_current_status_as_markdown()` is invoked in a new process and will keep running as long as the evaluation is in progress. You can use this method to return some markdown content that we will use to display on the GitLab issue page for the participants. You can show the evaluation progress, live scores, and other interesting information that can improve the submission experience for the participant. You can also display images, videos and audios. You can upload the media files to s3 and insert the link in your markdown content. If you need help with uploading the files to s3 (or any file hosting provider), please reach out to us.
### Launcher scripts
These are the scripts that are used to start the evaluation using participants' code. Typically, these include
1. Entrypoint scripts (a python script that imports the participant's prediction class and starts the evaluation loop and a bash script to call the python script)
2. Evaluation utilities
For example, let us consider the [MNIST starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit). It has a [`local_evaluation.py`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/blob/master/local_evaluation.py) and [`evaluation_utils`](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/tree/master/evaluation_utils) directory. During the evaluation, we use a [`predict.py`](data/predict.py) that comes from the evaluator repo to start the evaluation instead of the `local_evaluation.py`. We also replace the files in the [`evaluation_utils` directory in the starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/tree/master/evaluation_utils) using the [files from the evaluator repo](data/evaluation_utils). This will drop any changes that participants might have made and also gives the flexibility to add some hidden functions as needed.
**Note:** Only the files placed in the [`data`](data) directory of the evaluator repo can be placed in the participant's code during evaluations. Even among these files, the files that need to be mounted should be explicitly defined in `aicrowd.yaml`. [[refer]](aicrowd.yaml#L46)
In this repository, the flow looks somewhat like this during an evaluation
- AIcrowd systems reads [`aicrowd.yaml`](aicrowd.yaml#L56) for mountable files (defined under `evaluation.global.files`).
- AIcrowd systems place the following files in the participant's code.
+ File structure
* [`predict.py`](data/predict.py) at `/home/aicrowd/predict.py`
* [`run.sh`](data/run.sh) at `/home/aicrowd/run.sh`
* [`evaluation_utils/base_predictor.py`](data/evaluation_utils/base_predictor.py) at `/home/aicrowd/evaluation_utils/base_predictor.py`
* [`evaluation_utils/mnist_evaluator.py`](data/evaluation_utils/mnist_evaluator.py) at `/home/aicrowd/evaluation_utils/mnist_evaluator.py`
+ Please note that these files should exist inside [`data`](data) directory of the evaluator repository.
+ **The participant code is available at `/home/aicrowd` (`/home/aicrowd` acts as the participant's repo root).**
- AIcrowd systems run `/home/aicrowd/run.sh`.
- Once the participant's code executes successfully, we invoke `AIcrowdEvaluator(...).evaluate()`.
**Note:** The `base_predictor.py` from the [starter kit](https://gitlab.aicrowd.com/aicrowd/mnist-starter-kit/-/tree/master/evaluation_utils/base_predictor.py) and the [evaluator repo](data/evaluation_utils/base_predictor.py) showcase a simple use case of needing only two methods to be filled by the participants -- a setup method and a prediction method. We encourage you to modify this class as per your needs.
### Evaluator configuration
The orchestration for the evaluation is handled using the values defined in your `aicrowd.yaml` file. The file has several inline comments to guide you through different options available. For more details on how the orchestration works, please refer [https://gitlab.aicrowd.com/aicrowd/evaluator-templates/-/tree/master/predictions-evaluator](https://gitlab.aicrowd.com/aicrowd/evaluator-templates/-/tree/master/predictions-evaluator).
### Dataset configuration
#### File structure
You can specify the dataset in `aicrowd.yaml` under the `dataset` section. We recommend that you upload your dataset as a zip file having the following file structure.
```
data.zip
├── debug_ground_truth_data
│ └── ...
├── debug_test_data
│ └── ...
├── ground_truth_data
│ └── ...
└── test_data
└── ...
```
Directory | Mount point (env) | Phase | Exposed to participant | Description
--- | --- | --- | --- | ---
`debug_test_data` | `AICRODW_DATASET_DIR` | `evaluation.debug_run` | **Yes** | Ideally some validation data to check if the participant's code is bug free. We generally expose the logs for the `debug_run` phase so that it is easier for participants to debug their code.
`ground_truth_data` | `AICROWD_GROUND_TRUTH_DIR` | `evaluation.scoring` | No | Ground truth data needed for the evaluator code to score the submissions.
`test_data` | `AICROWD_DATASET_DIR` | `evaluation.runs[]` | **Yes** | Test data consumed by the participant's code to generate predictions.
With this configuration, the evaluation looks similar to this.
![](https://i.imgur.com/llqs5LY.png)
In some cases, you might want to split the dataset into multiple sections and run multiple instances of the inference to speed up the predictions. For example, let say you have a dataset containing 100,000 images and the evaluation is expected to take 4 hours. We recommend that you split your dataset into subsets so that each set can be evaluated in ~1 hour. We can run multiple instances of the participant's code in parallel to speed up the evaluation time. In this case you can restructure your dataset as
**Note:** If your end to end evaluation takes over 8 hours of time, please reach out to us.
```
data.zip
├── debug_ground_truth_data
│ └── ...
├── debug_test_data (contains a few public images to validate participant's code)
│ └── ...
├── ground_truth_data (ground truth data needed for evaluator code)
│ └── ...
├── test_data_1 (contains 250,000 images)
│ └── ...
├── test_data_2 (contains 250,000 images)
│ └── ...
├── test_data_3 (contains 250,000 images)
│ └── ...
└── test_data_4 (contains 250,000 images)
└── ...
```
In your `aicrowd.yaml`, you should specify the following
```yaml
evaluation:
debug_run:
dataset_path: debug_test_data
runs:
- name: predictions-for-set-1
dataset_path: test_data_1
- name: predictions-for-set-2
dataset_path: test_data_2
- name: predictions-for-set-3
dataset_path: test_data_3
- name: predictions-for-set-4
dataset_path: test_data_4
```
This configuration mounts the respective directories inside `data.zip` at `AICROWD_DATASET_PATH` during the inference.
An example implementation is available in the [`dataset-split`](https://gitlab.aicrowd.com/aicrowd/mnist-code-evaluator/-/tree/dataset-split) branch.
With this setup, the evaluation will look similar to this.
![](https://i.imgur.com/mqLzcLv.png)
#### Specifying dataset configuration
Let's consider two cases,
1. You have the dataset at https://something.domain/data.zip
2. You have the dataset hosted on s3.
For case 1, the configuration in`aicrowd.yaml` should look like
```yaml
dataset:
url: https://something.domain/data.zip
# If your dataset is 500MB (uncompressed), capacity should be 1GB.
capacity: 1Gi
# Extract the dataset.
extract:
enabled: true
command: unzip data.zip
```
For case 2, the configuration in `aicrowd.yaml` should look like
```yaml
dataset:
url: s3://<bucket>/<path>
# If your dataset is 500MB (uncompressed), capacity should be 1GB.
capacity: 1Gi
# Extract the dataset.
extract:
enabled: true
command: unzip data.zip
# S3 bucket configuration
s3:
access_key: <key>
secret_key: <key>
region: <region>
endpoint: <s3 endpoint>
```
👉 [Challenge page](https://gitlab.aicrowd.com/cocot_ngasu/gatao-like-father-like-son-2024-hd-tw)
[![Discord](https://img.shields.io/discord/565639094860775436.svg)](https://discord.gg/hAuevqx9Tj)