Skip to content
Snippets Groups Projects
Commit 3a412c86 authored by ashivani's avatar ashivani
Browse files

added challenge

parents
No related branches found
No related tags found
No related merge requests found
Pipeline #4831 failed
data/*
stages:
- rails-update
- test-baseline
- submit-baseline
.aicrowd-setup: &aicrowd-setup
image: python:3.8
before_script:
- git clone https://github.com/AIcrowd/aicrowd-cli.git && cd aicrowd-cli
- pip install -r requirements.txt
- python setup.py develop
- cd ..
- aicrowd evaluations login -e "${SUPER_USER_EMAIL}" -p "${SUPER_USER_PASSWORD}"
rails-update:
stage: rails-update
image: python:3.8-slim
before_script:
- pip install pyyaml requests markdown
script:
- python utils.py
only:
changes:
- README.md
test-baseline:
stage: test-baseline
image: python:3.8
before_script:
- apt install curl
- curl -o /usr/local/bin/jq http://stedolan.github.io/jq/download/linux64/jq && chmod +x /usr/local/bin/jq
- pip install yq
- pip install jupyter
script:
- export BASELINE=`yq -r .official_baseline aicrowd.yaml | awk -F. '{print $1}'`
- jupyter nbconvert --to python baselines/${BASELINE}.ipynb
- ipython baselines/${BASELINE}.py
artifacts:
paths:
- submission.csv
only:
changes:
- baselines/*
submit-baseline:
stage: submit-baseline
<<: *aicrowd-setup
script:
- apt install curl
- curl -o /usr/local/bin/jq http://stedolan.github.io/jq/download/linux64/jq && chmod +x /usr/local/bin/jq
- pip install yq
- export CHALLENGE_NAME=`yq -r .challenge_name aicrowd.yaml`
- export EVALUATIONS_API_TOKEN=`yq -r .aicrowd_evaluations_token ~/.aicrowd/config.yaml`
- export GRADER_ID=`curl -k -X GET "https://evaluations-api.internal.k8s.aicrowd.com/v1/graders/?name=${CHALLENGE_NAME}"
-H "accept:application/json"
-H "AUTHORIZATION:${EVALUATIONS_API_TOKEN}" | jq '.[length-1].id'`
- aicrowd evaluations submission create --file ./submission.csv -g ${GRADER_ID} --wait
only:
changes:
- baselines/*
# To be filled in later with an consistent contribution guide
- Q : Who writes that ?
\ No newline at end of file
> **🛠 Contribute:** Found a typo? Or any other change in the description that you would like to see? Please consider sending us a pull request in the [public repo of the challenge here](https://gitlab.aicrowd.com/aicrowd/practice-challenges/aicrowd_SCRBL_challenge).
# 🕵️ Introduction
<img src="./images/header.jpg" widhth=500px height=300px>
Can AI detect whether a piece of text has been altered or not?
We've provided you a dataset of scrambled and unscrambled texts. You have to classify them!
</br>Understand with code! Here is [`getting started code`](https://discourse.aicrowd.com/t/baseline-mnist/2757) for you.`😄`
# 💾 Dataset
The dataset consists of around `1 million` pieces of texts which has been sourced from the largest encyclopedia in world - `Wikipedia!`
The dataset is split into train,val and test such that each of the sets has an equal split of the two classes.
# 📁 Files
Following files are available in the `resources` section:
- `train.csv` - (`600,002` samples) This csv file contains 600,002 texts and their corresponding labels.
- `val.zip` - (`100,000` samples) This csv file contains 100,000 texts and their corresponding labels.
- `test.zip` - (`300,000` samples) This csv file contains 300,000 texts.
# 🚀 Submission
- Prepare a CSV containing header as [text,label] and predicted value as `scrambled` or `unscrambled`.
**Make your first submission [here](https://www.aicrowd.com/challenges/scrbl/submissions/new) 🚀 !!**
# 🖊 Evaluation Criteria
During evaluation [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) where,
<img src="https://latex.codecogs.com/gif.latex?F1%20%3D%202%20*%20%5Cfrac%7Bprecision*recall%7D%7Bprecision&plus;recall%7D"/> </br>
# 🔗 Links
- 💪 Challenge Page: [https://www.aicrowd.com/challenges/scrbl](https://www.aicrowd.com/challenges/scrbl)
- 🗣️ Discussion Forum: [https://www.aicrowd.com/challenges/scrbl/discussion](https://www.aicrowd.com/challenges/scrbl/discussion)
- 🏆 Leaderboard: [https://www.aicrowd.com/challenges/scrbl/leaderboards](https://www.aicrowd.com/challenges/scrbl/leaderboards)
# 📱 Contact
- [Rohit Midha](mailto:rohit@aicrowd.com)
- [Shraddhaa Mohan](mailto:shraddhaa@aicrowd.com)
- [Sharada Mohanty](mailto:mohanty@aicrowd.com)
---
challenge_name: scrbl
evaluation_repo: git@gitlab.aicrowd.com:aicrowd/practice-challenges/aicrowd_SCRBL_challenge_evaluator.git
data_url: https://s3.wasabisys.com/aicrowd-practice-challenges/public/scrbl/v0.1/
official_baseline: SCRBL_baseline.ipynb
authors:
- name: Sharada Mohanty
email: mohanty@aicrowd.com
version: '0.1'
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
images/header.jpg

13.4 KiB

utils.py 0 → 100644
import requests
import html
import markdown
import yaml
import os
class Utils():
def __init__(self):
self.base_url = "https://www.aicrowd.com/api/v1/challenges"
self.api_key = os.getenv("AICROWD_API_KEY")
self.headers = {'Authorization': 'Token token="' + self.api_key +'"', 'Content-Type': 'application/json'}
with open("aicrowd.yaml", "r") as f:
self.challenge = yaml.safe_load(f)['challenge_name']
with open("README.md", "r") as f:
self.content = f.read()
self.convert()
def convert(self):
self.content = markdown.markdown(self.content)
self.content = self.content.replace("\"","'")
self.content = self.content.replace("h1","h2")
html.unescape(self.content)
def create_challenge(self):
payload = {
"challenge": self.challenge.upper(),
"status_cd": "draft",
"tagline": "Classify Scrambled Text",
"perpetual_challenge": False,
"answer_file_s3_key": None,
"slug": "scrbl-classify-scrambled-text",
"submission_license": "",
"api_required": False,
"media_on_leaderboard": False,
"online_grading": True,
"description": self.content,
"evaluation": None,
"rules": None,
"prizes": None,
"resources": None,
"submission_instructions": None,
"subissions_page": True,
"license": "",
"dataset_description": None,
"image_file": {"url": "aicrowd_SCRBL_challenge/images/SCRBL.png"},
"show_leaderboard": True,
"grader_identifier": "AIcrowd_GRADER_POOL",
"online_submissions": True,
"grader_logs": False,
"require_registration": False,
"grading_history": False,
"post_challenge_submissions": True,
"submissions_downloadable": False,
"dataset_note": None,
"discussions_visible": True,
"latest_submission": False,
"other_scores_fieldnames": "",
"teams_allowed": True,
"max_team_participants": 5,
"team_freeze_seconds_before_end": 604800,
"hidden_challenge": False,
"team_freeze_time": None,
"score_title": "",
"score_secondary_title": "",
"primary_sort_order_cd": "",
"secondary_sort_order_cd": "",
"challenges_organizers_attributes": [{"organizer_id": 9, "challenge_id": 1}],
"category_challenges_attributes": [{"category_id": "8"}],
"dataset_files_attributes": [
{
"seq": 1,
"title": "Training Data",
"description": "File containing the attributes along with the corresponding labels related to it.",
"evaluation": False,
"visible": True,
"hosting_location": "External",
"external_url": f"https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/scrbl/v0.1/train.csv",
"external_file_size": ""
},
{
"seq": 2,
"title": "Testing Data",
"description": "File that will be used for actual evaluation for the leaderboard score but does not have the actual label corresponding to the attributes.",
"evaluation": False,
"visible": True,
"hosting_location": "External",
"external_url": f"https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/scrbl/v0.1/test.csv",
"external_file_size": ""
},
{
"seq": 3,
"title": "Sample Submission",
"description": "Sample Submission Format",
"evaluation": False,
"visible": True,
"hosting_location": "External",
"external_url": f"https://s3.eu-central-1.wasabisys.com/aicrowd-practice-challenges/public/scrbl/v0.1/sample_submission.csv",
"external_file_size": ""
},
]
}
return requests.post(self.base_url, json=payload, headers=self.headers)
def update_challenge(self):
payload = {"description": self.content}
request_url = f"{self.base_url}/{self.challenge}"
response = requests.patch(request_url, json=payload, headers=self.headers)
return response
def update_or_create(self):
response = self.update_challenge()
if response.status_code == 404:
response = self.create_challenge()
return response
if __name__ == "__main__":
utils = Utils()
print(utils.update_or_create().text)
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment