Commit 2027fb1e authored by spmohanty's avatar spmohanty
Browse files

Rename README file

parent 29cdd89c
# MyFoodRepo Experiments
Repository with the code for all experiments included in the MyFoodRepo Benchmark paper.
# Table of contents
- [Getting Started](#getting-started)
- [Running The Demo Project](#running-the-demo-project)
- [Template Details](#template-details)
- [Folder Structure](#folder-structure)
- [Main Components](#main-components)
# Getting Started
This template allows you to simply build and train deep learning models with checkpoints and tensorboard visualization.
## Downloading MyFoodRepo data
In order to use the template you have to:
Before getting started with the experiments the first thing is to download the Data. To make things simpler we are downloading data using AICrowd CLI.
1. Define a data loader class.
2. Define a model class that inherits from BaseModel.
3. Define a trainer class that inherits.
4. Define a configuration file with the parameters needed in an experiment.
5. Run the model using:
1. To use the CLI, first you have to make an account on AICrowd[http://aicrowd.com/] and get the API Key from profile.
2. Open the terminal and run
```shell
python main.py -c [path to configuration file]
bash download_data.sh <YOUR_API_KEY>
```
# Running The Demo Project
After executing the above command, you'll see the data folder in your current directory. It contains training and validation images and respective annotations.json.
A simple model for the mnist dataset is available to test the template.
To run the demo project:
## Preprocessing data
1. Start the training using:
Dataset in real-world settings are not very clean. We have performed some rudimentary pre-processing for you, such as
```shell
python main.py -c configs/simple_mnist_config.json
```
1. Checking and correcting if the image dimensions in the annotation file matches with the actual image size in training and validation set.
2. Annotations might contain some pictures which are rotated w.r.t the segmentations. We currently remove these pictures, although we encourage you to correct them if possible.
3. A polygon must have atleast 3 coordinates. In this pre-processing we are removing polygons which does not satisfy this criteria.
4. Bounding boxes are created manually, therefore to increase the data quality we redraw them using the polygon segments
2. Start Tensorboard visualization using:
To perform the pre-processing you must have the data placed in data folder. To run the pre-processing script, type `python pre-processing.py`. It will create `annotations_new.json` in training and validation data folders.
```shell
tensorboard --logdir=experiments/simple_mnist/logs
```
## Downloading COCO-Pre-trained models
# Template Details
Transfer Learning helps downstreaming the deep learning task and coverge better with less training data. We use pre-trained models as a starting point for training on MyFoodRepo dataset. We have made it easier for you to download these models. You just have to specify which model you need. Below are the models on which we performed our experiments -
## Folder Structure
- mask-r50 = Download Mask R-CNN ResNet50 model
- mask-r101 = Download Mask R-CNN ResNet50 model
- mask-rx101 = Download Mask R-CNN ResNet50 model
- htc-r50 = Download Mask R-CNN ResNet50 model
- htc-r101 = Download Mask R-CNN ResNet50 model
- htc-rx101 = Download Mask R-CNN ResNet50 model
- detectors-r50 = Download Mask R-CNN ResNet50 model
- all = To download all the models
```
├── main.py - here's an example of main that is responsible for the whole pipeline.
├── base - this folder contains the abstract classes of the project components
│ ├── base_data_loader.py - this file contains the abstract class of the data loader.
│ ├── base_model.py - this file contains the abstract class of the model.
│ └── base_train.py - this file contains the abstract class of the trainer.
├── model - this folder contains the models of your project.
│ └── simple_mnist_model.py
├── trainer - this folder contains the trainers of your project.
│ └── simple_mnist_trainer.py
|
├── data_loader - this folder contains the data loaders of your project.
│ └── simple_mnist_data_loader.py
├── configs - this folder contains the experiment and model configs of your project.
│ └── simple_mnist_config.json
├── datasets - this folder might contain the datasets of your project.
└── utils - this folder contains any utils you need.
├── config.py - util functions for parsing the config files.
├── dirs.py - util functions for creating directories.
└── utils.py - util functions for parsing arguments.
In terminal,
```shell
bash download_base_models.sh mask-r50
```
## Main Components
It will download and put the model in `./base_models/mask_rcnn` directory.
### Models
## Building a Model
You need to:
This part of README covers the available MMDetection training scripts that can be used to train a model.
1. Create a model class that inherits from **BaseModel**.
2. Override the **_build_model_** function which defines your model.
3. Call **_build_model_** function from the constructor.
### Experimental Scripts
### Trainers
The experimental scripts are setup for you to quickly get started with model training. We have created models with three different instance segmentation algorithms (Mask R-CNN, HTC, DetectoRS). The motivation for using these three is that, Mask R-CNN is among the first few methods proposed to solve instance segmentation. The anatomical architecture of instance segmentation is a 2-stage pipeline, where first stage does the feature extraction, and second does the region proposals. The table below breifly explains the difference and changes in these algorithms.
You need to:
| Algorithm | Stage 1 | Stage 2 |
|------------|:-------:|--------:|
| Mask R-CNN | FPN | 1 RPN |
| HTC | FPN | 3 RPNs |
| DetectoRS | RFP | 3 RPNs |
1. Create a trainer class that inherits from **BaseTrainer**.
2. Override the **_train_** function which defines the training logic.
- FPN - Feature Pyramid Network
- RFP - Recursive Feature Pyramid
- RPN - Region Proposal Network
**Note:** To add functionalities after each training epoch such as saving checkpoints or logs for tensorboard using Keras callbacks:
**Scripts**- Uses above described methods
1. Declare a callbacks array in your constructor.
2. Define an **_init_callbacks_** function to populate your callbacks array and call it in your constructor.
3. Pass the callbacks array to the **_fit_** function on the model object.
| Method | Backbone | Version | Description |
|------------|--------------|---------------|----------------------------------------------------------------|
| Mask R-CNN | ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Weighted Loss | Baseline + change in loss function of Mask R-CNN |
| | | Augmentation | Baseline + Albumentations augmentations |
| | | Multi-Scale | Baseline + Multi-scale training on 3 image resolutions |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNet101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNeXt101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| HTC | ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNet101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNeXt101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| DetectoRS | HTC+ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
**Note:** You can use **_fit_generator_** instead of **_fit_** to support generating new batches of data instead of loading the whole dataset at one time.
The training scripts are located under `experimental-scripts/` directory.
Feel free to change the scripts as per your requirements.
### Data Loaders
### Training Models
You need to:
The training scripts contains code with Weights and Biases (WandB) support. Good thing about WandB is that, you only have to configure once with the API key. Our scripts will help organize all the experiments you will perform for your research or product development.
To start with the model training, we assume that you have completed all the above steps. We assume that you have right CUDA version for your GPU. If so, run the following command in terminal:
1. Create a data loader class that inherits from **BaseDataLoader**.
2. Override the **_get_train_data()_** and the **_get_test_data()_** functions to return your train and test dataset splits.
```shell
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python mmdetection/tools/train.py ./experimental-scripts/mask_rcnn/baseline.py
```
**Note:** You can also define a different logic where the data loader class has a function **_get_next_batch_** if you want the data reader to read batches from your dataset each time.
This will save the model under `./work-dir/mask_rcnn/resnet50/baseline`.
### Configs
Easy huh!
You need to define a .json file that contains your experiment and model configurations such as the experiment name, the batch size, and the number of epochs.
## Generating Predictions (Inference)
### Main
Now that you have trained models. It's time to get the inferences from the learned model. We have prepared a inference script for you that will save all the predictions in the `json` file format.
Responsible for building the pipeline.
1. Getting inference for 1 model
1. Parse the config file
2. Create an instance of your data loader class.
3. Create an instance of your model class.
4. Create an instance of your trainer class.
5. Train your model using ".Train()" function on the trainer object.
The below script will help you to generate the predictions for a model and configuration. The variables set below are according to the current setting for this repository, ofcourse you can and should change it for other experiments.
### From Config
```shell
config=./experimental-scripts/mask_rcnn/baseline.py
checkpoint=./work-dir/mask_rcnn/resnet50/baseline/epoch_1.pth
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_1.json
data=./data/val/images
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python inference/predict_model.py \
--config $config \
--checkpoint $checkpoint \
--data $data
--format-only --eval-options jsonfile_prefix=predictions \
--out_file $out_file
```
Warning: You might want to use absolute path instead of relative path, if it doesn't work.
We can now load models without having to explicitly create an instance of each class. Look at:
2. Ensembling different models
1. from_config.py: this can load any config file set up to point to the right modules/classes to import
2. Look at configs/simple_mnist_from_config.json to get an idea of how this works from the config. Run it with:
At this point we assume that you have atleast two trained models with you. If yes, you can improve the score even further by using our ensembling approach.
Update the `./inference/models.py` as per your requirements. You need specify the configuration file path, checkpoint path, and output file path respectively. We have updated it to the basic need i.e. (ensembling of Mask R-CNN baseline model epoch 1 and epoch 2)
```shell
python from_config.py -c configs/simple_mnist_from_config.json
data=./data/val/images
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_ensemble.json
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python inference/infer.py \
--out_file $out_file \
--data $data
```
3. See conv_mnist_from_config.json (and the additional data_loader/model) to see how easy it is to run a different experiment with just a different config file:
## Evaluation
We use MS COCO defined evaluation metrics i.e. mAP (Mean Average Precision), mAR (Mean Average Recall). The scripts for evaluation can be found under `./eval/`.
To evaluate the performance of a model, one needs the ground truth annotations and predictions. We are using the directory structure of this repository, although you can change the paths as per your need.
```shell
python from_config.py -c configs/conv_mnist_from_config.json
annotations=./data/val/annotations_new.json
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_1.json
CUDA_VISIBLE_DEVICES=0 python eval/eval.py $out_file --ann $annotations
```
# Authors(s)
- Sharada Mohanty <mohanty@aicrowd.com>
- Gaurav Singhal <singhal.gaurav93@gmail.com>
Finish. We hope you had a nice time working with our repository.
\ No newline at end of file
# Getting Started
## Downloading MyFoodRepo data
Before getting started with the experiments the first thing is to download the Data. To make things simpler we are downloading data using AICrowd CLI.
1. To use the CLI, first you have to make an account on AICrowd[http://aicrowd.com/] and get the API Key from profile.
2. Open the terminal and run
```shell
bash download_data.sh <YOUR_API_KEY>
```
After executing the above command, you'll see the data folder in your current directory. It contains training and validation images and respective annotations.json.
## Preprocessing data
Dataset in real-world settings are not very clean. We have performed some rudimentary pre-processing for you, such as
1. Checking and correcting if the image dimensions in the annotation file matches with the actual image size in training and validation set.
2. Annotations might contain some pictures which are rotated w.r.t the segmentations. We currently remove these pictures, although we encourage you to correct them if possible.
3. A polygon must have atleast 3 coordinates. In this pre-processing we are removing polygons which does not satisfy this criteria.
4. Bounding boxes are created manually, therefore to increase the data quality we redraw them using the polygon segments
To perform the pre-processing you must have the data placed in data folder. To run the pre-processing script, type `python pre-processing.py`. It will create `annotations_new.json` in training and validation data folders.
## Downloading COCO-Pre-trained models
Transfer Learning helps downstreaming the deep learning task and coverge better with less training data. We use pre-trained models as a starting point for training on MyFoodRepo dataset. We have made it easier for you to download these models. You just have to specify which model you need. Below are the models on which we performed our experiments -
- mask-r50 = Download Mask R-CNN ResNet50 model
- mask-r101 = Download Mask R-CNN ResNet50 model
- mask-rx101 = Download Mask R-CNN ResNet50 model
- htc-r50 = Download Mask R-CNN ResNet50 model
- htc-r101 = Download Mask R-CNN ResNet50 model
- htc-rx101 = Download Mask R-CNN ResNet50 model
- detectors-r50 = Download Mask R-CNN ResNet50 model
- all = To download all the models
In terminal,
```shell
bash download_base_models.sh mask-r50
```
It will download and put the model in `./base_models/mask_rcnn` directory.
## Building a Model
This part of README covers the available MMDetection training scripts that can be used to train a model.
### Experimental Scripts
The experimental scripts are setup for you to quickly get started with model training. We have created models with three different instance segmentation algorithms (Mask R-CNN, HTC, DetectoRS). The motivation for using these three is that, Mask R-CNN is among the first few methods proposed to solve instance segmentation. The anatomical architecture of instance segmentation is a 2-stage pipeline, where first stage does the feature extraction, and second does the region proposals. The table below breifly explains the difference and changes in these algorithms.
| Algorithm | Stage 1 | Stage 2 |
|------------|:-------:|--------:|
| Mask R-CNN | FPN | 1 RPN |
| HTC | FPN | 3 RPNs |
| DetectoRS | RFP | 3 RPNs |
- FPN - Feature Pyramid Network
- RFP - Recursive Feature Pyramid
- RPN - Region Proposal Network
**Scripts**- Uses above described methods
| Method | Backbone | Version | Description |
|------------|--------------|---------------|----------------------------------------------------------------|
| Mask R-CNN | ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Weighted Loss | Baseline + change in loss function of Mask R-CNN |
| | | Augmentation | Baseline + Albumentations augmentations |
| | | Multi-Scale | Baseline + Multi-scale training on 3 image resolutions |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNet101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNeXt101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| HTC | ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNet101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| | ResNeXt101 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
| DetectoRS | HTC+ResNet50 | Baseline | Contains the baseline code provided by original authors |
| | | Combined | Baseline + Augmentation + Multi-Scale + Tuned Hyper-parameters |
The training scripts are located under `experimental-scripts/` directory.
Feel free to change the scripts as per your requirements.
### Training Models
The training scripts contains code with Weights and Biases (WandB) support. Good thing about WandB is that, you only have to configure once with the API key. Our scripts will help organize all the experiments you will perform for your research or product development.
To start with the model training, we assume that you have completed all the above steps. We assume that you have right CUDA version for your GPU. If so, run the following command in terminal:
```shell
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python mmdetection/tools/train.py ./experimental-scripts/mask_rcnn/baseline.py
```
This will save the model under `./work-dir/mask_rcnn/resnet50/baseline`.
Easy huh!
## Generating Predictions (Inference)
Now that you have trained models. It's time to get the inferences from the learned model. We have prepared a inference script for you that will save all the predictions in the `json` file format.
1. Getting inference for 1 model
The below script will help you to generate the predictions for a model and configuration. The variables set below are according to the current setting for this repository, ofcourse you can and should change it for other experiments.
```shell
config=./experimental-scripts/mask_rcnn/baseline.py
checkpoint=./work-dir/mask_rcnn/resnet50/baseline/epoch_1.pth
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_1.json
data=./data/val/images
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python inference/predict_model.py \
--config $config \
--checkpoint $checkpoint \
--data $data
--format-only --eval-options jsonfile_prefix=predictions \
--out_file $out_file
```
Warning: You might want to use absolute path instead of relative path, if it doesn't work.
2. Ensembling different models
At this point we assume that you have atleast two trained models with you. If yes, you can improve the score even further by using our ensembling approach.
Update the `./inference/models.py` as per your requirements. You need specify the configuration file path, checkpoint path, and output file path respectively. We have updated it to the basic need i.e. (ensembling of Mask R-CNN baseline model epoch 1 and epoch 2)
```shell
data=./data/val/images
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_ensemble.json
CUDA_VISIBLE_DEVICES=<DEVICE_ID> python inference/infer.py \
--out_file $out_file \
--data $data
```
## Evaluation
We use MS COCO defined evaluation metrics i.e. mAP (Mean Average Precision), mAR (Mean Average Recall). The scripts for evaluation can be found under `./eval/`.
To evaluate the performance of a model, one needs the ground truth annotations and predictions. We are using the directory structure of this repository, although you can change the paths as per your need.
```shell
annotations=./data/val/annotations_new.json
out_file=./work-dir/mask_rcnn/resnet50/baseline/predictions_1.json
CUDA_VISIBLE_DEVICES=0 python eval/eval.py $out_file --ann $annotations
```
Finish. We hope you had a nice time working with our repository.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment