Skip to content

Commit 5a9cfa9

Browse files
Masaaki-75Masaaki-75
authored
[Project] Medical dataset: Kvasir seg aliyun (open-mmlab#2678)
1 parent 2ea4784 commit 5a9cfa9

8 files changed

+372
-0
lines changed

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/README.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Kvasir-SEG Segmented Polyp Dataset from Aliyun (Kvasir SEG Aliyun)
2+
3+
## Description
4+
5+
This project supports **`Kvasir-SEG Segmented Polyp Dataset from Aliyun (Kvasir SEG Aliyun) `**, which can be downloaded from [here](https://tianchi.aliyun.com/dataset/84385).
6+
7+
### Dataset Overview
8+
9+
Colorectal cancer is the second most common cancer type among women and third most common among men. Polyps are precursors to colorectal cancer and therefore important to detect and remove at an early stage. Polyps are found in nearly half of the individuals at age 50 that undergo a colonoscopy screening, and their frequency increase with age.Polyps are abnormal tissue growth from the mucous membrane, which is lining the inside of the GI tract, and can sometimes be cancerous. Colonoscopy is the gold standard for detection and assessment of these polyps with subsequent biopsy and removal of the polyps. Early disease detection has a huge impact on survival from colorectal cancer. Increasing the detection of polyps has been shown to decrease risk of colorectal cancer. Thus, automatic detection of more polyps at an early stage can play a crucial role in prevention and survival from colorectal cancer.
10+
11+
The Kvasir-SEG dataset is based on the previous Kvasir dataset, which is the first multi-class dataset for gastrointestinal (GI) tract disease detection and classification. It contains annotated polyp images and their corresponding masks. The pixels depicting polyp tissue, the ROI, are represented by the foreground (white mask), while the background (in black) does not contain positive pixels. These images were collected and verified by experienced gastroenterologists from Vestre Viken Health Trust in Norway. The classes include anatomical landmarks, pathological findings and endoscopic procedures.
12+
13+
### Information Statistics
14+
15+
| Dataset Name | Anatomical Region | Task Type | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License |
16+
| ------------------------------------------------------ | ----------------- | ------------ | --------- | ------------ | --------------------- | ---------------------- | ------------ | --------------------------------------------------------- |
17+
| [kvasir-seg](https://tianchi.aliyun.com/dataset/84385) | abdomen | segmentation | endoscopy | 2 | 1000/-/- | yes/-/- | 2020 | [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
18+
19+
| Class Name | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
20+
| :--------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
21+
| background | 1000 | 84.72 | - | - | - | - |
22+
| polyp | 1000 | 15.28 | - | - | - | - |
23+
24+
Note:
25+
26+
- `Pct` means percentage of pixels in this category in all pixels.
27+
28+
### Visualization
29+
30+
![kvasir_seg_aliyun](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/endoscopy_images/kvasir_seg_aliyun/kvasir_seg_aliyun_dataset.png?raw=true)
31+
32+
### Dataset Citation
33+
34+
```
35+
@inproceedings{jha2020kvasir,
36+
title={Kvasir-seg: A segmented polyp dataset},
37+
author={Jha, Debesh and Smedsrud, Pia H and Riegler, Michael A and Halvorsen, P{\aa}l and Lange, Thomas de and Johansen, Dag and Johansen, H{\aa}vard D},
38+
booktitle={International Conference on Multimedia Modeling},
39+
pages={451--462},
40+
year={2020},
41+
organization={Springer}
42+
}
43+
```
44+
45+
### Prerequisites
46+
47+
- Python v3.8
48+
- PyTorch v1.10.0
49+
- pillow(PIL) v9.3.0
50+
- scikit-learn(sklearn) v1.2.0
51+
- [MIM](https://github.com/open-mmlab/mim) v0.3.4
52+
- [MMCV](https://github.com/open-mmlab/mmcv) v2.0.0rc4
53+
- [MMEngine](https://github.com/open-mmlab/mmengine) v0.2.0 or higher
54+
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) v1.0.0rc5
55+
56+
All the commands below rely on the correct configuration of `PYTHONPATH`, which should point to the project's directory so that Python can locate the module files. In `kvasir_seg_aliyun/` root directory, run the following line to add the current directory to `PYTHONPATH`:
57+
58+
```shell
59+
export PYTHONPATH=`pwd`:$PYTHONPATH
60+
```
61+
62+
### Dataset Preparing
63+
64+
- download dataset from [here](https://tianchi.aliyun.com/dataset/84385) and decompression data to path 'data/.'.
65+
- run script `"python tools/prepare_dataset.py"` to format data and change folder structure as below.
66+
- run script `"python ../../tools/split_seg_dataset.py"` to split dataset and generate `train.txt`, `val.txt` and `test.txt`. If the label of official validation set and test set cannot be obtained, we generate `train.txt` and `val.txt` from the training set randomly.
67+
68+
```none
69+
mmsegmentation
70+
├── mmseg
71+
├── projects
72+
│ ├── medical
73+
│ │ ├── 2d_image
74+
│ │ │ ├── endoscopy
75+
│ │ │ │ ├── kvasir_seg_aliyun
76+
│ │ │ │ │ ├── configs
77+
│ │ │ │ │ ├── datasets
78+
│ │ │ │ │ ├── tools
79+
│ │ │ │ │ ├── data
80+
│ │ │ │ │ │ ├── train.txt
81+
│ │ │ │ │ │ ├── val.txt
82+
│ │ │ │ │ │ ├── images
83+
│ │ │ │ │ │ │ ├── train
84+
│ │ │ │ | │ │ │ ├── xxx.png
85+
│ │ │ │ | │ │ │ ├── ...
86+
│ │ │ │ | │ │ │ └── xxx.png
87+
│ │ │ │ │ │ ├── masks
88+
│ │ │ │ │ │ │ ├── train
89+
│ │ │ │ | │ │ │ ├── xxx.png
90+
│ │ │ │ | │ │ │ ├── ...
91+
│ │ │ │ | │ │ │ └── xxx.png
92+
```
93+
94+
### Divided Dataset Information
95+
96+
***Note: The table information below is divided by ourselves.***
97+
98+
| Class Name | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
99+
| :--------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
100+
| background | 800 | 84.66 | 200 | 84.94 | - | - |
101+
| polyp | 800 | 15.34 | 200 | 15.06 | - | - |
102+
103+
### Training commands
104+
105+
To train models on a single server with one GPU. (default)
106+
107+
```shell
108+
mim train mmseg ./configs/${CONFIG_FILE}
109+
```
110+
111+
### Testing commands
112+
113+
To test models on a single server with one GPU. (default)
114+
115+
```shell
116+
mim test mmseg ./configs/${CONFIG_FILE} --checkpoint ${CHECKPOINT_PATH}
117+
```
118+
119+
<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/configs/fcn#results-and-models)
120+
121+
You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project. -->
122+
123+
## Checklist
124+
125+
- [x] Milestone 1: PR-ready, and acceptable to be one of the `projects/`.
126+
127+
- [x] Finish the code
128+
- [x] Basic docstrings & proper citation
129+
- [ ] Test-time correctness
130+
- [x] A full README
131+
132+
- [ ] Milestone 2: Indicates a successful model implementation.
133+
134+
- [ ] Training-time correctness
135+
136+
- [ ] Milestone 3: Good to be a part of our core package!
137+
138+
- [ ] Type hints and docstrings
139+
- [ ] Unit tests
140+
- [ ] Code polishing
141+
- [ ] Metafile.yml
142+
143+
- [ ] Move your modules into the core package following the codebase's file hierarchy structure.
144+
145+
- [ ] Refactor your modules into the core package following the codebase's file hierarchy structure.

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_kvasir-seg-aliyun-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py',
3+
'./kvasir-seg-aliyun_512x512.py', 'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.kvasir-seg-aliyun_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.0001)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_kvasir-seg-aliyun-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py',
3+
'./kvasir-seg-aliyun_512x512.py', 'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.kvasir-seg-aliyun_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.001)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_kvasir-seg-aliyun-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py',
3+
'./kvasir-seg-aliyun_512x512.py', 'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.kvasir-seg-aliyun_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.01)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/configs/fcn-unet-s5-d16_unet_1xb16-0.01lr-sigmoid-20k_kvasir-seg-aliyun-512x512.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py',
3+
'./kvasir-seg-aliyun_512x512.py', 'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.kvasir-seg-aliyun_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.01)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(
14+
num_classes=2, loss_decode=dict(use_sigmoid=True), out_channels=1),
15+
auxiliary_head=None,
16+
test_cfg=dict(mode='whole', _delete_=True))
17+
vis_backends = None
18+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/configs/kvasir-seg-aliyun_512x512.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
dataset_type = 'KvasirSEGAliyunDataset'
2+
data_root = 'data/'
3+
img_scale = (512, 512)
4+
train_pipeline = [
5+
dict(type='LoadImageFromFile'),
6+
dict(type='LoadAnnotations'),
7+
dict(type='Resize', scale=img_scale, keep_ratio=False),
8+
dict(type='RandomFlip', prob=0.5),
9+
dict(type='PhotoMetricDistortion'),
10+
dict(type='PackSegInputs')
11+
]
12+
test_pipeline = [
13+
dict(type='LoadImageFromFile'),
14+
dict(type='Resize', scale=img_scale, keep_ratio=False),
15+
dict(type='LoadAnnotations'),
16+
dict(type='PackSegInputs')
17+
]
18+
train_dataloader = dict(
19+
batch_size=16,
20+
num_workers=4,
21+
persistent_workers=True,
22+
sampler=dict(type='InfiniteSampler', shuffle=True),
23+
dataset=dict(
24+
type=dataset_type,
25+
data_root=data_root,
26+
ann_file='train.txt',
27+
data_prefix=dict(img_path='images/', seg_map_path='masks/'),
28+
pipeline=train_pipeline))
29+
val_dataloader = dict(
30+
batch_size=1,
31+
num_workers=4,
32+
persistent_workers=True,
33+
sampler=dict(type='DefaultSampler', shuffle=False),
34+
dataset=dict(
35+
type=dataset_type,
36+
data_root=data_root,
37+
ann_file='val.txt',
38+
data_prefix=dict(img_path='images/', seg_map_path='masks/'),
39+
pipeline=test_pipeline))
40+
test_dataloader = val_dataloader
41+
val_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU', 'mDice'])
42+
test_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU', 'mDice'])

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/datasets/kvasir-seg-aliyun_dataset.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
from mmseg.datasets import BaseSegDataset
2+
from mmseg.registry import DATASETS
3+
4+
5+
@DATASETS.register_module()
6+
class KvasirSEGAliyunDataset(BaseSegDataset):
7+
"""KvasirSEGAliyunDataset dataset.
8+
9+
In segmentation map annotation for KvasirSEGAliyunDataset,
10+
0 stands for background,which is included in 2 categories.
11+
``reduce_zero_label`` is fixed to False. The ``img_suffix``
12+
is fixed to '.png' and ``seg_map_suffix`` is fixed to '.png'.
13+
Args:
14+
img_suffix (str): Suffix of images. Default: '.png'
15+
seg_map_suffix (str): Suffix of segmentation maps. Default: '.png'
16+
reduce_zero_label (bool): Whether to mark label zero as ignored.
17+
Default to False..
18+
"""
19+
METAINFO = dict(classes=('background', 'polyp'))
20+
21+
def __init__(self,
22+
img_suffix='.png',
23+
seg_map_suffix='.png',
24+
reduce_zero_label=False,
25+
**kwargs) -> None:
26+
super().__init__(
27+
img_suffix=img_suffix,
28+
seg_map_suffix=seg_map_suffix,
29+
reduce_zero_label=reduce_zero_label,
30+
**kwargs)

‎projects/medical/2d_image/endoscopy/kvasir_seg_aliyun/tools/prepare_dataset.py‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
import glob
2+
import os
3+
4+
import numpy as np
5+
from PIL import Image
6+
7+
root_path = 'data/'
8+
img_suffix = '.jpg'
9+
seg_map_suffix = '.jpg'
10+
save_img_suffix = '.png'
11+
save_seg_map_suffix = '.png'
12+
tgt_img_dir = os.path.join(root_path, 'images/train/')
13+
tgt_mask_dir = os.path.join(root_path, 'masks/train/')
14+
os.system('mkdir -p ' + tgt_img_dir)
15+
os.system('mkdir -p ' + tgt_mask_dir)
16+
17+
18+
def filter_suffix_recursive(src_dir, suffix):
19+
# filter out file names and paths in source directory
20+
suffix = '.' + suffix if '.' not in suffix else suffix
21+
file_paths = glob.glob(
22+
os.path.join(src_dir, '**', '*' + suffix), recursive=True)
23+
file_names = [_.split('/')[-1] for _ in file_paths]
24+
return sorted(file_paths), sorted(file_names)
25+
26+
27+
def convert_label(img, convert_dict):
28+
arr = np.zeros_like(img, dtype=np.uint8)
29+
for c, i in convert_dict.items():
30+
arr[img == c] = i
31+
return arr
32+
33+
34+
def convert_pics_into_pngs(src_dir, tgt_dir, suffix, convert='RGB'):
35+
if not os.path.exists(tgt_dir):
36+
os.makedirs(tgt_dir)
37+
38+
src_paths, src_names = filter_suffix_recursive(src_dir, suffix=suffix)
39+
for i, (src_name, src_path) in enumerate(zip(src_names, src_paths)):
40+
tgt_name = src_name.replace(suffix, save_img_suffix)
41+
tgt_path = os.path.join(tgt_dir, tgt_name)
42+
num = len(src_paths)
43+
img = np.array(Image.open(src_path))
44+
if len(img.shape) == 2:
45+
pil = Image.fromarray(img).convert(convert)
46+
elif len(img.shape) == 3:
47+
pil = Image.fromarray(img)
48+
else:
49+
raise ValueError('Input image not 2D/3D: ', img.shape)
50+
51+
pil.save(tgt_path)
52+
print(f'processed {i+1}/{num}.')
53+
54+
55+
def convert_label_pics_into_pngs(src_dir,
56+
tgt_dir,
57+
suffix,
58+
convert_dict={
59+
0: 0,
60+
255: 1
61+
}):
62+
if not os.path.exists(tgt_dir):
63+
os.makedirs(tgt_dir)
64+
65+
src_paths, src_names = filter_suffix_recursive(src_dir, suffix=suffix)
66+
num = len(src_paths)
67+
for i, (src_name, src_path) in enumerate(zip(src_names, src_paths)):
68+
tgt_name = src_name.replace(suffix, save_seg_map_suffix)
69+
tgt_path = os.path.join(tgt_dir, tgt_name)
70+
71+
img = np.array(Image.open(src_path).convert('L'))
72+
img = convert_label(img, convert_dict)
73+
Image.fromarray(img).save(tgt_path)
74+
print(f'processed {i+1}/{num}.')
75+
76+
77+
if __name__ == '__main__':
78+
convert_pics_into_pngs(
79+
os.path.join(root_path, 'Kvasir-SEG/images'),
80+
tgt_img_dir,
81+
suffix=img_suffix)
82+
83+
convert_label_pics_into_pngs(
84+
os.path.join(root_path, 'Kvasir-SEG/masks'),
85+
tgt_mask_dir,
86+
suffix=seg_map_suffix)

0 commit comments

Comments
 (0)