Skip to content

Commit b24f422

Browse files
suyanzhou626suyanzhou626
authored
[Project] Medical semantic seg dataset: ISIC-2016 Task1 (open-mmlab#2708)
1 parent 78e036c commit b24f422

File tree

7 files changed

+392
-0
lines changed

7 files changed

+392
-0
lines changed

‎projects/medical/2d_image/dermoscopy/isic2016_task1/README.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# ISIC-2016 Task1
2+
3+
## Description
4+
5+
This project support **`ISIC-2016 Task1 `**, and the dataset used in this project can be downloaded from [here](https://challenge.isic-archive.com/data/#2016).
6+
7+
### Dataset Overview
8+
9+
The overarching goal of the challenge is to develop image analysis tools to enable the automated diagnosis of melanoma from dermoscopic images.
10+
11+
This challenge provides training data (~900 images) for participants to engage in all 3 components of lesion image analysis. A separate test dataset (~350 images) will be provided for participants to generate and submit automated results.
12+
13+
### Original Statistic Information
14+
15+
| Dataset name | Anatomical region | Task type | Modality | Num. Classes | Train/Val/Test Images | Train/Val/Test Labeled | Release Date | License |
16+
| ---------------------------------------------------------------- | ----------------- | ------------ | ---------- | ------------ | --------------------- | ---------------------- | ------------ | ---------------------------------------------------------------------- |
17+
| [ISIC-2016 Task1](https://challenge.isic-archive.com/data/#2016) | full body | segmentation | dermoscopy | 2 | 900/-/379- | yes/-/yes | 2016 | [CC-0](https://creativecommons.org/share-your-work/public-domain/cc0/) |
18+
19+
| Class Name | Num. Train | Pct. Train | Num. Val | Pct. Val | Num. Test | Pct. Test |
20+
| :---------: | :--------: | :--------: | :------: | :------: | :-------: | :-------: |
21+
| background | 900 | 82.08 | - | - | 379 | 81.98 |
22+
| skin lesion | 900 | 17.92 | - | - | 379 | 18.02 |
23+
24+
Note:
25+
26+
- `Pct` means percentage of pixels in this category in all pixels.
27+
28+
### Visualization
29+
30+
![bac](https://raw.githubusercontent.com/uni-medical/medical-datasets-visualization/main/2d/semantic_seg/dermoscopy/isic2016_task1/isic2016_task1.png)
31+
32+
### Prerequisites
33+
34+
- Python 3.8
35+
- PyTorch 1.10.0
36+
- pillow(PIL) 9.3.0
37+
- scikit-learn(sklearn) 1.2.0
38+
- [MIM](https://github.com/open-mmlab/mim) v0.3.4
39+
- [MMCV](https://github.com/open-mmlab/mmcv) v2.0.0rc4
40+
- [MMEngine](https://github.com/open-mmlab/mmengine) v0.2.0 or higher
41+
- [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) v1.0.0rc5
42+
43+
All the commands below rely on the correct configuration of PYTHONPATH, which should point to the project's directory so that Python can locate the module files. In isic2016_task1/ root directory, run the following line to add the current directory to PYTHONPATH:
44+
45+
```shell
46+
export PYTHONPATH=`pwd`:$PYTHONPATH
47+
```
48+
49+
### Dataset preparing
50+
51+
- download dataset from [here](https://challenge.isic-archive.com/data/#2016) and decompression data to path 'data/'.
52+
- run script `"python tools/prepare_dataset.py"` to split dataset and change folder structure as below.
53+
- run script `"python ../../tools/split_seg_dataset.py"` to split dataset and generate `train.txt` and `test.txt`. If the label of official validation set and test set can't be obtained, we generate `train.txt` and `val.txt` from the training set randomly.
54+
55+
```none
56+
mmsegmentation
57+
├── mmseg
58+
├── projects
59+
│ ├── medical
60+
│ │ ├── 2d_image
61+
│ │ │ ├── dermoscopy
62+
│ │ │ │ ├── isic2016_task1
63+
│ │ │ │ │ ├── configs
64+
│ │ │ │ │ ├── datasets
65+
│ │ │ │ │ ├── tools
66+
│ │ │ │ │ ├── data
67+
│ │ │ │ │ │ ├── train.txt
68+
│ │ │ │ │ │ ├── test.txt
69+
│ │ │ │ │ │ ├── images
70+
│ │ │ │ │ │ │ ├── train
71+
│ │ │ │ | │ │ │ ├── xxx.png
72+
│ │ │ │ | │ │ │ ├── ...
73+
│ │ │ │ | │   │   │ └── xxx.png
74+
│ │ │ │ │ │ │ ├── test
75+
│ │ │ │ | │ │ │ ├── yyy.png
76+
│ │ │ │ | │ │ │ ├── ...
77+
│ │ │ │ | │   │   │ └── yyy.png
78+
│ │ │ │ │ │ ├── masks
79+
│ │ │ │ │ │ │ ├── train
80+
│ │ │ │ | │ │ │ ├── xxx.png
81+
│ │ │ │ | │ │ │ ├── ...
82+
│ │ │ │ | │   │   │ └── xxx.png
83+
│ │ │ │ │ │ │ ├── test
84+
│ │ │ │ | │ │ │ ├── yyy.png
85+
│ │ │ │ | │ │ │ ├── ...
86+
│ │ │ │ | │   │   │ └── yyy.png
87+
```
88+
89+
### Training commands
90+
91+
```shell
92+
mim train mmseg ./configs/${CONFIG_PATH}
93+
```
94+
95+
To train on multiple GPUs, e.g. 8 GPUs, run the following command:
96+
97+
```shell
98+
mim train mmseg ./configs/${CONFIG_PATH} --launcher pytorch --gpus 8
99+
```
100+
101+
### Testing commands
102+
103+
```shell
104+
mim test mmseg ./configs/${CONFIG_PATH} --checkpoint ${CHECKPOINT_PATH}
105+
```
106+
107+
<!-- List the results as usually done in other model's README. [Example](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/configs/fcn#results-and-models)
108+
109+
You should claim whether this is based on the pre-trained weights, which are converted from the official release; or it's a reproduced result obtained from retraining the model in this project. -->
110+
111+
## Results
112+
113+
### ISIC-2016 Task1
114+
115+
| Method | Backbone | Crop Size | lr | mIoU | mDice | config |
116+
| :-------------: | :------: | :-------: | :----: | :--: | :---: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
117+
| fcn_unet_s5-d16 | unet | 512x512 | 0.01 | - | - | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_isic2016-task1-512x512.py) |
118+
| fcn_unet_s5-d16 | unet | 512x512 | 0.001 | - | - | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_isic2016-task1-512x512.py) |
119+
| fcn_unet_s5-d16 | unet | 512x512 | 0.0001 | - | - | [config](https://github.com/open-mmlab/mmsegmentation/tree/dev-1.x/projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_isic2016-task1-512x512.py) |
120+
121+
## Checklist
122+
123+
- [x] Milestone 1: PR-ready, and acceptable to be one of the `projects/`.
124+
125+
- [x] Finish the code
126+
127+
- [x] Basic docstrings & proper citation
128+
129+
- [x] Test-time correctness
130+
131+
- [x] A full README
132+
133+
- [x] Milestone 2: Indicates a successful model implementation.
134+
135+
- [x] Training-time correctness
136+
137+
- [ ] Milestone 3: Good to be a part of our core package!
138+
139+
- [ ] Type hints and docstrings
140+
141+
- [ ] Unit tests
142+
143+
- [ ] Code polishing
144+
145+
- [ ] Metafile.yml
146+
147+
- [ ] Move your modules into the core package following the codebase's file hierarchy structure.
148+
149+
- [ ] Refactor your modules into the core package following the codebase's file hierarchy structure.

‎projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.0001-20k_isic2016-task1-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py', './isic2016-task1_512x512.py',
3+
'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.isic2016-task1_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.0001)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.001-20k_isic2016-task1-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py', './isic2016-task1_512x512.py',
3+
'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.isic2016-task1_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.001)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/dermoscopy/isic2016_task1/configs/fcn-unet-s5-d16_unet_1xb16-0.01-20k_isic2016-task1-512x512.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'mmseg::_base_/models/fcn_unet_s5-d16.py', './isic2016-task1_512x512.py',
3+
'mmseg::_base_/default_runtime.py',
4+
'mmseg::_base_/schedules/schedule_20k.py'
5+
]
6+
custom_imports = dict(imports='datasets.isic2016-task1_dataset')
7+
img_scale = (512, 512)
8+
data_preprocessor = dict(size=img_scale)
9+
optimizer = dict(lr=0.01)
10+
optim_wrapper = dict(optimizer=optimizer)
11+
model = dict(
12+
data_preprocessor=data_preprocessor,
13+
decode_head=dict(num_classes=2),
14+
auxiliary_head=None,
15+
test_cfg=dict(mode='whole', _delete_=True))
16+
vis_backends = None
17+
visualizer = dict(vis_backends=vis_backends)

‎projects/medical/2d_image/dermoscopy/isic2016_task1/configs/isic2016-task1_512x512.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
dataset_type = 'ISIC2017Task1'
2+
data_root = 'data/'
3+
img_scale = (512, 512)
4+
train_pipeline = [
5+
dict(type='LoadImageFromFile'),
6+
dict(type='LoadAnnotations'),
7+
dict(type='Resize', scale=img_scale, keep_ratio=False),
8+
dict(type='RandomFlip', prob=0.5),
9+
dict(type='PhotoMetricDistortion'),
10+
dict(type='PackSegInputs')
11+
]
12+
test_pipeline = [
13+
dict(type='LoadImageFromFile'),
14+
dict(type='Resize', scale=img_scale, keep_ratio=False),
15+
dict(type='LoadAnnotations'),
16+
dict(type='PackSegInputs')
17+
]
18+
train_dataloader = dict(
19+
batch_size=16,
20+
num_workers=4,
21+
persistent_workers=True,
22+
sampler=dict(type='InfiniteSampler', shuffle=True),
23+
dataset=dict(
24+
type=dataset_type,
25+
data_root=data_root,
26+
ann_file='train.txt',
27+
data_prefix=dict(img_path='images/', seg_map_path='masks/'),
28+
pipeline=train_pipeline))
29+
val_dataloader = dict(
30+
batch_size=1,
31+
num_workers=4,
32+
persistent_workers=True,
33+
sampler=dict(type='DefaultSampler', shuffle=False),
34+
dataset=dict(
35+
type=dataset_type,
36+
data_root=data_root,
37+
ann_file='test.txt',
38+
data_prefix=dict(img_path='images/', seg_map_path='masks/'),
39+
pipeline=test_pipeline))
40+
test_dataloader = val_dataloader
41+
val_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU', 'mDice'])
42+
test_evaluator = dict(type='IoUMetric', iou_metrics=['mIoU', 'mDice'])

‎projects/medical/2d_image/dermoscopy/isic2016_task1/datasets/isic2016-task1_dataset.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
from mmseg.datasets import BaseSegDataset
2+
from mmseg.registry import DATASETS
3+
4+
5+
@DATASETS.register_module()
6+
class ISIC2017Task1(BaseSegDataset):
7+
"""ISIC2017Task1 dataset.
8+
9+
In segmentation map annotation for ISIC2017Task1,
10+
``reduce_zero_label`` is fixed to False. The ``img_suffix``
11+
is fixed to '.png' and ``seg_map_suffix`` is fixed to '.png'.
12+
13+
Args:
14+
img_suffix (str): Suffix of images. Default: '.png'
15+
seg_map_suffix (str): Suffix of segmentation maps. Default: '.png'
16+
reduce_zero_label (bool): Whether to mark label zero as ignored.
17+
Default to False.
18+
"""
19+
METAINFO = dict(classes=('normal', 'skin lesion'))
20+
21+
def __init__(self,
22+
img_suffix='.png',
23+
seg_map_suffix='.png',
24+
reduce_zero_label=False,
25+
**kwargs) -> None:
26+
super().__init__(
27+
img_suffix=img_suffix,
28+
seg_map_suffix=seg_map_suffix,
29+
reduce_zero_label=reduce_zero_label,
30+
**kwargs)

‎projects/medical/2d_image/dermoscopy/isic2016_task1/tools/prepare_dataset.py‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
import glob
2+
import os
3+
import shutil
4+
5+
import numpy as np
6+
from PIL import Image
7+
8+
9+
def check_maskid(train_imgs):
10+
for i in train_masks:
11+
img = Image.open(i)
12+
print(np.unique(np.array(img)))
13+
14+
15+
def reformulate_file(image_list, mask_list):
16+
file_list = []
17+
for idx, (imgp,
18+
maskp) in enumerate(zip(sorted(image_list), sorted(mask_list))):
19+
item = {'image': imgp, 'label': maskp}
20+
file_list.append(item)
21+
return file_list
22+
23+
24+
def check_file_exist(pair_list):
25+
rel_path = os.getcwd()
26+
for idx, sample in enumerate(pair_list):
27+
image_path = sample['image']
28+
assert os.path.exists(os.path.join(rel_path, image_path))
29+
if 'label' in sample:
30+
mask_path = sample['label']
31+
assert os.path.exists(os.path.join(rel_path, mask_path))
32+
print('all file path ok!')
33+
34+
35+
def convert_maskid(mask):
36+
# add mask id conversion
37+
arr_mask = np.array(mask).astype(np.uint8)
38+
arr_mask[arr_mask == 255] = 1
39+
return Image.fromarray(arr_mask)
40+
41+
42+
def process_dataset(file_lists, part_dir_dict):
43+
for ith, part in enumerate(file_lists):
44+
part_dir = part_dir_dict[ith]
45+
for sample in part:
46+
# read image and mask
47+
image_path = sample['image']
48+
if 'label' in sample:
49+
mask_path = sample['label']
50+
51+
basename = os.path.basename(image_path)
52+
targetname = basename.split('.')[0] # from image name
53+
54+
# check image file
55+
img_save_path = os.path.join(root_path, 'images', part_dir,
56+
targetname + save_img_suffix)
57+
if not os.path.exists(img_save_path):
58+
if not image_path.endswith('.png'):
59+
src = Image.open(image_path)
60+
src.save(img_save_path)
61+
else:
62+
shutil.copy(image_path, img_save_path)
63+
64+
if mask_path is not None:
65+
mask_save_path = os.path.join(root_path, 'masks', part_dir,
66+
targetname + save_seg_map_suffix)
67+
if not os.path.exists(mask_save_path):
68+
# check mask file
69+
mask = Image.open(mask_path).convert('L')
70+
# convert mask id
71+
mask = convert_maskid(mask)
72+
if not mask_path.endswith('.png'):
73+
mask.save(mask_save_path)
74+
else:
75+
mask.save(mask_save_path)
76+
77+
# print image num
78+
part_dir_folder = os.path.join(root_path, 'images', part_dir)
79+
print(
80+
f'{part_dir} has {len(os.listdir(part_dir_folder))} images completed!' # noqa
81+
)
82+
83+
84+
if __name__ == '__main__':
85+
86+
root_path = 'data/' # original file
87+
img_suffix = '.jpg'
88+
seg_map_suffix = '.png'
89+
save_img_suffix = '.png'
90+
save_seg_map_suffix = '.png'
91+
92+
train_imgs = glob.glob('data/ISBI2016_ISIC_Part1_Training_Data/*' # noqa
93+
+ img_suffix)
94+
train_masks = glob.glob(
95+
'data/ISBI2016_ISIC_Part1_Training_GroundTruth/*' # noqa
96+
+ seg_map_suffix)
97+
98+
test_imgs = glob.glob('data/ISBI2016_ISIC_Part1_Test_Data/*' + img_suffix)
99+
test_masks = glob.glob(
100+
'data/ISBI2016_ISIC_Part1_Test_GroundTruth/*' # noqa
101+
+ seg_map_suffix)
102+
103+
assert len(train_imgs) == len(train_masks)
104+
assert len(test_imgs) == len(test_masks)
105+
106+
print(f'training images: {len(train_imgs)}, test images: {len(test_imgs)}')
107+
108+
os.system('mkdir -p ' + root_path + 'images/train/')
109+
os.system('mkdir -p ' + root_path + 'images/test/')
110+
os.system('mkdir -p ' + root_path + 'masks/train/')
111+
os.system('mkdir -p ' + root_path + 'masks/test/')
112+
113+
train_pair_list = reformulate_file(train_imgs, train_masks)
114+
test_pair_list = reformulate_file(test_imgs, test_masks)
115+
116+
check_file_exist(train_pair_list)
117+
check_file_exist(test_pair_list)
118+
119+
part_dir_dict = {0: 'train/', 1: 'test/'}
120+
process_dataset([train_pair_list, test_pair_list], part_dir_dict)

0 commit comments

Comments
 (0)