A small bug in tools/analyse_logs.py caused by wrong plot_iter in some cases.

[Y-M-Y]

Hi, thank you for constructing such a fabulous toolbox. I meet a problem in using tools/analyse_logs.py. To be specific, there might be some bugs with the variable plot_iter in plotting iter-based curves.

Describe the bug
When using tools/analyse_logs.py to plot curves of iter-based variables such lr and loss, the curve will be wrong like the following figs. The lr schedule is poly, the curve should be a line, but it is wrong from iter200 to iter1344. So as the loss curve.

When the iteration of validation(eg. 1334) is larger than the interval of evaluation(eg. 200), the curve will be wrong. The code in analyse_logs generated the wrong plot_iter as x.

Reproduction

1.What command or script did you run?
python tools/analyze_logs.py xxx.log.json --keys lr

2.Here's my json logs.
logfile.zip

Environment

MMSegmentation v0.22.1

Bug fix

I have already analyzed the bug, it must be the error of plot_iter caused by 'iter': in the first 'mode': 'val' line in .json logs. The if...continue (line46-lin47) filtering too much useful iters in the list plot_iter. Maybe we can fix it by ignoring the iter of val mode.

                 for idx in range(len(epoch_logs[metric])):
                        if pre_iter > epoch_logs['iter'][idx]:
                            continue
                        pre_iter = epoch_logs['iter'][idx]
                        plot_iters.append(epoch_logs['iter'][idx])
                        plot_values.append(epoch_logs[metric][idx])

However, due to the limitation of my ability, I can't fix it. Sorry!

[MengzhangLI]

Hi, thanks for your kindely remainder. We would fix it as soon as possible!

Best,

[MengzhangLI]

Mark: This file needs seaborn package while our default conda environment does not need it.

[MengzhangLI]

Try to visulize https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_tiny_fp16_512x512_160k_ade20k/upernet_convnext_tiny_fp16_512x512_160k_ade20k_20220227_124553.log.json

[Y-M-Y]

Try to visulize https://download.openmmlab.com/mmsegmentation/v0.5/convnext/upernet_convnext_tiny_fp16_512x512_160k_ade20k/upernet_convnext_tiny_fp16_512x512_160k_ade20k_20220227_124553.log.json

Thanks，this log also acts good in my environmental. In this log，the "iter" of the "mode": "val" is 250, which is smaller than evaluation interval = 16000， so it won't cause the error.

[MengzhangLI]

I find line46-lin47 is desiged for skip val line.

In this case pre_iter is 32000 > epoch_logs['iter'][idx] 250.

When I scaned your log.json, I can not figure it out why your val mode is iter:1334, seems like you train your model on single GPU with 1,334 images in validation. Am I right?

So this phenomenon should we sum up to too many images (1334) in validation + too small val interval (interval=200)? At first several epochs 1334 is larger than its neighbour in epoch_logs['iter'].

[MengzhangLI]

By the way, I suggest use resnet50v1c pretrained model rather than checkpoint model trained on cityscapes. Because I think ImageNet1K as an upstream dataset may help your tasks and cityscapes dataset maybe unrelated with your dataset.

[Y-M-Y]

The summary is right! I just started learning about segmentation and I'm not familiar with some habits or parameters. Thank you very much for your patience.

[Y-M-Y]

By the way, I suggest use resnet50v1c pretrained model rather than checkpoint model trained on cityscapes. Because I think ImageNet1K as an upstream dataset may help your tasks and cityscapes dataset maybe unrelated with your dataset.

Thanks again, my dataset is from kaggle steel-defect-seg. The dataset is really imbalanced and I'm trying to improve the mIoU. Your suggestion means a lot to me!

[MengzhangLI]

I have made a PR to fix this bug:#1428.

Just use

  if epoch_logs['mode'][idx] == 'train':
      plot_iters.append(epoch_logs['iter'][idx])
      plot_values.append(epoch_logs[metric][idx])

to replace

  if pre_iter > epoch_logs['iter'][idx]:
      continue

Best,

[Y-M-Y]

Oh！I have tried it, but I wrote it in the wrong way! Such a pity!
I used

  if epoch_logs['mode'][idx] is 'train':
      plot_iters.append(epoch_logs['iter'][idx])
      plot_values.append(epoch_logs[metric][idx])

Reality is I confused is and ==!!!

Thanks a lot！I will learn more about python basic.