Francesco Croce, Naman D Singh, Matthias Hein

University of Tübingen

Paper

Abstract While a large amount of work has focused on designing adversarial attacks against image classifiers, only a few methods exist to attack semantic segmentation models. We show that attacking segmentation models presents task-specific challenges, for which we propose novel solutions. Our final evaluation protocol outperforms existing methods, and shows that those can overestimate the robustness of the models. Additionally, so far adversarial training, the most successful way for obtaining robust image classifiers, could not be successfully applied to semantic segmentation. We argue that this is because the task to be learned is more challenging, and requires significantly higher computational effort than for image classification. As a remedy, we show that by taking advantage of recent advances in robust ImageNet classifiers, one can train adversarially robust segmentation models at limited computational cost by fine-tuning robust backbones.

Main dependencies: PyTorch-2.0.0, torchvision-0.15.0, timm-0.6.2, AutoAttack

Run runner_infer.sh with the models config (.yaml) file from configs folder.

This computes the final adversarial robustness for the particular dataset and model passed as arguments within the .yaml file.

SLURM type setup in runner.sh , run with location_of_config file and num_of_gpu as arguments.

For non-SLURM directly run train.py with location_of_config file and num_of_gpu as arguments.

• For UperNet with ConvNext (both Tiny and Small versions) backbone for ADE20K

  • Clean-training: config-file: ade20k_convnext_cvst.yaml set BACKBONE with CONVNEXT-S_CVST for Small model.
  • Adversarial-training: config-file: ade20k_convnext_rob_cvst.yaml set BACKBONE with CONVNEXT-S_CVST for Small model.

• For UperNet with ConvNext (both Tiny and Small versions) backbone for PASCALVOC

  • Clean-training: config-file: pascalvoc_convnext_cvst.yaml set BACKBONE with CONVNEXT-S_CVST for Small model.
  • Adversarial-training: config-file: pascalvoc_convnext_rob_cvst.yaml set BACKBONE with CONVNEXT-S_CVST for Small model.

• For SegMenter with Vit-S backbone for ADE20K dataset

  • Adversarial-training: config-file: ade20k_segmenter_clean.yaml, set ADVERSARIAL to FALSE for clean training.

We make our robust models publically available. mIoU is reported for clean evaluation and with SEA evaluation (Adv.) at two perturbation strengths.

*SegMenter-ViT-S model available soon. Note: the models are trained including the background class for both VOC and ADE20K.

Robust pre-trained backbone models were taken from Revisiting-AT* github repository.

For UperNet we always use the ConvNext backbone with Convolution Stem (CvSt).

If you use our code/models consider citing us with the follwong BibTex entry:

@article{croce2023robust,
 title={Robust Semantic Segmentation: Strong Adversarial Attacks and Fast Training of Robust Models}, 
 author={Francesco Croce and Naman D Singh and Matthias Hein},
 year={2023},
 journal={arXiv:2306.12941}}

Also consider citing SegPGD if you use SEA attack, as their loss funciton makes up a part of SEA evaluation.

The code in this repo is partially based on the following publically available codebases.

• https://github.com/hszhao/semseg
• https://github.com/Tramac/awesome-semantic-segmentation-pytorch
• https://github.com/rstrudel/segmenter)
• https://github.com/facebookresearch/ConvNeXt/tree/main/semantic_segmentation
• https://huggingface.co/docs/transformers/main/en/model_doc/upernet#transformers.UperNetForSemanticSegmentation