Release 0.12.2 (#901)

* fix pytorch deepspeech Dockerfile (#820)

* Bump to version 0.12.2-dev (#819)

* Update docs with new object detection datasets/models/metrics (#818)

* added object detection metrics

* adding xView and APRICOT docs

* update dataset doc

* fix typo

* don't throw spurious errors on container shutdown after armory exec or launch (#828)

* don't throw spurious errors on container shutdown after armory exec or launch

* only log a clean container exit when the exit is known clean

* Docs update (faq) (#831)

* added object detection metrics

* update faq with canonical preprocessing

* minor grammar nitpicks

* Canonical preprocessing for adversarial datasets and non-scenario datasets (#829)

* canonical adversarial datasets

* canonical datasets

* canonical datasets

* canonical datasets

* canonical datasets

* refactored ucf, xview, and apricot

* black formatting

Co-authored-by: lucas.cadalzo <[email protected]>

* Update datasets, licensing, scenarios, baseline models, and metrics (#826)

* Updating datasets docs

* Updating adversarial dataset docs

* Updating metrics docs

* Fix formatting

* Fix formatting

* Update licensing

* Update scenarios doc

* Fix formatting

* Revert "Fix formatting"

This reverts commit cac7eb9.

* Update baseline model docs

* Remove precision/recall metrics from doc

* Update datasets.md

* Updating dataset docs

* Updating dataset docs

* Fix MNIST dtype

* Update scenarios doc (#844)

* updated ASR scenario documentation

* update ASR documentation

* updated multimodal scenario documentation

* updated xview scenario documentation

* updated APRICOT scenario documentation

* updated APRICOT scenario documentation

* updated poison scenario documentation

* updated UCF101 scenario documentation

* updated UCF101 scenario documentation

* Defences fix (#839)

* chmod

* ensure dtype output

* variable y (#841)

* update download command line (#840)

* Update attack params (#845)

* update

* updated to max of test and dev datasets

* fast untarring when possible (#832)

* fast untarring when possible

* standard output

* Piping from stdin (#842)

* accept standard input

* example

* exception -> error

* reject configs on raw stdin

Co-authored-by: Adam Jacobson <[email protected]>

* xView model gpu fix (#853)

* added object detection metrics

* remove hardcoded device_type of cpu

* iou warning logic (#861)

* added object detection metrics

* warning is now thrown if the following holds for exactly one of the two boxes: all nonzero coordinates from a box are < 1

* scaled epsilon (#865)

* Better ASR Labels (#863)

* remove int check to enable str inputs

* label updates

* weight saving hack (#870)

* working FGSM config for ASR + lint (#859)

* apricot patch targeted AP metric (#866)

* added object detection metrics

* checkpoint progress with apricot_patch_targeted_AP_per_class metric; still need to decide how to handle duplicate predictions of targeted class

* ignore duplicate true positives for a single patch box; updated comments and changed variable names

* adding apricot patch metadata used in apricot_patch_targeted_AP_per_class metric

* minor edits to variable name and comments

* update apricot test to incorporate new metric

* make APRICOT_PATCHES a dict with id's for keys

* Update host-requirements.txt (#873)

* Add --skip-attack flag (#877)

* added object detection metrics

* added --skip-attack flag

* allow for passing --skip-benign and --skip-attack

* truncate very long videos (#880)

* Dataset index and slicing (#878)

* update split naming

* update dataset loading

* updated docs

* updated token tests

* working tests for slicing

* allow ordering and duplicates

* Sanity check model configurations (#875)

* Initial sanity checking

* Initial sanity checking

* New sanity checks. Ensuring baseline scenario compliance

* Moving to pytest framework

* Integrated into armory run command with validate-config flag

* Move test_config folder

* Explict check for classifier model

* Add specific ground truth box set

* Black formatting

* Updating docs

* Fix typo

* Pin to ART 1.4.2 and move to end of container installs (#887)

* added object detection metrics

* pin to ART 1.4.2 and move to end of docker container installs

* move ART installation to framework Dockerfiles

* add ART install to dev Dockerfiles

* UCF101 shape bug (#889)

* updated preprocessing

* revert

* Update xview results (#879)

* updated baseline results for xview

* formatting

* fix typos

Co-authored-by: lucas.cadalzo <[email protected]>

* Split metrics when targeted attack is present (#884)

* Split adversarial metrics relative to ground truth and targeted labels

* Flake8

* Updating metrics doc

* added pgd_patch art_experimental attack (#883)

* added pgd_patch art_experimental attack

* ran format_json

* set use_gpu false

* removing accidentally added files

Co-authored-by: lucas.cadalzo <[email protected]>

* updated configs (#897)

* Add type annotations to baseline models (#885)

* Add type annotations to baseline models

* Add docstrings to baseline models

* Update type annotations

* Update type annotation

* GTSRB canonical preprocessing (#888)

* Add canonical preprocessing for GTSRB

* Update canonical preprocessing for GTSRB

* Update GTSRB canonical preprocessing

* Update documentation

* Update scenario documentation

* Deepspeech nan rebase (#891)

* updated docker builds

* update docker file

* update

* xView perturbation metric (#898)

* added object detection metrics

* update l0 norm to divide by number of elements in array; update xView scenarios to use l0 norm

* update test_metrics.py

* Docs for --skip flags (#900)

* added object detection metrics

* adding docs for --skip flags

* Release version 0.12.2

Co-authored-by: Adam Jacobson <[email protected]>
Co-authored-by: lcadalzo <[email protected]>
Co-authored-by: davidslater <[email protected]>
Co-authored-by: lucas.cadalzo <[email protected]>
Co-authored-by: kevinmerchant <[email protected]>
Co-authored-by: yusong-tan <[email protected]>
Co-authored-by: Adam Jacobson <[email protected]>

Author: 7 people

README.md

@@ -102,7 +102,7 @@ evaluated against that given scenario. For more information please see our
 
 # FAQs
 Please see the [frequently asked questions](docs/faqs.md) documentation for more information on:
-* Datasets returning NumPy arrays
+* Dataset format and preprocessing
 * Access to underlying models from wrapped classifiers.
 
 # Contributing

armory/__init__.py

@@ -27,7 +27,7 @@
 
 
 # Semantic Version
-__version__ = "0.12.1"
+__version__ = "0.12.2"
 DEV = "-dev"
 
 

armory/__main__.py

@@ -26,7 +26,7 @@
 from armory.eval import Evaluator
 from armory.docker import images
 from armory.utils import docker_api
-from armory.utils.configuration import load_config
+from armory.utils.configuration import load_config, load_config_stdin
 
 logger = logging.getLogger(__name__)
 
@@ -66,7 +66,10 @@ def __call__(self, parser, namespace, values, option_string=None):
             setattr(namespace, self.dest, values)
 
 
-DEFAULT_SCENARIO = "https://github.com/twosixlabs/armory-example/blob/master/scenario_download_configs/scenarios-set1.json"
+OLD_SCENARIOS = [
+    "https://github.com/twosixlabs/armory-example/blob/master/scenario_download_configs/scenarios-set1.json"
+]
+DEFAULT_SCENARIO = "https://github.com/twosixlabs/armory-example/blob/master/scenario_download_configs/scenarios-set2.json"
 
 
 class DownloadConfig(argparse.Action):
@@ -159,6 +162,25 @@ def _docker_image(parser):
     )
 
 
+def _docker_image_optional(parser):
+    parser.add_argument(
+        "--docker-image",
+        default=images.TF1,
+        metavar="<docker image>",
+        type=str,
+        help="docker image framework: 'tf1', 'tf2', or 'pytorch'",
+        action=DockerImage,
+    )
+
+
+def _skip_docker_images(parser):
+    parser.add_argument(
+        "--skip-docker-images",
+        action="store_true",
+        help="Whether to skip downloading docker images",
+    )
+
+
 def _no_docker(parser):
     parser.add_argument(
         "--no-docker",
@@ -209,7 +231,10 @@ def _set_outputs(config, output_dir, output_filename):
 def run(command_args, prog, description):
     parser = argparse.ArgumentParser(prog=prog, description=description)
     parser.add_argument(
-        "filepath", metavar="<json_config>", type=str, help="json config file"
+        "filepath",
+        metavar="<json_config>",
+        type=str,
+        help="json config file. Use '-' to accept standard input or pipe.",
     )
     _debug(parser)
     _interactive(parser)
@@ -243,21 +268,43 @@ def run(command_args, prog, description):
         action="store_true",
         help="Skip benign inference and metric calculations",
     )
+    parser.add_argument(
+        "--skip-attack",
+        action="store_true",
+        help="Skip attack generation and metric calculations",
+    )
+    parser.add_argument(
+        "--validate-config",
+        action="store_true",
+        help="Validate model configuration against several checks",
+    )
 
     args = parser.parse_args(command_args)
     coloredlogs.install(level=args.log_level)
 
     try:
-        config = load_config(args.filepath)
+        if args.filepath == "-":
+            if sys.stdin.isatty():
+                logging.error(
+                    "Cannot read config from raw 'stdin'; must pipe or redirect a file"
+                )
+                sys.exit(1)
+            logger.info("Reading config from stdin...")
+            config = load_config_stdin()
+        else:
+            config = load_config(args.filepath)
     except ValidationError as e:
         logger.error(
             f"Could not validate config: {e.message} @ {'.'.join(e.absolute_path)}"
         )
         sys.exit(1)
     except json.decoder.JSONDecodeError:
-        logger.exception(f"Could not decode {args.filepath} as a json file.")
-        if not args.filepath.lower().endswith(".json"):
-            logger.warning(f"{args.filepath} is not a '*.json' file")
+        if args.filepath == "-":
+            logger.error("'stdin' did not provide a json-parsable input")
+        else:
+            logger.error(f"Could not decode '{args.filepath}' as a json file.")
+            if not args.filepath.lower().endswith(".json"):
+                logger.warning(f"{args.filepath} is not a '*.json' file")
         sys.exit(1)
     _set_gpus(config, args.use_gpu, args.no_gpu, args.gpus)
     _set_outputs(config, args.output_dir, args.output_filename)
@@ -270,6 +317,8 @@ def run(command_args, prog, description):
         check_run=args.check,
         num_eval_batches=args.num_eval_batches,
         skip_benign=args.skip_benign,
+        skip_attack=args.skip_attack,
+        validate_config=args.validate_config,
     )
     sys.exit(exit_code)
 
@@ -295,6 +344,8 @@ def download(command_args, prog, description):
     """
     parser = argparse.ArgumentParser(prog=prog, description=description)
     _debug(parser)
+    _docker_image_optional(parser)
+    _skip_docker_images(parser)
     parser.add_argument(
         metavar="<download data config file>",
         dest="download_config",
@@ -325,12 +376,16 @@ def download(command_args, prog, description):
         model_weights.download_all(args.download_config, args.scenario)
         return
 
-    if not armory.is_dev():
-        logger.info("Downloading all docker images....")
+    if args.skip_docker_images:
+        logger.info("Skipping docker image downloads...")
+    elif armory.is_dev():
+        logger.info("Dev version. Must build docker images locally with build-dev.sh")
+    else:
+        logger.info("Downloading all docker images...")
         _pull_docker_images()
 
     logger.info("Downloading requested datasets and model weights...")
-    config = {"sysconfig": {"docker_image": images.TF1}}
+    config = {"sysconfig": {"docker_image": args.docker_image}}
 
     rig = Evaluator(config)
     cmd = "; ".join(

armory/art_experimental/attacks/pgd_patch.py

@@ -0,0 +1,34 @@
+from art.attacks.evasion import ProjectedGradientDescent
+import numpy as np
+
+
+class PGDPatch(ProjectedGradientDescent):
+    def __init__(self, estimator, **kwargs):
+        super().__init__(estimator=estimator, **kwargs)
+
+    def generate(self, x, y=None, **generate_kwargs):
+        if "ymin" in generate_kwargs:
+            ymin = generate_kwargs["ymin"]
+        else:
+            raise ValueError("generate_kwargs did not define 'ymin'")
+
+        if "xmin" in generate_kwargs:
+            xmin = generate_kwargs["xmin"]
+        else:
+            raise ValueError("generate_kwargs did not define 'xmin'")
+
+        mask = np.zeros(shape=x.shape[1:])
+        if "patch_ratio" in generate_kwargs:
+            patch_ratio = generate_kwargs["patch_ratio"]
+            ymax = ymin + int(x.shape[1] * patch_ratio ** 0.5)
+            xmax = xmin + int(x.shape[2] * patch_ratio ** 0.5)
+            mask[ymin:ymax, xmin:xmax, :] = 1
+        elif "patch_height" in generate_kwargs and "patch_width" in generate_kwargs:
+            patch_height = generate_kwargs["patch_height"]
+            patch_width = generate_kwargs["patch_width"]
+            mask[ymin : ymin + patch_height, xmin : xmin + patch_width, :] = 1
+        else:
+            raise ValueError(
+                "generate_kwargs did not define 'patch_ratio', or it did not define 'patch_height' and 'patch_width'"
+            )
+        return super().generate(x, y=y, mask=mask)

armory/art_experimental/defences/jpeg_compression_multichannel_image.py

@@ -20,6 +20,7 @@ def __init__(
         mins=None,
         ranges=None,
         n_channels=14,
+        dtype=np.float32,
     ):
         super().__init__(
             clip_values,
@@ -30,11 +31,15 @@ def __init__(
         )
         if mins is None:
             mins = (0.0,) * n_channels  # identity operation
-        self.mins = mins
+        if len(mins) != n_channels:
+            raise ValueError(f"mins must have {n_channels} values, one per channel")
+        self.mins = np.array(mins, dtype=dtype)
 
         if ranges is None:
             ranges = (1.0,) * n_channels  # identity operation
-        self.ranges = ranges
+        if len(ranges) != n_channels:
+            raise ValueError(f"ranges must have {n_channels} values, one per channel")
+        self.ranges = np.array(ranges, dtype=dtype)
 
     def __call__(self, x, y=None):
         x = (x - self.mins) / self.ranges

armory/art_experimental/defences/jpeg_compression_normalized.py

@@ -17,6 +17,7 @@ def __init__(
         apply_predict=False,
         means=None,
         stds=None,
+        dtype=np.float32,
     ):
         super().__init__(
             clip_values,
@@ -29,13 +30,13 @@ def __init__(
             means = (0.0, 0.0, 0.0)  # identity operation
         if len(means) != 3:
             raise ValueError("means must have 3 values, one per channel")
-        self.means = means
+        self.means = np.array(means, dtype=dtype)
 
         if stds is None:
             stds = (1.0, 1.0, 1.0)  # identity operation
         if len(stds) != 3:
             raise ValueError("stds must have 3 values, one per channel")
-        self.stds = stds
+        self.stds = np.array(stds, dtype=dtype)
 
     def __call__(self, x, y=None):
         x = x * self.stds + self.means

armory/art_experimental/defences/video_compression_normalized.py

@@ -19,7 +19,7 @@ def __init__(
         transpose=None,
         means=None,
         stds=None,
-        dtype=None,
+        dtype=np.float32,
         same_video=True,
     ):
         super().__init__(
@@ -51,8 +51,6 @@ def __init__(
             raise ValueError("stds must have 3 values, one per channel")
         self.stds = stds
 
-        if dtype is None:
-            dtype = "float32"
         self.dtype = dtype
 
         self.same_video = same_video
@@ -72,7 +70,6 @@ def __call__(self, x, y=None):
         if self.same_video:
             x = x.reshape(x_shape, order="C")
         x = (x - self.means) / self.stds
-        if self.dtype == "float32":
-            x = x.astype(np.float32)
+        x = x.astype(self.dtype)
         x = x.transpose(np.argsort(self.transpose))
         return x, y

armory/baseline_models/keras/cifar.py

@@ -1,14 +1,18 @@
 """
 CNN model for 32x32x3 image classification
 """
-import tensorflow as tf
+from typing import Optional
 
+import tensorflow as tf
 from tensorflow.keras.models import Sequential
 from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D
 from art.classifiers import KerasClassifier
 
 
 def make_cifar_model(**kwargs) -> tf.keras.Model:
+    """
+    This is a simple CNN for CIFAR-10 and does not achieve SotA performance
+    """
     model = Sequential()
     model.add(
         Conv2D(
@@ -42,7 +46,9 @@ def make_cifar_model(**kwargs) -> tf.keras.Model:
     return model
 
 
-def get_art_model(model_kwargs, wrapper_kwargs, weights_path=None):
+def get_art_model(
+    model_kwargs: dict, wrapper_kwargs: dict, weights_path: Optional[str] = None
+) -> KerasClassifier:
     model = make_cifar_model(**model_kwargs)
     if weights_path:
         model.load_weights(weights_path)

armory/baseline_models/keras/densenet121_resisc45.py

@@ -3,6 +3,8 @@
 
 Model contributed by: MITRE Corporation
 """
+from typing import Optional
+
 from art.classifiers import KerasClassifier
 import numpy as np
 import tensorflow as tf
@@ -24,6 +26,11 @@ def mean_std():
 
 
 def make_densenet121_resisc_model(**model_kwargs) -> tf.keras.Model:
+    """
+    This model is an ImageNet-pretrained DenseNet-121 that is fine-tuned on RESISC-45.
+    The initial layers transform the input from canonical form to the expected input
+    format for the DenseNet-121.
+    """
     input = tf.keras.Input(shape=(256, 256, 3))
 
     # Preprocessing layers
@@ -63,7 +70,9 @@ def make_densenet121_resisc_model(**model_kwargs) -> tf.keras.Model:
     return new_model
 
 
-def get_art_model(model_kwargs, wrapper_kwargs, weights_path):
+def get_art_model(
+    model_kwargs: dict, wrapper_kwargs: dict, weights_path: Optional[str] = None
+) -> KerasClassifier:
     model = make_densenet121_resisc_model(**model_kwargs)
     if weights_path:
         model.load_weights(weights_path)

armory/baseline_models/keras/inception_resnet_v2.py

@@ -1,14 +1,18 @@
 """
 Inception_ResNet_v2 CNN model for 299x299x3 image classification
 """
+from typing import Optional
+
 from art.classifiers import KerasClassifier
 import tensorflow as tf
 from tensorflow.keras.applications.inception_resnet_v2 import InceptionResNetV2
 from tensorflow.keras.layers import Lambda
 from tensorflow.keras.models import Model
 
 
-def get_art_model(model_kwargs, wrapper_kwargs, weights_path=None):
+def get_art_model(
+    model_kwargs: dict, wrapper_kwargs: dict, weights_path: Optional[str] = None
+):
     input = tf.keras.Input(shape=(224, 224, 3))
 
     # Preprocessing layers