Changelog

Breaking Changes

• Datasets now are delivered by default with canonical preprocessing (#748, #749, #750, #761, #768, #776, #777)
• Absolute weights paths are passed into get_art_model function (#771)

Dependency/library updates

• Update pytorch to 1.60 and torchvision to 0.7.0 (#765)
• Incorporate TF object detection API (#746)
• Create new container for ASR scenario (#763, #811)

Scenarios

• Integrate So2Sat data/model with image classification scenario (#764, #767)
• New object detection scenario with xView and APRICOT datasets/models (#755, #756, #758, #762, #770, #772, #783)
• New ASR scenario with data/model (#763, #813)

Changelog

Breaking Changes

• Configs using cifar/mnist datasets require updated docker container
• Custom scenarios require additional parameters

Datasets

• Enhanced Pytorch data loaders (#707)
• TFDS version upgrade (#722). NOTE: requires use of 0.11.1 or newer Docker container for CIFAR/MNIST datasets

Scenarios

• Updated poisoning scenario for Eval 1 Round 1 (#660)
• Enable use_label in audio and video scenarios (#686)
• Set classifier inputs to be immutable (#725)

Attacks

• Allow targeted attacks with systematic label selection such as a round-robin scheme or random targets (#690)
• Wrapper for ART patch attacks to enable better integration with Armory (#694)

Adversarial Datasets

• Label adversarial examples for RESISC45 dataset (#655)

Evaluation Infrastructure

• Custom pathing for evaluations (#663)
• Add flag for number of evaluation batches (#668)
• Disable shuffling of data during evaluation (#675)
• Create perturbation-accuracy plots from .json outputs (#689)
• Option to skip benign classification (#713)
• Add a --no-gpu flag (#733)

Metrics

• Computational resource usage metrics (#703)
• Image patch area metric (#701)
• Video metrics (#717)
• Allow multiple perturbation metrics (#735)

Documentation

• Update command line argument documentation (#681)
• Add dataset licensing details (#685)

User interface

• Allow armory configure to modify existing configuration (#736)

Changelog

Release notes

The 0.10.x branch switches to using ART v1.3 (performers needing ART 1.2 support should stay on the 0.9.x branch) (#648). The TF2 Docker container for 0.10.x now uses Tensorflow 2.2.0 and the Pytorch container for 0.10.x uses Pytorch 1.5 with torchvision 0.6.0.

Adversarial datasets

Updated Librispeech adversarial dataset (#635)

Changelog

Release notes

The 0.9.x branch will be last version to support ART v.1.2.

Infrastructure

• Fix windows compatibility issue with output directory names (#640)

Scenarios

• Fix GPU memory leak in UCF101 scenario (#637)

Changelog

Infrastructure

• Enable multiple external repos to be pulled (#617)

Configuration Files

• Add ART default parameters to configs for clarity (#623)

Scenarios

• Update sincnet baseline scenario to utilize full audio during inference (#625 #607)
• Allow separate models for training on poisoned and cleaned datasets (#611)
• Make poisoned splits deterministic (#624)

Examples

• Different training and inference mode examples in armory-example (twosixlabs/armory-example#44) and in librispeech (#625)

Datasets

• Experimental subset of PyTorch native DataLoaders (#618)
• Enable targeted attack labels in preloaded datasets (#622)

Changelog

Infrastructure

• Configurable output directory names (#594)
• Increase SHM size for Docker containers (#595)
• Optionally send evaluation results to a MongoDB instance (#581)
• Enable armory download in no-docker mode (#569)
• Document how to add non-Python paths as needed (#579)
• Improve error messages (#563, #564)
• Bugfix: SSL verification configuration check for external repo downloads (#566)
• Remove unused field budget from config files (#580)
• Fix port mapping for Docker containers (#593)

Attacks

• Fix PGD attacks to accept optional ground truth labels (#597)

Defenses

• Add new JpegCompression defense (#588)

Scenarios

• Make results of SincNet deterministic (#575)
• Improve randomization of poisoning scenario (#586)
• Update baseline metrics for image classification scenario (#589)

Datasets

• Add documentation for adversarial datasets (#596)
• Add initial documentation of dataset splits (#599)
• Create paired adversarial dataset for Imagenet adversarial samples (#576)
• Create paired adversarial dataset for RESISC45 (#568)

Metrics

• Add SNR perturbation metrics (#573)

Changelog

Infrastructure

• Private external repo env variable changed from GITHUB_TOKEN -> ARMORY_GITHUB_TOKEN
• Update access pattern for armory paths (#514)
  • Now uses paths.runtime_paths()
• Enabled armory to be ran on host without docker using --no-docker (#480)
  • Documentation
• Going forward there should be very light pulls needed for new armory docker containers (#477)
  • Recommend running armory clean after installation
• Custom docker containers are now enabled using armory launch (#467)
  • Also available in a configuration file as before
• Made SSL verification part of the global config (True by default) (#488)
  • May need to recreate global config with armory configure
• Both external_repo_dir and the root of cloned repositories are now added to python sys path during evaluation.
  • See documentation for more details
• Add --gpus flag for exec, launch, and run (#531)
• Various changes to how output and temp directories are created (No user impact)

Scenario Outputs

• Now use uuid for scenario output directory (#503)
• Added timestamp to output JSON (#524)

Jupyter

• Launching armory containers with --jupyter now uses token authentication (#551)

Datasets

• Enabled datasets to return either NumPy generator (default) or tf.data.Dataset (#481)
• Added pre-built adversarial datasets for UCF, RESISC and Librispeech (#541 #519 #530)

Video Scenario

• Fixed adversarial attack on video scenario (#464)

Poisoning Scenario

• Enable poison scenario to have no filtering component (#518)
• Update pre-processing function for baseline model (#526)

Examples

• Added TF Graph MNIST model and example config (#542)

Release Notes

This is the first stable release of Armory, which includes Scenarios to be evaluated against. If you've previously ran Armory we recommend that you clear your ~/.armory directory. Going forward transitions should be handled gracefully and this won't be needed.

0.6.0 is a major overhaul of the underlying framework.

Changelog

Documentation

• Significantly more documentation has been published regarding how to utilize the armory framework:
  https://github.com/twosixlabs/armory/tree/r0.6/docs

Scenarios

• Instead of eval_files the underlying evaluation is ran on a Scenario. Please see the documentation for more information: https://github.com/twosixlabs/armory/blob/r0.6/docs/scenarios.md
• These scenarios wrap underlying functionality like returning outputs into serialized files

Configuration Files

• We now have semi-strict schema verification. Detailed information regarding the new schema can be found in the docs: https://github.com/twosixlabs/armory/blob/r0.6/docs/configuration_files.md

Metrics

• New baseline metrics such as categorical_accuracy have been added for convenience. Users can still generate their own statistics in a custom scenario and simply return them as a dictionary. See the documentaiton for more information on metrics: https://github.com/twosixlabs/armory/blob/r0.6/docs/metrics.md

Baseline models and pretrained weights

• Baseline models and pretrained weights are available. Please see the documentation for more detailed information: https://github.com/twosixlabs/armory/blob/r0.6/docs/baseline_models.md

Examples

• Official scenarios as well as examples of how to create custom scenarios and configs have been added to armory-example repo

Changelog

Configuration

• New armory configure script added to set global configurations (e.g. the directories of all mounted volumes within the containers)
• armory --version now writes version number to stdout

Docker

• TF2 image now uses tf2.1 installed through conda for performance improvements

Datasets

• German Traffic Sign Dataset added

Examples

• Moved to a single source of examples in armory-example repo

Release Notes

• Since the mounting permissions have changes (see below) you'll need to remove ~/.armory if you had a previous installation
  • sudo rm -R ~/.armory
• Upon running 0.4 a new global config json will be placed in ~/.armory. This new config enables you to specify a directory where you want to store cached datasets on host.

Changelog

Docker

• Docker will now run as host user/group inside the contianer
  • Makes it so saved files will have user permissions when written onto mounted volumes
• Pin all versions in Dockerfile to improve file size as we add new docker images
• Increase ART version to 1.1.1
• New armory clean method which removes out of date docker images from your system

Configuration Files

• Dataset field now has two required fields
  • Module and Name
  • Mimics the other config fields
  • Enables custom dataset modules to be used

Jupyter

• Now launches jupyter lab instead of jupyter notebook

Packaging

• Automated docker image uploads
• Fixed some bugs with import methods