What's Changed

• round to nearest int when exporting, rather than rounding down by @lcadalzo in #1860
• DP-instahide poison defense by @swsuggs in #1841
• fix video tracking export bug by @lcadalzo in #1862
• use isort to sort python imports by @mwartell in #1789
• Sleeper agent fix by @swsuggs in #1863

Full Changelog: v0.16.2...v0.16.3

This release is to support evaluation period 6. Please note that there are now only two flavors of Armory containers:

1. twosixarmory/armory:0.16.2 which has merged PyTorch, TF2, and CARLA_MOT flavors
2. twosixarmory/pytorch-deepspeech which has to stand separate because it could not be folded into armory

Headlines

• Simplify custom metrics by @ppark-twosixtech in #1806
• replace carla-mot and pytorch image names to "armory" in scenario_configs by @mwartell in #1838
• Eval6 - CARLA MOT test data by @yusong-tan in #1827
• Eval 6 - CARLA Overhead Object Detection Test Data by @yusong-tan in #1851
• bump art to 1.13 by @lcadalzo in #1830

Other changes

• fix intermittent macOS CI timing results by @mwartell in #1794
• Clear jupyter notebook output images by @mwartell in #1795
• User init by @davidslater in #1783
• remove GARD-specific image not in codebase by @davidslater in #1799
• better global metrics from config by @davidslater in #1753
• ignore deletes in pre-commit hooks by @davidslater in #1809
• Merge pytorch, tf2, and carla mot images by @christopherwoodall in #1786
• Patch readme by @christopherwoodall in #1817
• remove optional art dependency by @mwartell in #1833
• fix numpy ci failures by @lcadalzo in #1835
• Eval6 audio test triggers by @swsuggs in #1831
• Instrumentation Examples with Probes, Meters, and Writers by @ppark-twosixtech in #1821
• updating VideoTrackingExporter class for new label format by @lcadalzo in #1825
• runs scenario checks on self-hosted runner by @mwartell in #1844
• audio poisoning baseline results by @swsuggs in #1848

Full Changelog: v0.16.1...v0.16.2

Headlines

• Update to ART 1.12.2 by @christopherwoodall in #1763
• Add TIDE Metrics by @ppark-twosixtech in #1762

Scenario / Dataset / Model Updates

• functions for mapping between MOT and COCO by @davidslater in #1715
• Modify how armory handles targeted attacks by @lcadalzo in #1730
• enable configurable tensorflow c++ logging by @davidslater in #1744
• make carla attacks backwards compatible by @lcadalzo in #1731
• Eval6 export tweaks by @swsuggs in #1737
• Oom error poisoning by @davidslater in #1767

Documentation

• document hyrda-lightning install for no-docker users by @mwartell in #1682
• Eval 6 - Fine-tune CARLA MOT by @yusong-tan in #1686
• Documentation & Dockerfile cleanup/fixes by @christopherwoodall in #1681
• pip 21 issue note by @davidslater in #1706
• Add live demo to example Notebook and README by @christopherwoodall in #1741
• Update scenario docs by @davidslater in #1754
• add pytest note for docker testing by @davidslater in #1770
• Get activations by batch for explanatory model to enable better scalability @davidslater in #1771
• Additional baseline results by @swsuggs in #1766
• update readme by @davidslater in #1773

Bug Fixes

• change output directory of CARLA video tracking exporter by @lcadalzo in #1713
• fix sleeper agent paths by @davidslater in #1721
• resisc sweep bug by @lcadalzo in #1740
• Fix xview sweep bug by @lcadalzo in #1739
• Activation bug resnet50 by @davidslater in #1768
• Retain data type for arrays passed into "relabel" function by @jprokos26 in #1772

New Contributors

• @ppark-twosixtech made their first contribution in #1762

Full Changelog: v0.16.0...v0.16.1

Scenario / Dataset / Model Updates

• New scenarios for CARLA multi-object tracking (carla-mot), CARLA overhead [multimodal] object detection, sleeper agent poisoning, and audio poisoning. See scenario_configs/eval6 for evaluation configs. A baseline HuBERT model is also provided there for ASR.
• New datasets to support the above scenarios, including two CARLA datasets (MOT and overhead [multimodal] object detection), and speech commands.

Dependency / Environment Updates

• Updated to ART 1.12.1, Pytorch 1.12, TensorFlow 2.10, and CUDA 11.6
• Updated docker / conda / pip installations to use environment.yml for conda and pyproject.toml for pip.
• Updated dockerfiles to leverage conda/pip installations.
• Removed all *requirements.txt files in favor of single installation pipeline.
• New Docker image carla-mot, for the new scenario. NOTE: may not be supported in version 0.17+

installing extra dependencies with pip

Previously you had to make complex selections from the *-requirements.txt files to run armory. Those dependencies now get installed along with armory. For example:

pip install armory-testbed[pytorch]

will bring the needed pytorch packages when installing armory. The other sets are named tensorflow and deepspeech. As always editable installations are supported, for example:

pip install --editable armory-testbed[tensorflow]

Detailed Changelog with PRs

• Task metrics by @davidslater in #1495
• Release fix by @davidslater in #1535
• Mkdocs update by @davidslater in #1534
• add global meter by @davidslater in #1528
• update mkdoc structure by @davidslater in #1551
• Remove unneeded scripts by @davidslater in #1544
• migrate adaptive attack tutorial by @lcadalzo in #1541
• allow total_wer() to take in list or np array by @lcadalzo in #1555
• Revert "omit push to pypi temporarily" by @mwartell in #1594
• bring v0.15.4 master back into develop by @mwartell in #1602
• speech commands dataset by @swsuggs in #1611
• Sleeper agent integration by @swsuggs in #1609
• Audio poisoning by @swsuggs in #1615
• Update Build System by @christopherwoodall in #1617
• black + yamllint + flake by @davidslater in #1629
• Build update by @davidslater in #1630
• Launch / Exec - better defaults by @davidslater in #1636
• remove [all] target from pyproject and CI because it lags CI badly by @mwartell in #1642
• buildable docker containers by @davidslater in #1640
• version take 2 by @davidslater in #1635
• version and ci omnibus merge by @mwartell in #1653
• Carla mutli-object tracking by @swsuggs in #1648
• Eval 6 - CARLA MOT Patch Attacks by @yusong-tan in #1624
• Eval 6 - CARLA MOT Baseline Model by @yusong-tan in #1625
• Eval 6 - CARLA MOT Dev Data by @yusong-tan in #1626
• Eval 6 - CARLA MOT Example Configs by @yusong-tan in #1627
• Baseline poison configs and bug fix by @swsuggs in #1632
• update imagenette version by @lcadalzo in #1568
• Total wer fix by @davidslater in #1643
• dockerfile for MOT by @davidslater in #1667
• Hubert ASR - replacement for deep speech without the dependencies by @davidslater in #1644
• update scenario for configurability by @davidslater in #1668
• Carla mot dataset by @swsuggs in #1658
• Pre release documentation cleanup by @christopherwoodall in #1650
• build and release new docker image to support MOT scenario by @christopherwoodall in #1663
• Eval6 Object Detection Integration by @jprokos26 in #1628
• Eval6 - CARLA Object Detection Dev Data by @yusong-tan in #1605
• Eval6 - CARLA Object Detection Train/Val Data by @yusong-tan in #1604
• Eval 6 - CARLA Object Detection Depth Attack Update by @yusong-tan in #1606
• Eval6 - CARLA Object Detection Scenario Configs by @yusong-tan in #1607
• Export carla MOT by @swsuggs in #1665
• Mot metrics meter by @davidslater in #1672
• WIP: MOT: update task results by @davidslater in #1669
• release 0.16.0 by @mwartell in #1673

New Contributors

• @christopherwoodall made their first contribution in #1617
• @jprokos26 made their first contribution in #1628

Full Changelog: v0.15.4...v0.16.0

Changelog

Datasets

• updated test sets for CARLA video tracking and object detection (#1579)

Poisoning

• added a CIFAR10 test backdoor trigger (#1586)
• added an SGD version of the baseline resnet18 (#1587)

Documentation

• Updated scenarios.md with baseline results for CARLA dev sets (#1552) and CARLA test sets (#1585)

There was a build error in v0.15.2 so it should not be used. Use v0.15.3 (this release) or later.

Changelog

Poisoning

• Added attack success rate metric for witches brew scenario (#1548)
• Fixed issue with witches brew scenario where the attack was being fed an untrained model (#1558)

CARLA Video tracking

• Enabled clipping of videos to mitigate GPU OOM issue (#1547)
• Upgraded to ART 1.10.3, which includes an improvement to AdversarialTexture video tracking attack (#1550)
• Corrected misspelled defense module in video tracking default config (#1557)

Dependencies

• Upgraded to ART 1.10.3 (#1550)

Documentation

• Updated scenarios.md file with baseline results for Eval 5 scenarios (#1552)

Changelog

Eval 5

Add multimodal weights to CARLA scenario on new data #1515
Updated CARLA video attack #1516
Add numerous baseline poisoning configs, including perfect filtering #1484
Updated to ART 1.10.2 #1525

Bug Fixes

Config fixes and cleanup #1501 #1502 #1507 #1525
DAPRICOT bug fixes pertaining to metric measurement #1488

User Experience

Added ability to export inputs at user-defined places in the code, e.g. in model code following preprocessing operations #1488
Reduced time to import armory #1521

Documentation

Poisoning documentation update #1503
Minor updates to export API #1488

Instrumentation and Measurement

Writer updates - enable limits on max record size and total results size to prevent blowing up disk space. #1499
Add disconnect method to hub class #1519
Added ExportMeter class for exporting inputs at arbitrary places in code #1488

Development and Testing

Update release yaml #1483
Add black[jupyter] for linting (added the jupyter extension) #1500
Use version string from armory as the canonical method #1522

Refactor

Postprocessing moved to own module. Unused webapi module removed. #1505
Poisoning scenario and fairness metrics refactoring #1503
Updated compute and perturbation metrics. Moved to armory.metrics.compute and armory.metrics.perturbation, improved testing, and enabled namespace-based access. #1490 #1493

Changelog

User Experience

New Armory instrument module revamps existing metrics computation and provides users with substantially enhanced flexibility in recording/measuring experimental artifacts of interest (#1387)
Added ability to export images with predicted and ground-truth bounding boxes overlaid for CARLA object detection and video tracking (#1261). Also saves bounding box data as coco-formatted json (#1398)
Revamped logging including more user control (#1307)
Log more informative output path info (#1366)
Armory now saves log files to the per-run output directory beside the output json file (#1307)
Deprecation of TF1 (#1290)

Dependency/library updates

Updated to black 22.3 (#1383)
Updated to ART 1.10.1 (#1427)
Updated to Pytorch 1.10, TensorFlow 2.8, and CUDA 11.3 (#1337)
Updated to TFDS v4 (#1323)
Removed apache_beam and pandas dependencies in the main codebase, as well as moved pymongo from requirements to host-requirements (#1277)

Datasets

New CARLA dataset for model training including train/val splits; mitigated annotation issues present in version 1 (#1412)
New CARLA object detection dev set (#1431); mitigated annotation issues present in version 1 (#1449)
New CARLA video tracking dev set (#1431)
Move UCF101 data clipping to the beginning of the dataset preprocessing pipeline instead of the baseline model's forward pass (and implicitly, the gradient computation) (#1309)

Attacks

Added support for a dirty-label poisoning attack on Cifar10 (#1403)
Integrated a new clean-label poisoning attack from ART, Witches’ Brew (#1406)
New CARLA AdversarialPatchPyTorch attack with substantial speed improvements; also includes depth-channel perturbation (#1400)
Updated CARLA Robust DPatch attack to support depth-channel perturbation (#1409)
Updated CARLA video tracking attack Adversarial Texture to support per-frame perspective transform of patch (#1410)

Models

New model weights for CARLA object detection, both RGB and multi-modal (#1445)
Update DeepSpeech 2 to version 3 (PyTorch Lightning) (#1293)

Scenarios

Added a new scenario to support the Witches’ Brew poisoning attack (#1406)
Integration of CIFAR10 dataset and trigger for poisoning scenario (#1403)
Added example of using entailment-based adversarial target labels for ASR scenario (#1407)

Metrics

New Armory instrument module (#1387)
Integrated entailment metric for ASR (#1407)
Added mean success rate metric for CARLA video tracking (#1408)
Additional statistics measured for poisoning scenarios, particularly for filter-based defenses (#1393)
Added poisoning-specific metrics that attempt to characterize the fairness or bias of models and filters (#1360)

Documentation

Added Jupyter notebook tutorial for executing Armory scenarios step-by-step (#1441)
Updated metrics.md with instructions for using new Armory instrument feature (#1387)

Default Configs

"docker_image" field no longer requires version number (see scenario config files for examples)
Organized default configs by scenario and evaluation period (#1374)
New CIFAR10 poisoning config (#1403)
New config for ASR entailment (#1407)

Performance Improvement / Bug Fixes

Fixed bugs present when running Armory with the --jupyter flag (#1416)
Provide consistent ART data pathing (#1352)
Fixed issue with GPU memory usage in PyTorch containers (#1341)
Refactored Docker builds such that there is an Armory base image (twosixarmory/base) that will be built infrequently (likely quarterly) and pushed to dockerhub (#1307)
Refactor of sample exporting feature (#1297)
Properly save model weights to Armory data directory instead of tmp (#1281)

Changelog

Bug Fixes

Fixed bug pertaining to carla_AP_per_class metric that was introduced in Armory 0.14.3.

Changelog

Dependency/library updates

Updated twosixarmory Docker images to ART 1.9.1 and to include tensorboardX dependency (#1240)

User experience

Integrated with ART's Tensorboard feature. Users can now view Tensorboard plots of losses and gradients for supported ART attacks that take a summary_writer kwarg. tfevent files are outputted to same Armory scenario output dir as results JSON's. (#1240)

Documentation

Updated scenarios.md with baseline results for CARLA object detection and video tracking scenarios (#1245)

Default Configs

Added default configs for baseline defended models for CARLA object detection and video tracking scenarios