Unadversarial Examples: Designing Objects for Robust Vision

Overview

Unadversarial Examples: Designing Objects for Robust Vision

This repository contains the code necessary to replicate the major results of our paper:

Unadversarial Examples: Designing Objects for Robust Vision
Hadi Salman*, Andrew Ilyas*, Logan Engstrom*, Sai Vemprala, Aleksander Madry, Ashish Kapoor
Paper
Blogpost (MSR)
Blogpost (Gradient Science)

@article{salman2020unadversarial,
  title={Unadversarial Examples: Designing Objects for Robust Vision},
  author={Hadi Salman and Andrew Ilyas and Logan Engstrom and Sai Vemprala and Aleksander Madry and Ashish Kapoor},
  journal={arXiv preprint arXiv:2012.12235},
  year={2020}
}

Getting started

The following steps will get you set up with the required packages (additional packages are required for the 3D textures setting, described below):

  1. Clone our repo: git clone https://github.com/microsoft/unadversarial.git

  2. Install dependencies:

    conda create -n unadv python=3.7
    conda activate unadv
    pip install -r requirements.txt
    

Generating unadversarial examples for CIFAR10

Here we show a quick example how to generate unadversarial examples for CIFAR-10. Similar procedure can be used with ImageNet. The entry point of our code is main.py (see the file for a full description of arguments).

1- Download a pretrained CIFAR10 models, e.g.,

mkdir pretrained-models & 
wget -O pretrained-models/cifar_resnet50.ckpt "https://www.dropbox.com/s/yhpp4yws7sgi6lj/cifar_nat.pt?raw=1"

2- Run the following script

python -m src.main \
      --out-dir OUT_DIR \
      --exp-name demo \
      --dataset cifar \
      --data /tmp \
      --arch resnet50 \
      --model-path pretrained-models/cifar_resnet50.ckpt \
      --patch-size 10 \
      --patch-lr 0.001 \
      --training-mode booster \
      --epochs 30 \
      --adv-train 0

You can see the trained patches images in outdir/demo/save/ as training evolves.

3- Now you can evaluate the pretrained model on a boosted CIFAR10-C dataset (trained patch overlaid on CIFAR-10, then corruptions are added). Simply run

python -m src.evaluate_corruptions \
      --out-dir OUT_DIR \
      --exp-name demo \
      --model-path OUT_DIR/demo/checkpoint.pt.best \
      --args-from-store data,dataset,arch,patch_size

This will evaluate the pretrained model on various corruptions and display the results in the terminal.

4- That's it!

Generating 3D unadversarial textures

The following steps were tested on these configurations:

  • Ubuntu 16.04, 8 x NVIDIA 1080Ti/2080Ti, 2x10-core Intel CPUs (w/ HyperThreading, 40 virtual cores), CUDA 10.2
  • Ubuntu 18.04, 2 x NVIDIA K80, 1x12-core Intel CPU, CUDA 10.2

1- Choose a dataset to use as background images; we used ImageNet in our paper, for which you will need to have ImageNet in PyTorch ImageFolder format somewhere on your machine. If you don't have that, you can just use solid colors as the backgrounds (though the results might not match the paper).

2- Install the requirements: you will need a machine with CUDA 10.2 installed (this process might work with other versions of CUDA but we only tested 10.2), as well as docker, nvidia-docker, and the requirements mentioned earlier in the README.

3- Go to the docker/ folder and run docker build --tag TAG ., changing TAG to your preferred name for your docker instance. This will build a docker instance with all the requirements installed!

4- Open launch.py and edit the IMAGENET_TRAIN and IMAGENET_VAL variables to point to the ImageNet dataset, if it's installed and you want to use it. Either way, change TAG on the last line of the file with whatever you named your docker instance in the last step.

5- Alter the parameters in src/configs/config.json according to your setup; the only things we would recommend altering are num_texcoord_renderers (this should not exceed the number of CPU cores you have available), exp_name (the name of the output folder, which will be created inside OUT_DIR from the previous step), and dataset (if you are using ImageNet, you can leave this be, otherwise change it to solids to use solid colors as the backgrounds).

6- From inside the docker folder, run python launch.py [--with-imagenet] --out-dir OUT_DIR --gpus GPUS from the same folder. The --with-imagenet argument should only be provided if you followed step four. The OUT_DIR argument should point to where you want the resulting models/output saved, and the GPUS argument should be a comma-separated list of GPU IDs that you would like to run the job on.

7- This process should open a new terminal (inside your docker instance). In this terminal, run GPU_MODE=0 bash run_imagenet.sh [bus|warplane|ship|truck|car] /src/configs/config.json /out

8- Your 3D unadversarial texture should now be generating! Output, including example renderings, the texture itself, and the model checkpoint will be saved to $(OUT_DIR)/$(exp_name).

An example texture that you would get for the warplane is

Simulating 3D Unadversarial Objects in AirSim

Coming soon!

Environments, 3D models, along with python API for controlling these objects and running online object recognition inside Microsoft's AirSim high-fidelity simulator.

Maintainers

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

You might also like...
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

A task-agnostic vision-language architecture as a step towards General Purpose Vision
A task-agnostic vision-language architecture as a step towards General Purpose Vision

Towards General Purpose Vision Systems By Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem Overview Welcome to the official code base f

This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

vision-transformer-from-scratch This repository includes several kinds of vision transformers from scratch so that one beginner can understand the the

Source code, datasets and trained models for the paper Learning Advanced Mathematical Computations from Examples (ICLR 2021), by François Charton, Amaury Hayat (ENPC-Rutgers) and Guillaume Lample

Maths from examples - Learning advanced mathematical computations from examples This is the source code and data sets relevant to the paper Learning a

Several simple examples for popular neural network toolkits calling custom CUDA operators.
Several simple examples for popular neural network toolkits calling custom CUDA operators.

Neural Network CUDA Example Several simple examples for neural network toolkits (PyTorch, TensorFlow, etc.) calling custom CUDA operators. We provide

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.
Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

A repo with study material, exercises, examples, etc for Devnet SPAUTO
A repo with study material, exercises, examples, etc for Devnet SPAUTO

MPLS in the SDN Era -- DevNet SPAUTO Get right to the study material: Checkout the Wiki! A lab topology based on MPLS in the SDN era book used for 30

🐸STT integration examples

🐸 STT 0.9.x Examples These are various examples on how to use or integrate 🐸 STT using our packages. It is a good way to just try out 🐸 STT before

transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛
transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

transfer_adv CVPR-2021 AIC-VI: unrestricted Adversarial Attacks on ImageNet CVPR2021 安全AI挑战者计划第六期赛道2:ImageNet无限制对抗攻击 介绍 : 深度神经网络已经在各种视觉识别问题上取得了最先进的性能。

Comments
  • How to use unadversarial patches in the testing stage?

    How to use unadversarial patches in the testing stage?

    Thank you for your amazing work.

    Could you please tell me how to add patches to each sample during the testing stage? I have read the code and it seems like the ground-truth is needed to help each sample add the corresponding category of patches during testing. I wonder if this is fair. Or maybe I made mistake…… image

    Looking forward to your reply! Thanks a lot.

    opened by wqfdewifi 2
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
This repository contains the source code of our work on designing efficient CNNs for computer vision

Efficient networks for Computer Vision This repo contains source code of our work on designing efficient networks for different computer vision tasks:

Sachin Mehta 386 Nov 26, 2022
ColossalAI-Examples - Examples of training models with hybrid parallelism using ColossalAI

ColossalAI-Examples This repository contains examples of training models with Co

HPC-AI Tech 185 Jan 9, 2023
Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering (NAACL 2021)

Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering Abstract In open-domain question answering (QA), retrieve-and-read mec

Clova AI Research 34 Apr 13, 2022
gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks.

gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks. It is built on top of the OpenAI Gym toolkit.

Robin Henry 99 Dec 12, 2022
PHOTONAI is a high level python API for designing and optimizing machine learning pipelines.

PHOTONAI is a high level python API for designing and optimizing machine learning pipelines. We've created a system in which you can easily select and

Medical Machine Learning Lab - University of Münster 57 Nov 12, 2022
A certifiable defense against adversarial examples by training neural networks to be provably robust

DiffAI v3 DiffAI is a system for training neural networks to be provably robust and for proving that they are robust. The system was developed for the

SRI Lab, ETH Zurich 202 Dec 13, 2022
A vision library for performing sliced inference on large images/small objects

SAHI: Slicing Aided Hyper Inference A vision library for performing sliced inference on large images/small objects Overview Object detection and insta

Open Business Software Solutions 2.3k Jan 4, 2023
This repository contains PyTorch code for Robust Vision Transformers.

This repository contains PyTorch code for Robust Vision Transformers.

null 117 Dec 7, 2022
Contains code for the paper "Vision Transformers are Robust Learners".

Vision Transformers are Robust Learners This repository contains the code for the paper Vision Transformers are Robust Learners by Sayak Paul* and Pin

Sayak Paul 103 Jan 5, 2023
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Vision Longformer This project provides the source code for the vision longformer paper. Multi-Scale Vision Longformer: A New Vision Transformer for H

Microsoft 209 Dec 30, 2022