On the Adversarial Robustness of Visual Transformer

Related tags

Deep Learning paper
Overview

On the Adversarial Robustness of Visual Transformer

Code for our paper "On the Adversarial Robustness of Visual Transformers"

Paper link: https://arxiv.org/abs/2103.15670

Evaluation

Install dependencies:

pip install -r requirements.txt

White-box attack test

To test adversarial robustness under white-box attack

python white_box_test.py --data_dir $DATA_DIR --mode foolbox --model vit_small_patch16_224
  • The --mode flag decides the evaluation approach:
    • --mode foolbox applies a single attack (default: LinfPGD) for evalution.
    • --mode auto applies AutoAttack for evaluation.
    • --mode foolbox-filter applies the frequency-based attack for evaluation.
    • --mode evaluate evaluates the clean accuracy.
    • --mode count counts the number of parameters.

Black-box attack test

To test the transferability of adversarial examples generated by different models

python black_box_test.py --data_dir $DATA_DIR

Adversarial training

Go to the training code:

cd training

Install dependencies:

pip install -r requirements.txt

Run:

python train.py --dir {OUTPUT_DIR} --model vit_base_patch16_224_in21k --method {pgd|trades}

You may set `--accum-steps {N}' for gradient accumulation in case that GPU memory is not enough.

Comments
  • datasets preprare

    datasets preprare

    I used the same as t2t_vit github (https://github.com/yitu-opensource/T2T-ViT) data prepare structure on white_box_test.py . and my command is python3 white_box_test.py --data_dir imagenet --mode foolbox --model vit_small_patch16_224 --pretrained True . But I got the awful acc .Below

    sample size is : 1000 clean accuracy: 0.0 % Model vit_small_patch16_224 robust accuracy for LinfPGD perturbations with Step 40, Linf norm ≤ 0.001 : 0.0 % Step 40, Linf norm ≤ 0.003 : 0.0 % Step 40, Linf norm ≤ 0.005 : 0.0 % Step 40, Linf norm ≤ 0.008 : 0.0 % Step 40, Linf norm ≤ 0.01 : 0.0 % Step 40, Linf norm ≤ 0.1 : 0.0 %

    Could u tell me why I got ? and what should I do? Tks u. :))

    opened by JIA-HONG-CHU 3
  • The use of pretrain weights in adversarial training

    The use of pretrain weights in adversarial training

    Hi @RulinShao , thanks a lot, for your impressive work. I'm trying to reproduce some of your experiments, I notice that in this repo you use timm for the pretrained version of vit-16. When I try to do the training process like you did in train.py, it seems that I fail with loading the pretrained weights of vit_base_patch16_224_in21k. The details of error : RuntimeError: Expected hasRecord("version") to be true, but got false.

    env: torch==1.9.0 torchvision==0.10.0 running on Colab

    I also tried older versions of torch, they all failed with loading pretrain weights with other error information like" Only one file(not dir) is allowed in the zipfile", it seems that the problem is about the compressed weights in zip format used in timm

    Could you please tell me the environment you use when implementing this work? thanks a lot!

    opened by caposerenity 3
  • white box attack for auto-attack

    white box attack for auto-attack

    hello , I want to ask what is the auto-attack setting in the white box, because the result of running the default value in vit_small_patch16_224 is different from the result of your paper.

    my command, python3 white_box_test.py --data_dir imagenet --mode auto --model vit_small_patch16_224

    my result, sample size is : 1000 clean accuracy: 73.8 % Model vit_small_patch16_224 robust accuracy for AutoAttack perturbations with Linf norm ≤ 0.001 : 28.4 % Linf norm ≤ 0.003 : 1.2 % Linf norm ≤ 0.005 : 0.0 % Linf norm ≤ 0.008 : 0.0 % Linf norm ≤ 0.01 : 0.0 % Linf norm ≤ 0.1 : 0.0 %

    tks for reply.

    opened by JIA-HONG-CHU 2
  • Adversarial Training

    Adversarial Training

    Hi @RulinShao, it's a quite solid work. But I have some questions about adversarial training. (1) In your paper, you used ViT-B/4 but here I only see vit_base_patch2. So, your adv-training model is still ViT-B/4, right? (2) In your code, I feel little bit confused about eval. What's the purpose for it? You mean model.eval()? If so, then model.train() is applied. Can you give an explanation for it? https://github.com/RulinShao/on-the-adversarial-robustness-of-visual-transformer/blob/ff8e3b7ffadd5f6776205d1b1e2b705a75bebb57/training/train.py#L45-L46 (3) In the paper, you downsampled the weights for patch embeddings and what're the codes for such operation? I only see the modification of patch_size . https://github.com/RulinShao/on-the-adversarial-robustness-of-visual-transformer/blob/ff8e3b7ffadd5f6776205d1b1e2b705a75bebb57/training/timm_vit/vit.py#L283-L286 (4) CIFAR-10 data resolution is 32 and the model in your command line is vit_base_patch16_224_in21k . Such large patch size for CIFAR-10 could highly do harm to acc%. Did you use such model to do the experiments? I only see the results of ViT-B/4 in the paper. BTW, you always keep datasize in 32 instead of upsampling 224, right? Did you try to upsample CIFAR-10 to 224 and use patch size 16 for adversarial training?

    Thanks for your help!

    opened by Yeez-lee 2
  • timm data loader for the ViT models

    timm data loader for the ViT models

    Thanks a lot for your interesting work!

    If I see it correctly, the script white_box_test.py uses data loader defined in the function white_box_test.get_val_loader for the ViT models obtained from timm. It uses default ImageNet values mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225].

    It seems that timm documentation suggests a different set of transforms for data preprocessing: https://rwightman.github.io/pytorch-image-models/models/vision-transformer/

    The difference seems to be that it rescales the images to the size of 248 and uses mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5] for normalization.

    When I use their input transform I obtain larger clean accuracy for transformer models than the one provided in the paper. It looks similar to the one provided in the paperswithcode page: https://paperswithcode.com/lib/timm/vision-transformer

    Perhaps adding the transforms from the timm package to the code could help in evaluation.

    Thanks!

    opened by yatsurama 0
Owner
Rulin Shao
Rulin Shao
GitHub
Pytorch implementation for "Adversarial Robustness under Long-Tailed Distribution" (CVPR 2021 Oral)

Adversarial Long-Tail This repository contains the PyTorch implementation of the paper: Adversarial Robustness under Long-Tailed Distribution, CVPR 20

Tong WU 89 Dec 15, 2022
Multitask Learning Strengthens Adversarial Robustness

Multitask Learning Strengthens Adversarial Robustness

Columbia University 15 Jun 10, 2022
Implementations of orthogonal and semi-orthogonal convolutions in the Fourier domain with applications to adversarial robustness

Orthogonalizing Convolutional Layers with the Cayley Transform This repository contains implementations and source code to reproduce experiments for t

CMU Locus Lab 36 Dec 30, 2022
Improving adversarial robustness by a coupling rejection strategy

Adversarial Training with Rectified Rejection The code for the paper Adversarial Training with Rectified Rejection. Environment settings and libraries

Tianyu Pang 29 Jan 6, 2023
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Adversarial Robustness Toolbox (ART) is a Python library for Machine Learning Security. ART provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference. ART supports all popular machine learning frameworks (TensorFlow, Keras, PyTorch, MXNet, scikit-learn, XGBoost, LightGBM, CatBoost, GPy, etc.), all data types (images, tables, audio, video, etc.) and machine learning tasks (classification, object detection, speech recognition, generation, certification, etc.).

null 3.4k Jan 4, 2023
Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness

Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness Code for Paper "Imbalanced Gradients: A Subtle Cause of Overestimated Adv

Hanxun Huang 11 Nov 30, 2022
Code repository accompanying the paper "On Adversarial Robustness: A Neural Architecture Search perspective"

On Adversarial Robustness: A Neural Architecture Search perspective Preparation: Clone the repository: https://github.com/tdchaitanya/nas-robustness.g

Chaitanya Devaguptapu 4 Nov 10, 2022
Hierarchical-Bayesian-Defense - Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference (Openreview)

Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical V

LBK 20 Dec 2, 2022
Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

Alex Pashevich 62 Dec 24, 2022
VSR-Transformer - This paper proposes a new Transformer for video super-resolution (called VSR-Transformer).

VSR-Transformer By Jiezhang Cao, Yawei Li, Kai Zhang, Luc Van Gool This paper proposes a new Transformer for video super-resolution (called VSR-Transf

Jiezhang Cao 225 Nov 13, 2022
improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

null 310 Dec 28, 2022
Visual Adversarial Imitation Learning using Variational Models (VMAIL)

Visual Adversarial Imitation Learning using Variational Models (VMAIL) This is the official implementation of the NeurIPS 2021 paper. Project website

null 14 Nov 18, 2022
SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

SAAVN SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,IC

YinfengYu 10 Aug 30, 2022
Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

Sukrut Rao 32 Dec 13, 2022
Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

null 28 Dec 25, 2022
transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

transfer_adv CVPR-2021 AIC-VI: unrestricted Adversarial Attacks on ImageNet CVPR2021 安全AI挑战者计划第六期赛道2：ImageNet无限制对抗攻击 介绍 ： 深度神经网络已经在各种视觉识别问题上取得了最先进的性能。

null 25 Dec 8, 2022
Adversarial-Information-Bottleneck - Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck (NeurIPS21)

NeurIPS 2021 Title: Distilling Robust and Non-Robust Features in Adversarial Exa

LBK 35 Dec 26, 2022
Causal-Adversarial-Instruments - PyTorch Implementation for Developing Library of Investigating Adversarial Examples on A Causal View by Instruments

Causal-Adversarial-Instruments This is a PyTorch Implementation code for develop

LBK 26 Dec 28, 2022
Super-Fast-Adversarial-Training - A PyTorch Implementation code for developing super fast adversarial training

Super-Fast-Adversarial-Training This is a PyTorch Implementation code for develo

LBK 26 Dec 2, 2022