[NeurIPS 2021]: Are Transformers More Robust Than CNNs? (Pytorch implementation & checkpoints)

Related tags

Deep Learning transformer robustness
Overview

Are Transformers More Robust Than CNNs?

Pytorch implementation for NeurIPS 2021 Paper: Are Transformers More Robust Than CNNs?

Our implementation is based on DeiT.

Introduction

Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations.

With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs.

Pretrained models

We provide both pretrained vanilla models and adversarially trained models.

Vanilla Training

Main Results

Pretrained Model ImageNet ImageNet-A ImageNet-C Stylized-ImageNet
Res50-Ori download link 76.9 3.2 57.9 8.3
Res50-Align download link 76.3 4.5 55.6 8.2
Res50-Best download link 75.7 6.3 52.3 10.8
DeiT-Small download link 76.8 12.2 48.0 13.0

Model Size

ResNets:

  • ResNets fully aligned (with DeiT's training recipe) model, denoted as res*:
Model Size Pretrained Model ImageNet ImageNet-A ImageNet-C Stylized-ImageNet
Res18* 11.69M download link 67.83 1.92 64.14 7.92
Res50* 25.56M download link 76.28 4.53 55.62 8.17
Res101* 44.55M download link 77.97 8.84 49.19 11.60
  • ResNets best model (for Out-of-Distribution (OOD) generalization), denoted as res-best:
Model Size Pretrained Model ImageNet ImageNet-A ImageNet-C Stylized-ImageNet
Res18-best 11.69M download link 66.81 2.03 62.65 9.45
Res50-best 25.56M download link 75.74 6.32 52.25 10.77
Res101-best 44.55M download link 77.83 11.49 47.35 13.28

DeiTs:

Model Size Pretrained Model ImageNet ImageNet-A ImageNet-C Stylized-ImageNet
DeiT-Mini 9.98M download link 72.89 8.19 54.68 9.88
DeiT-Small 22.05M download link 76.82 12.21 47.99 12.98

Model Distillation

Architecture Pretrained Model ImageNet ImageNet-A ImageNet-C Stylized-ImageNet
Teacher DeiT-Small download link 76.8 12.2 48.0 13.0
Student Res50*-Distill download link 76.7 5.2 54.2 9.8
Teacher Res50* download link 76.3 4.5 55.6 8.2
Student DeiT-S-Distill download link 76.2 10.9 49.3 11.9

Adversarial Training

Pretrained Model Clean Acc PGD-100 Auto Attack
Res50-ReLU download link 66.77 32.26 26.41
Res50-GELU download link 67.38 40.27 35.51
DeiT-Small download link 66.50 40.32 35.50

Vanilla Training

Data preparation

Download and extract ImageNet train and val images from http://image-net.org/. The directory structure is the standard layout for the torchvision, and the training and validation data is expected to be in the train folder and val folder respectively:

/path/to/imagenet/
  train/
    class1/
      img1.jpeg
    class2/
      img2.jpeg
  val/
    class1/
      img3.jpeg
    class/2
      img4.jpeg

Environment

Install dependencies:

pip3 install -r requirements.txt

Training Scripts

To train a ResNet model on ImageNet run:

bash script/res.sh

To train a DeiT model on ImageNet run:

bash script/deit.sh

Generalization to Out-of-Distribution Sample

Data Preparation

Download and extract ImageNet-A, ImageNet-C, Stylized-ImageNet val images:

/path/to/datasets/
  val/
    class1/
      img1.jpeg
    class/2
      img2.jpeg

Evaluation Scripts

To evaluate pre-trained models, run:

bash script/generation_to_ood.sh

It is worth noting that for ImageNet-C evaluation, the error rate is calculated based on the Noise, Blur, Weather and Digital categories.

Adversarial Training

To perform adversarial training on ResNet run:

bash script/advres.sh

To do adversarial training on DeiT run:

bash scripts/advdeit.sh

Robustness to Adversarial Example

PGD Attack Evaluation

To evaluate the pre-trained models, run:

bash script/eval_advtraining.sh

AutoAttack Evaluation

./autoattack contains the AutoAttack public package, with a little modification to best support ImageNet evaluation.

cd autoattack/
bash autoattack.sh

Patch Attack Evaluation

Please refer to PatchAttack

Citation

If you use our code, models or wish to refer to our results, please use the following BibTex entry:

@inproceedings{bai2021transformers,
  title     = {Are Transformers More Robust Than CNNs?},
  author    = {Bai, Yutong and Mei, Jieru and Yuille, Alan and Xie, Cihang},
  booktitle = {Thirty-Fifth Conference on Neural Information Processing Systems},
  year      = {2021},
}
Comments
  • The adversarial training script is showing strange trend

    The adversarial training script is showing strange trend

    Hi The adversarial training script is showing strage trend, after certain epochs top-1 accuracy has fallen to 1.6% from around 21%. Is it normal?

    I used the script for adv training as: python -m torch.distributed.launch --nproc_per_node=4 --master_port=5672 --use_env main_adv_deit.py --model deit_small_patch16_224_adv --batch-size 128 --data-path /datasets/imagenet-ilsvrc2012 --attack-iter 1 --attack-epsilon 4 --attack-step-size 4 --epoch 100 --reprob 0 --no-repeated-aug --sing singln --drop 0 --drop-path 0 --start_epoch 0 --warmup-epochs 10 --cutmix 0 --output_dir save/deit_adv/deit_small_patch16_224

    Here is the training log (till 40 epochs): {"train_lr": 1.0000000000000031e-06, "train_loss": 6.885785259502969, "test_0_loss": 6.7725973782139715, "test_0_acc1": 0.806, "test_0_acc5": 2.804, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 0, "n_parameters": 22050664} {"train_lr": 1.0000000000000031e-06, "train_loss": 6.885785259502969, "test_0_loss": 6.7725973782139715, "test_0_acc1": 0.806, "test_0_acc5": 2.804, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 0, "n_parameters": 22050664} {"train_lr": 1.0000000000000031e-06, "train_loss": 6.846427675869634, "test_0_loss": 6.689390176393554, "test_0_acc1": 1.192, "test_0_acc5": 4.378, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 1, "n_parameters": 22050664} {"train_lr": 0.00020090000000000288, "train_loss": 6.701197089479981, "test_0_loss": 5.865309043488896, "test_0_acc1": 5.43, "test_0_acc5": 14.672, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 2, "n_parameters": 22050664} {"train_lr": 0.00040079999999998546, "train_loss": 6.543532955179588, "test_0_loss": 5.340847122768371, "test_0_acc1": 9.812, "test_0_acc5": 23.782, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 3, "n_parameters": 22050664} {"train_lr": 0.0006006999999999715, "train_loss": 6.4769038225916455, "test_0_loss": 5.03732673006796, "test_0_acc1": 13.248, "test_0_acc5": 29.602, "test_5_loss": 6.844994894602477, "test_5_acc1": 0.55, "test_5_acc5": 1.958, "epoch": 4, "n_parameters": 22050664} {"train_lr": 0.0008006000000000287, "train_loss": 6.315360357340196, "test_0_loss": 5.300121459301969, "test_0_acc1": 10.944, "test_0_acc5": 25.546, "test_5_loss": 6.55244832365313, "test_5_acc1": 2.756, "test_5_acc5": 7.6525, "epoch": 5, "n_parameters": 22050664} {"train_lr": 0.0010004999999999689, "train_loss": 6.190600837687318, "test_0_loss": 4.9149362563476755, "test_0_acc1": 14.35, "test_0_acc5": 31.418, "test_5_loss": 6.55244832365313, "test_5_acc1": 2.756, "test_5_acc5": 7.6525, "epoch": 6, "n_parameters": 22050664} {"train_lr": 0.0012004000000000647, "train_loss": 6.088374964529566, "test_0_loss": 5.50498779843575, "test_0_acc1": 10.254, "test_0_acc5": 24.242, "test_5_loss": 6.55244832365313, "test_5_acc1": 2.756, "test_5_acc5": 7.6525, "epoch": 7, "n_parameters": 22050664} {"train_lr": 0.0014002999999999238, "train_loss": 6.08913704293142, "test_0_loss": 4.774700349977363, "test_0_acc1": 14.72, "test_0_acc5": 32.384, "test_5_loss": 6.55244832365313, "test_5_acc1": 2.756, "test_5_acc5": 7.6525, "epoch": 8, "n_parameters": 22050664} {"train_lr": 0.0016001999999999618, "train_loss": 6.150533516344121, "test_0_loss": 5.227625224198276, "test_0_acc1": 10.67, "test_0_acc5": 25.058, "test_5_loss": 6.55244832365313, "test_5_acc1": 2.756, "test_5_acc5": 7.6525, "epoch": 9, "n_parameters": 22050664} {"train_lr": 0.0018001000000000126, "train_loss": 6.101692359891536, "test_0_loss": 5.141786843786161, "test_0_acc1": 11.414, "test_0_acc5": 26.346, "test_5_loss": 6.756372647642403, "test_5_acc1": 2.309, "test_5_acc5": 6.5675, "epoch": 10, "n_parameters": 22050664} {"train_lr": 0.001951301233713633, "train_loss": 6.093319233182332, "test_0_loss": 4.774902591320924, "test_0_acc1": 14.368, "test_0_acc5": 31.786, "test_5_loss": 6.756372647642403, "test_5_acc1": 2.309, "test_5_acc5": 6.5675, "epoch": 11, "n_parameters": 22050664} {"train_lr": 0.001941176365109525, "train_loss": 6.128870297345421, "test_0_loss": 5.251185640492503, "test_0_acc1": 11.492, "test_0_acc5": 26.726, "test_5_loss": 6.756372647642403, "test_5_acc1": 2.309, "test_5_acc5": 6.5675, "epoch": 12, "n_parameters": 22050664} {"train_lr": 0.0019301276034588222, "train_loss": 6.053121808859752, "test_0_loss": 4.758080562108309, "test_0_acc1": 16.252, "test_0_acc5": 34.882, "test_5_loss": 6.756372647642403, "test_5_acc1": 2.309, "test_5_acc5": 6.5675, "epoch": 13, "n_parameters": 22050664} {"train_lr": 0.0019181658525555538, "train_loss": 6.0439764577136055, "test_0_loss": 4.586510399862962, "test_0_acc1": 16.69, "test_0_acc5": 35.526, "test_5_loss": 6.756372647642403, "test_5_acc1": 2.309, "test_5_acc5": 6.5675, "epoch": 14, "n_parameters": 22050664} {"train_lr": 0.0019053029172036828, "train_loss": 5.91496213320062, "test_0_loss": 4.488940908904268, "test_0_acc1": 17.398, "test_0_acc5": 36.698, "test_5_loss": 7.4814025707452325, "test_5_acc1": 1.2555, "test_5_acc5": 4.2435, "epoch": 15, "n_parameters": 22050664} {"train_lr": 0.0018915514915675221, "train_loss": 6.002524321551898, "test_0_loss": 4.450921233922186, "test_0_acc1": 17.934, "test_0_acc5": 37.114, "test_5_loss": 7.4814025707452325, "test_5_acc1": 1.2555, "test_5_acc5": 4.2435, "epoch": 16, "n_parameters": 22050664} {"train_lr": 0.0018769251466436458, "train_loss": 5.878266204508851, "test_0_loss": 4.308091710831062, "test_0_acc1": 20.404, "test_0_acc5": 41.2, "test_5_loss": 7.4814025707452325, "test_5_acc1": 1.2555, "test_5_acc5": 4.2435, "epoch": 17, "n_parameters": 22050664} {"train_lr": 0.0018614383168689135, "train_loss": 5.789360093222343, "test_0_loss": 4.410817133793065, "test_0_acc1": 18.154, "test_0_acc5": 38.082, "test_5_loss": 7.4814025707452325, "test_5_acc1": 1.2555, "test_5_acc5": 4.2435, "epoch": 18, "n_parameters": 22050664} {"train_lr": 0.0018451062858745686, "train_loss": 5.750880390286541, "test_0_loss": 4.467262921391278, "test_0_acc1": 19.266, "test_0_acc5": 39.462, "test_5_loss": 7.4814025707452325, "test_5_acc1": 1.2555, "test_5_acc5": 4.2435, "epoch": 19, "n_parameters": 22050664} {"train_lr": 0.0018279451714032378, "train_loss": 5.764791792602562, "test_0_loss": 4.67896575738586, "test_0_acc1": 17.392, "test_0_acc5": 37.15, "test_5_loss": 7.557907587735987, "test_5_acc1": 0.9905, "test_5_acc5": 3.1575, "epoch": 20, "n_parameters": 22050664} {"train_lr": 0.0018099719094030393, "train_loss": 5.759131700348416, "test_0_loss": 4.419680974762636, "test_0_acc1": 19.966, "test_0_acc5": 40.798, "test_5_loss": 7.557907587735987, "test_5_acc1": 0.9905, "test_5_acc5": 3.1575, "epoch": 21, "n_parameters": 22050664} {"train_lr": 0.0017912042373137494, "train_loss": 5.710006896111605, "test_0_loss": 4.2751427415236405, "test_0_acc1": 20.356, "test_0_acc5": 41.114, "test_5_loss": 7.557907587735987, "test_5_acc1": 0.9905, "test_5_acc5": 3.1575, "epoch": 22, "n_parameters": 22050664} {"train_lr": 0.0017716606765619972, "train_loss": 5.68051082098322, "test_0_loss": 4.154385426833091, "test_0_acc1": 21.638, "test_0_acc5": 43.102, "test_5_loss": 7.557907587735987, "test_5_acc1": 0.9905, "test_5_acc5": 3.1575, "epoch": 23, "n_parameters": 22050664} {"train_lr": 0.0017513605142823508, "train_loss": 5.693617649811158, "test_0_loss": 4.25816687512535, "test_0_acc1": 20.994, "test_0_acc5": 41.96, "test_5_loss": 7.557907587735987, "test_5_acc1": 0.9905, "test_5_acc5": 3.1575, "epoch": 24, "n_parameters": 22050664} {"train_lr": 0.0017303237842843694, "train_loss": 5.6821105527839695, "test_0_loss": 4.421267043301026, "test_0_acc1": 19.116, "test_0_acc5": 39.094, "test_5_loss": 8.897946262237587, "test_5_acc1": 0.514, "test_5_acc5": 1.892, "epoch": 25, "n_parameters": 22050664} {"train_lr": 0.001708571247280513, "train_loss": 5.69677297047955, "test_0_loss": 4.398700253595852, "test_0_acc1": 19.178, "test_0_acc5": 39.536, "test_5_loss": 8.897946262237587, "test_5_acc1": 0.514, "test_5_acc5": 1.892, "epoch": 26, "n_parameters": 22050664} {"train_lr": 0.0016861243703990647, "train_loss": 5.740358965097666, "test_0_loss": 4.446112109237348, "test_0_acc1": 19.972, "test_0_acc5": 40.84, "test_5_loss": 8.897946262237587, "test_5_acc1": 0.514, "test_5_acc5": 1.892, "epoch": 27, "n_parameters": 22050664} {"train_lr": 0.0016630053059970855, "train_loss": 5.712303198760838, "test_0_loss": 4.1932648324234245, "test_0_acc1": 21.566, "test_0_acc5": 42.98, "test_5_loss": 8.897946262237587, "test_5_acc1": 0.514, "test_5_acc5": 1.892, "epoch": 28, "n_parameters": 22050664} {"train_lr": 0.0016392368698000565, "train_loss": 5.74558376472631, "test_0_loss": 4.124165606513972, "test_0_acc1": 21.932, "test_0_acc5": 43.39, "test_5_loss": 8.897946262237587, "test_5_acc1": 0.514, "test_5_acc5": 1.892, "epoch": 29, "n_parameters": 22050664} {"train_lr": 0.0016148425183847566, "train_loss": 5.3731044158518175, "test_0_loss": 4.680003530995935, "test_0_acc1": 15.374, "test_0_acc5": 33.588, "test_5_loss": 11.426024395688863, "test_5_acc1": 0.1075, "test_5_acc5": 0.45, "epoch": 30, "n_parameters": 22050664} {"train_lr": 0.0015898463260310706, "train_loss": 4.259690835869474, "test_0_loss": 5.981102620495181, "test_0_acc1": 5.786, "test_0_acc5": 14.55, "test_5_loss": 11.426024395688863, "test_5_acc1": 0.1075, "test_5_acc5": 0.45, "epoch": 31, "n_parameters": 22050664} {"train_lr": 0.0015642729609628443, "train_loss": 4.075305948785836, "test_0_loss": 5.933592574686403, "test_0_acc1": 4.598, "test_0_acc5": 13.066, "test_5_loss": 11.426024395688863, "test_5_acc1": 0.1075, "test_5_acc5": 0.45, "epoch": 32, "n_parameters": 22050664} {"train_lr": 0.001538147661004018, "train_loss": 4.167220209940351, "test_0_loss": 6.295307500501207, "test_0_acc1": 3.228, "test_0_acc5": 9.566, "test_5_loss": 11.426024395688863, "test_5_acc1": 0.1075, "test_5_acc5": 0.45, "epoch": 33, "n_parameters": 22050664} {"train_lr": 0.001511496208671658, "train_loss": 4.134730825523774, "test_0_loss": 5.972504679850104, "test_0_acc1": 3.806, "test_0_acc5": 11.758, "test_5_loss": 11.426024395688863, "test_5_acc1": 0.1075, "test_5_acc5": 0.45, "epoch": 34, "n_parameters": 22050664} {"train_lr": 0.0014843449057311518, "train_loss": 4.365966309007885, "test_0_loss": 6.5958606600380065, "test_0_acc1": 2.156, "test_0_acc5": 7.3, "test_5_loss": 12.839466273136347, "test_5_acc1": 0.002, "test_5_acc5": 0.0035, "epoch": 35, "n_parameters": 22050664} {"train_lr": 0.00145672054724078, "train_loss": 4.49492947772729, "test_0_loss": 6.905164708865429, "test_0_acc1": 1.588, "test_0_acc5": 5.264, "test_5_loss": 12.839466273136347, "test_5_acc1": 0.002, "test_5_acc5": 0.0035, "epoch": 36, "n_parameters": 22050664} {"train_lr": 0.0014286503951072877, "train_loss": 4.562651729769558, "test_0_loss": 6.958603466617245, "test_0_acc1": 1.594, "test_0_acc5": 5.226, "test_5_loss": 12.839466273136347, "test_5_acc1": 0.002, "test_5_acc5": 0.0035, "epoch": 37, "n_parameters": 22050664} {"train_lr": 0.0014001621511816529, "train_loss": 4.620032903101804, "test_0_loss": 6.883705623624269, "test_0_acc1": 1.946, "test_0_acc5": 5.582, "test_5_loss": 12.839466273136347, "test_5_acc1": 0.002, "test_5_acc5": 0.0035, "epoch": 38, "n_parameters": 22050664} {"train_lr": 0.0013712839299212382, "train_loss": 4.635755813831715, "test_0_loss": 7.244386745887312, "test_0_acc1": 0.964, "test_0_acc5": 3.976, "test_5_loss": 12.839466273136347, "test_5_acc1": 0.002, "test_5_acc5": 0.0035, "epoch": 39, "n_parameters": 22050664} {"train_lr": 0.0013420442306441068, "train_loss": 4.83734727265547, "test_0_loss": 7.3013705145603405, "test_0_acc1": 1.686, "test_0_acc5": 4.768, "test_5_loss": 14.802937962195847, "test_5_acc1": 0.0, "test_5_acc5": 0.0, "epoch": 40, "n_parameters": 22050664}

    opened by ksouvik52 8
  • Is epsilon 4/255 or 8/255 in adversarial training?

    Is epsilon 4/255 or 8/255 in adversarial training?

    Hi, Thanks for this excellent work! You stated in your paper "with the constrain that maximum per-pixel change epsilon = 4/255". However, in this implementation, the argument attack-epsilon (4) is multiplied by an Image Scale which is 2/255, and the real epsilon should be 8/255 instead of 4/255?

    opened by ShihuaHuang95 5
  • About the output of autoattack. And the program will be killed by themself.

    About the output of autoattack. And the program will be killed by themself.

    When I use your code of autoatteck, the output I get is very different, and the program will be killed by them self. After output the APGD-CE accuracy, the terminal output Killed. the part output of mine :

    apgd-ce - 1/147 - 256 out of 256 successfully perturbed
0.74022
apgd-ce - 2/147 - 256 out of 256 successfully perturbed
0.7351
apgd-ce - 3/147 - 256 out of 256 successfully perturbed
0.72998
apgd-ce - 4/147 - 256 out of 256 successfully perturbed
0.72486
apgd-ce - 5/147 - 256 out of 256 successfully perturbed
0.71974
apgd-ce - 6/147 - 256 out of 256 successfully perturbed
0.71462
apgd-ce - 7/147 - 256 out of 256 successfully perturbed
0.7095
apgd-ce - 8/147 - 256 out of 256 successfully perturbed
0.70438
apgd-ce - 9/147 - 256 out of 256 successfully perturbed
0.69926
apgd-ce - 10/147 - 256 out of 256 successfully perturbed
0.69414
apgd-ce - 11/147 - 256 out of 256 successfully perturbed
0.68902
apgd-ce - 12/147 - 256 out of 256 successfully perturbed
0.6839
apgd-ce - 13/147 - 256 out of 256 successfully perturbed
0.67878
apgd-ce - 14/147 - 256 out of 256 successfully perturbed
0.67366
apgd-ce - 15/147 - 256 out of 256 successfully perturbed

    the follow is the part output of yours(autoattack/log)

    apgd-ce - 1/65 - 193 out of 512 successfully perturbed
0.65892
apgd-ce - 2/65 - 116 out of 512 successfully perturbed
0.65402
apgd-ce - 3/65 - 245 out of 512 successfully perturbed
0.64836
apgd-ce - 4/65 - 283 out of 512 successfully perturbed
0.64416
apgd-ce - 5/65 - 210 out of 512 successfully perturbed
0.6403
apgd-ce - 6/65 - 193 out of 512 successfully perturbed
0.63828
apgd-ce - 7/65 - 101 out of 512 successfully perturbed
0.6342
apgd-ce - 8/65 - 204 out of 512 successfully perturbed
0.62892
apgd-ce - 9/65 - 264 out of 512 successfully perturbed
0.62626

    Thanks for your reply

    opened by 1104662797 3
  • Unknown model (deit_small_patch16_224)

    Unknown model (deit_small_patch16_224)

    Cannot create DeiT Small model.

    I have timm=0.3.2 installed. I try to evaluate DeiT Small on ImageNet and get the following error.

    [xxx@xxx vit-vs-cnn]$ CUDA_VISIBLE_DEVICES=0 python imagenet_robust.py -a 'deit_small_patch16_224' --test-batch 256  --data data_path  --evaluate --imagenet-a --ckpt ckpt/deitsmall.pth --sing singln
Traceback (most recent call last):
  File "/home/research/domain_sim/vit-vs-cnn/imagenet_robust.py", line 394, in <module>
    main()
  File "/home/research/domain_sim/vit-vs-cnn/imagenet_robust.py", line 233, in main
    model = create_model(
  File "/home/anaconda3/envs/domain2/lib/python3.9/site-packages/timm/models/factory.py", line 59, in create_model
    raise RuntimeError('Unknown model (%s)' % model_name)
RuntimeError: Unknown model (deit_small_patch16_224)
    opened by natanielruiz 2
  • Pre-trained Model link not working- Adversarial Training DeiT-small

    Pre-trained Model link not working- Adversarial Training DeiT-small

    Thank you for sharing your awesome work.

    I've tried to download the pre-trained model of DeiT-small (Adversarial Training).

    The current link: https://drive.google.com/file/d/1U5XmAUQkSlw5Q1ZhsVriBEOQEk-bPfFU/view?usp=sharing does not have a pth file so I'm a little bit confused.

    It would be great if you update the link.

    Thank you

    opened by jongwook-Han 1
  • Adversarial training is not working

    Adversarial training is not working

    Hi, as you suggested in the earlier thread, we tried with the exact same settings and argument that you provided to do train on deit tiny adversarially. However, it is still not working, only 18% accuracy after 52 epochs.

    {"train_lr": 9.999999999999953e-07, "train_loss": 6.91152723201459, "test_0_loss": 6.872483930142354, "test_0_acc1": 0.26, "test_0_acc5": 1.12, "test_5_loss": 6.9177000566849856, "test_5_acc1": 0.1805, "test_5_acc5": 0.6765, "epoch": 0, "n_parameters": 5717416} {"train_lr": 9.999999999999953e-07, "train_loss": 6.900039605433039, "test_0_loss": 6.850005263940539, "test_0_acc1": 0.362, "test_0_acc5": 1.578, "test_5_loss": 6.9177000566849856, "test_5_acc1": 0.1805, "test_5_acc5": 0.6765, "epoch": 1, "n_parameters": 5717416} {"train_lr": 0.00040089999999999305, "train_loss": 6.754413659242894, "test_0_loss": 6.145018349987379, "test_0_acc1": 3.726, "test_0_acc5": 10.93, "test_5_loss": 6.9177000566849856, "test_5_acc1": 0.1805, "test_5_acc5": 0.6765, "epoch": 2, "n_parameters": 5717416} {"train_lr": 0.0008008000000000196, "train_loss": 6.625597798519379, "test_0_loss": 5.638628653967449, "test_0_acc1": 7.224, "test_0_acc5": 18.596, "test_5_loss": 6.9177000566849856, "test_5_acc1": 0.1805, "test_5_acc5": 0.6765, "epoch": 3, "n_parameters": 5717416} {"train_lr": 0.001200700000000036, "train_loss": 6.560625480233337, "test_0_loss": 5.315861636678607, "test_0_acc1": 10.064, "test_0_acc5": 23.916, "test_5_loss": 6.9177000566849856, "test_5_acc1": 0.1805, "test_5_acc5": 0.6765, "epoch": 4, "n_parameters": 5717416} {"train_lr": 0.0016005999999999657, "train_loss": 6.54946454628099, "test_0_loss": 5.134192888048774, "test_0_acc1": 11.984, "test_0_acc5": 27.634, "test_5_loss": 5.716198673022533, "test_5_acc1": 6.383, "test_5_acc5": 16.22425, "epoch": 5, "n_parameters": 5717416} {"train_lr": 0.0020004999999999914, "train_loss": 6.542691466810225, "test_0_loss": 5.129832990186304, "test_0_acc1": 12.648, "test_0_acc5": 28.482, "test_5_loss": 5.716198673022533, "test_5_acc1": 6.383, "test_5_acc5": 16.22425, "epoch": 6, "n_parameters": 5717416} {"train_lr": 0.002400399999999976, "train_loss": 6.536189370113406, "test_0_loss": 5.018598655821495, "test_0_acc1": 13.942, "test_0_acc5": 31.118, "test_5_loss": 5.716198673022533, "test_5_acc1": 6.383, "test_5_acc5": 16.22425, "epoch": 7, "n_parameters": 5717416} {"train_lr": 0.002800300000000059, "train_loss": 6.503189501430777, "test_0_loss": 4.9134670785048495, "test_0_acc1": 15.256, "test_0_acc5": 33.352, "test_5_loss": 5.716198673022533, "test_5_acc1": 6.383, "test_5_acc5": 16.22425, "epoch": 8, "n_parameters": 5717416} {"train_lr": 0.0032002000000000983, "train_loss": 6.5094980549373975, "test_0_loss": 4.851811613246408, "test_0_acc1": 15.578, "test_0_acc5": 34.102, "test_5_loss": 5.716198673022533, "test_5_acc1": 6.383, "test_5_acc5": 16.22425, "epoch": 9, "n_parameters": 5717416} {"train_lr": 0.0036000999999999585, "train_loss": 6.511975479259384, "test_0_loss": 5.143745462633598, "test_0_acc1": 13.898, "test_0_acc5": 30.834, "test_5_loss": 5.7486809707954, "test_5_acc1": 7.313, "test_5_acc5": 17.87725, "epoch": 10, "n_parameters": 5717416} {"train_lr": 0.0039023577500088853, "train_loss": 6.418486285981515, "test_0_loss": 5.218988336208, "test_0_acc1": 12.054, "test_0_acc5": 27.72, "test_5_loss": 5.7486809707954, "test_5_acc1": 7.313, "test_5_acc5": 17.87725, "epoch": 11, "n_parameters": 5717416} {"train_lr": 0.003882057134063648, "train_loss": 6.32594983450038, "test_0_loss": 5.0386335921455325, "test_0_acc1": 14.276, "test_0_acc5": 31.454, "test_5_loss": 5.7486809707954, "test_5_acc1": 7.313, "test_5_acc5": 17.87725, "epoch": 12, "n_parameters": 5717416} {"train_lr": 0.0038599040893470705, "train_loss": 6.353343641300567, "test_0_loss": 4.799374374913163, "test_0_acc1": 16.064, "test_0_acc5": 34.754, "test_5_loss": 5.7486809707954, "test_5_acc1": 7.313, "test_5_acc5": 17.87725, "epoch": 13, "n_parameters": 5717416} {"train_lr": 0.0038359204782395374, "train_loss": 6.200804713199274, "test_0_loss": 5.33436276099656, "test_0_acc1": 11.77, "test_0_acc5": 26.914, "test_5_loss": 5.7486809707954, "test_5_acc1": 7.313, "test_5_acc5": 17.87725, "epoch": 14, "n_parameters": 5717416} {"train_lr": 0.0038101299696696937, "train_loss": 6.372054211956134, "test_0_loss": 5.132588769103652, "test_0_acc1": 14.968, "test_0_acc5": 32.62, "test_5_loss": 5.900749441910766, "test_5_acc1": 6.776, "test_5_acc5": 16.361, "epoch": 15, "n_parameters": 5717416} {"train_lr": 0.0037825580157558377, "train_loss": 6.417677939938699, "test_0_loss": 4.666486162294277, "test_0_acc1": 15.988, "test_0_acc5": 34.454, "test_5_loss": 5.900749441910766, "test_5_acc1": 6.776, "test_5_acc5": 16.361, "epoch": 16, "n_parameters": 5717416} {"train_lr": 0.0037532318266873923, "train_loss": 6.26336412826221, "test_0_loss": 5.38978447795143, "test_0_acc1": 12.402, "test_0_acc5": 28.042, "test_5_loss": 5.900749441910766, "test_5_acc1": 6.776, "test_5_acc5": 16.361, "epoch": 17, "n_parameters": 5717416} {"train_lr": 0.003722180343872929, "train_loss": 6.142464170686537, "test_0_loss": 4.77877281021782, "test_0_acc1": 15.232, "test_0_acc5": 33.184, "test_5_loss": 5.900749441910766, "test_5_acc1": 6.776, "test_5_acc5": 16.361, "epoch": 18, "n_parameters": 5717416} {"train_lr": 0.0036894342113766073, "train_loss": 6.004803928301679, "test_0_loss": 5.506222593730944, "test_0_acc1": 10.49, "test_0_acc5": 24.774, "test_5_loss": 5.900749441910766, "test_5_acc1": 6.776, "test_5_acc5": 16.361, "epoch": 19, "n_parameters": 5717416} {"train_lr": 0.0036550257456777735, "train_loss": 6.135914517725877, "test_0_loss": 5.406659782199775, "test_0_acc1": 11.604, "test_0_acc5": 27.232, "test_5_loss": 6.950556825081355, "test_5_acc1": 2.013, "test_5_acc5": 5.54425, "epoch": 20, "n_parameters": 5717416} {"train_lr": 0.0036189889037779527, "train_loss": 6.093383449254086, "test_0_loss": 5.010170250158621, "test_0_acc1": 13.934, "test_0_acc5": 30.648, "test_5_loss": 6.950556825081355, "test_5_acc1": 2.013, "test_5_acc5": 5.54425, "epoch": 21, "n_parameters": 5717416} {"train_lr": 0.0035813592496895356, "train_loss": 6.059879529152176, "test_0_loss": 5.371730933796497, "test_0_acc1": 10.762, "test_0_acc5": 25.292, "test_5_loss": 6.950556825081355, "test_5_acc1": 2.013, "test_5_acc5": 5.54425, "epoch": 22, "n_parameters": 5717416} {"train_lr": 0.00354217391933767, "train_loss": 6.073066240544323, "test_0_loss": 5.048453500617107, "test_0_acc1": 13.242, "test_0_acc5": 29.126, "test_5_loss": 6.950556825081355, "test_5_acc1": 2.013, "test_5_acc5": 5.54425, "epoch": 23, "n_parameters": 5717416} {"train_lr": 0.0035014715839127445, "train_loss": 6.08733743352951, "test_0_loss": 5.331494154414533, "test_0_acc1": 10.974, "test_0_acc5": 25.072, "test_5_loss": 6.950556825081355, "test_5_acc1": 2.013, "test_5_acc5": 5.54425, "epoch": 24, "n_parameters": 5717416} {"train_lr": 0.003459292411705762, "train_loss": 6.1378554502646505, "test_0_loss": 5.264302222605172, "test_0_acc1": 10.784, "test_0_acc5": 25.304, "test_5_loss": 7.786235228686171, "test_5_acc1": 0.8985, "test_5_acc5": 2.83875, "epoch": 25, "n_parameters": 5717416} {"train_lr": 0.0034156780284671424, "train_loss": 6.1356256470786965, "test_0_loss": 5.568371077798074, "test_0_acc1": 8.102, "test_0_acc5": 19.622, "test_5_loss": 7.786235228686171, "test_5_acc1": 0.8985, "test_5_acc5": 2.83875, "epoch": 26, "n_parameters": 5717416} {"train_lr": 0.003370671476327724, "train_loss": 6.071214107396029, "test_0_loss": 5.589009375886435, "test_0_acc1": 9.484, "test_0_acc5": 22.578, "test_5_loss": 7.786235228686171, "test_5_acc1": 0.8985, "test_5_acc5": 2.83875, "epoch": 27, "n_parameters": 5717416} {"train_lr": 0.00332431717132075, "train_loss": 5.972238908568732, "test_0_loss": 5.707102177010388, "test_0_acc1": 7.978, "test_0_acc5": 19.746, "test_5_loss": 7.786235228686171, "test_5_acc1": 0.8985, "test_5_acc5": 2.83875, "epoch": 28, "n_parameters": 5717416} {"train_lr": 0.0032766608595485736, "train_loss": 6.1025936421539955, "test_0_loss": 5.42314580428013, "test_0_acc1": 11.556, "test_0_acc5": 26.76, "test_5_loss": 7.786235228686171, "test_5_acc1": 0.8985, "test_5_acc5": 2.83875, "epoch": 29, "n_parameters": 5717416} {"train_lr": 0.0032277495720376523, "train_loss": 6.084920492580087, "test_0_loss": 5.923481827123914, "test_0_acc1": 6.536, "test_0_acc5": 16.768, "test_5_loss": 7.121632807848168, "test_5_acc1": 1.17125, "test_5_acc5": 3.44575, "epoch": 30, "n_parameters": 5717416} {"train_lr": 0.003177631578323426, "train_loss": 6.115835655793298, "test_0_loss": 5.460594815469596, "test_0_acc1": 9.702, "test_0_acc5": 23.214, "test_5_loss": 7.121632807848168, "test_5_acc1": 1.17125, "test_5_acc5": 3.44575, "epoch": 31, "n_parameters": 5717416} {"train_lr": 0.003126356338814952, "train_loss": 6.0647800623846475, "test_0_loss": 5.592425890023786, "test_0_acc1": 10.268, "test_0_acc5": 23.954, "test_5_loss": 7.121632807848168, "test_5_acc1": 1.17125, "test_5_acc5": 3.44575, "epoch": 32, "n_parameters": 5717416} {"train_lr": 0.0030739744559831173, "train_loss": 6.054409027099609, "test_0_loss": 5.690968008889499, "test_0_acc1": 9.494, "test_0_acc5": 22.702, "test_5_loss": 7.121632807848168, "test_5_acc1": 1.17125, "test_5_acc5": 3.44575, "epoch": 33, "n_parameters": 5717416} {"train_lr": 0.003020537624422026, "train_loss": 6.063710031606597, "test_0_loss": 4.698054779361473, "test_0_acc1": 15.238, "test_0_acc5": 32.862, "test_5_loss": 7.121632807848168, "test_5_acc1": 1.17125, "test_5_acc5": 3.44575, "epoch": 34, "n_parameters": 5717416} {"train_lr": 0.0029660985798329645, "train_loss": 5.910670252202703, "test_0_loss": 5.064415029280474, "test_0_acc1": 13.038, "test_0_acc5": 29.468, "test_5_loss": 7.283367584701997, "test_5_acc1": 1.14525, "test_5_acc5": 3.321, "epoch": 35, "n_parameters": 5717416} {"train_lr": 0.0029107110469803756, "train_loss": 5.995033393136794, "test_0_loss": 4.890588184388418, "test_0_acc1": 14.9, "test_0_acc5": 32.556, "test_5_loss": 7.283367584701997, "test_5_acc1": 1.14525, "test_5_acc5": 3.321, "epoch": 36, "n_parameters": 5717416} {"train_lr": 0.002854429686672256, "train_loss": 5.9739233070044975, "test_0_loss": 5.277839526410142, "test_0_acc1": 12.19, "test_0_acc5": 28.048, "test_5_loss": 7.283367584701997, "test_5_acc1": 1.14525, "test_5_acc5": 3.321, "epoch": 37, "n_parameters": 5717416} {"train_lr": 0.002797310041816381, "train_loss": 5.918417158744318, "test_0_loss": 5.211479980214925, "test_0_acc1": 13.658, "test_0_acc5": 30.232, "test_5_loss": 7.283367584701997, "test_5_acc1": 1.14525, "test_5_acc5": 3.321, "epoch": 38, "n_parameters": 5717416} {"train_lr": 0.002739408482605983, "train_loss": 5.925514445745116, "test_0_loss": 5.419682004859031, "test_0_acc1": 10.596, "test_0_acc5": 24.802, "test_5_loss": 7.283367584701997, "test_5_acc1": 1.14525, "test_5_acc5": 3.321, "epoch": 39, "n_parameters": 5717416} {"train_lr": 0.002680782150889308, "train_loss": 5.883729684505341, "test_0_loss": 5.13738645015431, "test_0_acc1": 13.256, "test_0_acc5": 29.476, "test_5_loss": 7.820061174479342, "test_5_acc1": 1.0445, "test_5_acc5": 3.129, "epoch": 40, "n_parameters": 5717416} {"train_lr": 0.0026214889037779947, "train_loss": 5.8768603530623835, "test_0_loss": 5.099718636911188, "test_0_acc1": 14.214, "test_0_acc5": 32.066, "test_5_loss": 7.820061174479342, "test_5_acc1": 1.0445, "test_5_acc5": 3.129, "epoch": 41, "n_parameters": 5717416} {"train_lr": 0.002561587256548306, "train_loss": 5.914836130887389, "test_0_loss": 5.2416254764783865, "test_0_acc1": 11.636, "test_0_acc5": 26.856, "test_5_loss": 7.820061174479342, "test_5_acc1": 1.0445, "test_5_acc5": 3.129, "epoch": 42, "n_parameters": 5717416} {"train_lr": 0.0025011363248938225, "train_loss": 5.900150206830386, "test_0_loss": 5.101568935318627, "test_0_acc1": 13.276, "test_0_acc5": 30.104, "test_5_loss": 7.820061174479342, "test_5_acc1": 1.0445, "test_5_acc5": 3.129, "epoch": 43, "n_parameters": 5717416} {"train_lr": 0.00244019576658606, "train_loss": 5.86289567250809, "test_0_loss": 4.907099856829994, "test_0_acc1": 14.74, "test_0_acc5": 32.012, "test_5_loss": 7.820061174479342, "test_5_acc1": 1.0445, "test_5_acc5": 3.129, "epoch": 44, "n_parameters": 5717416} {"train_lr": 0.0023788257225985108, "train_loss": 5.835502566836721, "test_0_loss": 5.569610283303093, "test_0_acc1": 11.684, "test_0_acc5": 26.968, "test_5_loss": 7.334822424390113, "test_5_acc1": 1.49225, "test_5_acc5": 4.25875, "epoch": 45, "n_parameters": 5717416} {"train_lr": 0.002317086757755297, "train_loss": 5.902843760119544, "test_0_loss": 5.3058019126750535, "test_0_acc1": 13.534, "test_0_acc5": 29.732, "test_5_loss": 7.334822424390113, "test_5_acc1": 1.49225, "test_5_acc5": 4.25875, "epoch": 46, "n_parameters": 5717416} {"train_lr": 0.0022550398009607707, "train_loss": 5.798361852205248, "test_0_loss": 5.197937856175087, "test_0_acc1": 13.114, "test_0_acc5": 29.348, "test_5_loss": 7.334822424390113, "test_5_acc1": 1.49225, "test_5_acc5": 4.25875, "epoch": 47, "n_parameters": 5717416} {"train_lr": 0.0021927460850704548, "train_loss": 5.73072993350353, "test_0_loss": 4.734242562521595, "test_0_acc1": 17.252, "test_0_acc5": 35.896, "test_5_loss": 7.334822424390113, "test_5_acc1": 1.49225, "test_5_acc5": 4.25875, "epoch": 48, "n_parameters": 5717416} {"train_lr": 0.0021302670864610006, "train_loss": 5.69394142336125, "test_0_loss": 5.058369449217657, "test_0_acc1": 14.906, "test_0_acc5": 32.252, "test_5_loss": 7.334822424390113, "test_5_acc1": 1.49225, "test_5_acc5": 4.25875, "epoch": 49, "n_parameters": 5717416} {"train_lr": 0.002067664464360847, "train_loss": 5.709019121363294, "test_0_loss": 4.3592056012351925, "test_0_acc1": 19.068, "test_0_acc5": 39.056, "test_5_loss": 10.509043875979218, "test_5_acc1": 0.72375, "test_5_acc5": 2.323, "epoch": 50, "n_parameters": 5717416} {"train_lr": 0.002005000000000026, "train_loss": 5.700551097960972, "test_0_loss": 4.753117966484123, "test_0_acc1": 15.846, "test_0_acc5": 33.87, "test_5_loss": 10.509043875979218, "test_5_acc1": 0.72375, "test_5_acc5": 2.323, "epoch": 51, "n_parameters": 5717416} {"train_lr": 0.0019423355356391193, "train_loss": 5.7724813230031975, "test_0_loss": 4.528603377741876, "test_0_acc1": 18.206, "test_0_acc5": 37.726, "test_5_loss": 10.509043875979218, "test_5_acc1": 0.72375, "test_5_acc5": 2.323, "epoch": 52, "n_parameters": 5717416}

    opened by ksouvik52 12
Owner
Yutong Bai
CS Ph.D student @ JHU, CCVL
Yutong Bai
GitHub
Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

QSORT QSORT(Quick + Simple Online and Realtime Tracking) is a simple online and realtime tracking algorithm for 2D multiple object tracking in video s

Yonghye Kwon 8 Jul 27, 2022
PyTorch implementation of NeurIPS 2021 paper: "CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration"

PyTorch implementation of NeurIPS 2021 paper: "CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration"

null 76 Jan 3, 2023
Study of human inductive biases in CNNs and Transformers.

Are Convolutional Neural Networks or Transformers more like human vision? This repository contains the code and fine-tuned models of popular Convoluti

Shikhar Tuli 39 Dec 8, 2022
[NeurIPS 2021] Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training Code for NeurIPS 2021 paper "Better Safe Than Sorry: Preventing Delu

Lue Tao 29 Sep 20, 2022
[CVPR 2021] Teachers Do More Than Teach: Compressing Image-to-Image Models (CAT)

CAT arXiv Pytorch implementation of our method for compressing image-to-image models. Teachers Do More Than Teach: Compressing Image-to-Image Models Q

Snap Research 160 Dec 9, 2022
This repository provides the official implementation of 'Learning to ignore: rethinking attention in CNNs' accepted in BMVC 2021.

inverse_attention This repository provides the official implementation of 'Learning to ignore: rethinking attention in CNNs' accepted in BMVC 2021. Le

Firas Laakom 5 Jul 8, 2022
SIEM Logstash parsing for more than hundred technologies

LogIndexer Pipeline Logstash Parsing Configurations for Elastisearch SIEM and OpenDistro for Elasticsearch SIEM Why this project exists The overhead o

null 146 Dec 29, 2022
Research shows Google collects 20x more data from Android than Apple collects from iOS. Block this non-consensual telemetry using pihole blocklists.

pihole-antitelemetry Research shows Google collects 20x more data from Android than Apple collects from iOS. Block both using these pihole lists. Proj

Adrian Edwards 290 Jan 9, 2023
We are More than Our JOints: Predicting How 3D Bodies Move

We are More than Our JOints: Predicting How 3D Bodies Move Citation This repo contains the official implementation of our paper MOJO: @inproceedings{Z

null 72 Oct 20, 2022
An AI Assistant More Than a Toolkit

tymon An AI Assistant More Than a Toolkit The reason for creating framework tymon is simple. making AI more like an assistant, helping us to complete

TymonXie 46 Oct 24, 2022
CNNs for Sentence Classification in PyTorch

Introduction This is the implementation of Kim's Convolutional Neural Networks for Sentence Classification paper in PyTorch. Kim's implementation of t

Shawn Ng 956 Dec 19, 2022
Equivariant CNNs for the sphere and SO(3) implemented in PyTorch

Equivariant CNNs for the sphere and SO(3) implemented in PyTorch

Jonas Köhler 893 Dec 28, 2022
Pytorch Implementations of large number classical backbone CNNs, data enhancement, torch loss, attention, visualization and some common algorithms.

Torch-template-for-deep-learning Pytorch implementations of some **classical backbone CNNs, data enhancement, torch loss, attention, visualization and

Li Shengyan 270 Dec 31, 2022
📦 PyTorch based visualization package for generating layer-wise explanations for CNNs.

Explainable CNNs ?? Flexible visualization package for generating layer-wise explanations for CNNs. It is a common notion that a Deep Learning model i

Ashutosh Hathidara 183 Dec 15, 2022
Official PyTorch implementation of Less is More: Pay Less Attention in Vision Transformers.

Less is More: Pay Less Attention in Vision Transformers Official PyTorch implementation of Less is More: Pay Less Attention in Vision Transformers. By

null 73 Jan 1, 2023
This repository contains PyTorch code for Robust Vision Transformers.

This repository contains PyTorch code for Robust Vision Transformers.

null 117 Dec 7, 2022
Adversarial-Information-Bottleneck - Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck (NeurIPS21)

NeurIPS 2021 Title: Distilling Robust and Non-Robust Features in Adversarial Exa

LBK 35 Dec 26, 2022
Official PyTorch implementation of MX-Font (Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts)

Introduction Pytorch implementation of Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Expert. | paper Song Park1

Clova AI Research 97 Dec 23, 2022
PyTorch implementation of the Quasi-Recurrent Neural Network - up to 16 times faster than NVIDIA's cuDNN LSTM

Quasi-Recurrent Neural Network (QRNN) for PyTorch Updated to support multi-GPU environments via DataParallel - see the the multigpu_dataparallel.py ex

Salesforce 1.3k Dec 28, 2022