In the original paper, the authors state that "we remove the whole head (two linear layers) and replace it by a single, zero-initialized linear layer outputting the number of classes required by the target dataset. We found this to be a little more robust than simply re-initializing the very last layer."

May I know which code snippet is related to this?

Training on the 3090 gets slower and slower as time goes on but the 2080ti doesn't have this problem

torch 1.8.0.dev20201130+cu110 torchvision 0.9.0.dev20201130+cu110 NVIDIA-SMI 455.23.04 Driver Version: 455.23.04 CUDA Version: 11.1

Hi, I checked your code at https://github.com/jeonsworld/ViT-pytorch/blob/878ebc5bd12255d2fffd6c0257b83ee075607a79/models/modeling.py#L154.

Your implementation is: Attention(LayerNorm(x)) + x, but the original Transformer is: LayerNorm(x +Attention(x)). Is this an error or deliberately implemented like this?

@jeonsworld okay this is completely different questions, but I should ask it because i have not seen it anywhere else. how did you save tensorboard so we can just click on it and see it? I want to do it as well. should I save it in some format or do anything special? please direct me to any link/material that can help me with that. Thanks :)

when i used code，the error occurs error location：

 models\modeling.py", line 195, in load_from
query_weight = np2th(weights[pjoin(ROOT, ATTENTION_Q, "kernel")]).view(self.hidden_size, self.hidden_size).t()
File "d:\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 259, in __getitem__
raise KeyError("%s is not a file in the archive" % key)
KeyError: 'Transformer/encoderblock_0\\MultiHeadDotProductAttention_1/query\\kernel is not a file in the archive'

I would like to ask where should I put this ViT-H_14.npz ? I created a checkpint folder and just put the ViT-H_14.npz in there，but I got this error。 the INFO：01/12/2021 19:51:55 - INFO - models.modeling - load_pretrained: resized variant: torch.Size([1, 257, 1280]) to torch.Size([1, 730, 1280]) my input： imgsize(384*384),batch.size(64){train.batch=eval.batch}. Is there anything I haven't modified？

Hi

I tried your notebook but the link is dead I think. I got the error of forbidden.

HTTPError                                 Traceback (most recent call last)
<ipython-input-4-14f159ea9fa3> in <module>
      1 # Test Image
      2 img_url = "https://images.mypetlife.co.kr/content/uploads/2019/04/09192811/welsh-corgi-1581119_960_720.jpg"
----> 3 urlretrieve(img_url, "attention_data/img.jpg")
      4 
      5 # Prepare Model
...
HTTP Error 403: Forbidden

Thanks for your hard work! I wonder if there are some pre-trained weights only using Imagenet2012? I found that the pre-trained ResNet provided by torchvision may be pre-trained only on Imagenet2012 so I want to take ViT and ResNet for a fair comparison.

Hi, by executing this

python3 train.py --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz

I encounter the error:

Traceback (most recent call last):
  File "train.py", line 17, in <module>
    from apex import amp
  File "/home/tiger/.local/lib/python3.7/site-packages/apaex/__init__.py", line 13, in <module>
    from pyramid.session import UnencryptedCookieSessionFactoryConfig
ImportError: cannot import name 'UnencryptedCookieSessionFactoryConfig' from 'pyramid.session' (unknown location)

Thank you so much for sharing your codes. I try to employ Vit as the encoder and follow a common decoder to build a segmentation network. I train it from scratch but found the loss can't drop since the beginning of training, and the results keep near 0. Is there any trick for training Vit correctly? Is it very important to load the pre-train model to fine-tune? Here is my configuration: patch_size=16 hidden_size=16*16*3 mlp_dim = 3072 dropout_rate = 0.1 num_heads = 12 num_layers = 12 lr=3e-4 opt=Adam weight_decay=0.0

Thanks for your great repo !

I cannot find the link to download the imagenet-21k dataset, so is there any way to download Imagenet-21k now? Thanks a lot~

Thanks for creating and uploading this easily usable repo!

In addition to the validation accuracy on the entire validation set that is printed out by default, we printed out the training accuracies of the model and we observe that the training accuracy is 6-8% lower than the validation accuracy. Is that reasonable/accurate since we usually expect the training accuracy to be higher than the validation accuracy?

This was for a ViT-B_16 model, pretrained on ImageNet-21k and during the fine-tuning phase on CIFAR 10. To get the training accuracies, we used model(x)[0] to get the logits, loss and predictions for each batch and used the AverageMeter() to calculate the running accuracies. Additionally, to get the accurate training accuracy over the entire training set, we passed the training set to a copy valid() (with only changes to print statements). Both the running training accuracy and the training accuracy over the entire training set was lower than the validation accuracy by 6-8%. For instance, after 10k steps, training accuracy was 92.9% (over entire train set) and validation accuracy was 98.7%. We used most of the default hyperparameters (besides batch size and fp_16) and did not make other changes to the code.

Please let us know if this lower training accuracy is expected or if its calculation is incorrect. Thanks in advance.

Hi, thanks for releasing this code.

I have tried to run the CIFAR-10 (as well as CIFAR-100) example, but in both cases the validation (and training) loss do not decrease, and the validation accuracy gets stuck in 0.01. Is there any hyper-parameter that I need to change from the example code?

Thanks!

Hi, I am using ViT-H_14 pre-trained to perform binary classification of biomedical images. The dataset I have available is very small: I use about 300 images to perform fine tuning and about 30 images for validation. The goal is to classify the images based on the aggressiveness of the tumor represented (Low grade (0) - High grade(1)). However, I noticed that during the prediction, each image is always associated with the label 0, and going to look on the logits, i found that are always produced logits identical pairs (eg [[ 6.877057e-10 -6.877057e-10]]), which are translated into probability pairs of about (0.49,0.51).

Searching the various forums I found many different tips: vary the learning rate (which I decreased to 1e-8), decrease the batch size (from 8 to 2), etc.. Unfortunately none of this works. The last thing I want to try is to increase considerably the number of epochs (at the moment I have trained for only 100 epochs), but before doing so I wanted to see if someone had a more specific suggestion, or even if someone can tell me if this architecture is too much for a dataset so small.

Thanks a lot in advance