关于论文中Attention Masking，对于这个创新点有比较大的疑惑。因为个人觉得只需要根据mask，把token的内容置位-inf就可以了，不需要交互结束之后，再在softmax过程中加上用mask进行制约。

另外，这篇文章提出的方法是不是只能在预测的时候减少计算量，因为在训练的时候进行Self-Attention，还是所有token都进行了Self-Attention，这样的文中的计算量就没有下降。反而因为加入了预测模块，计算量还上升了。

Hi~ Thx for your great work and excellent code!

I have some questions about dyswin code, hope you could help me out.

1. In code: https://github.com/raoyongming/DynamicViT/blob/84b4e2a9b1f11199bd1e2ff506969b0d64e6f55b/models/dyswin.py#L683

   when I input a tensor with shape (2, 3, 224,224), the if condition is activated(len(x) is equal to 2, and the following operation obviously goes wrong.

   However, when I set batchsize to another number, this error is disappeared. Could you please explain the code here?

2. When I use lvvit-s pre-trained model as inference backbone, I find that the output token length of lvvit-s is cut shorter compared with standard lvvit output token length, and it's correct, right?

   But when I change the backbone from lvvit-s to swin-s, the output token length is the same as standard swin-s

   For example, if input tensor with shape (4, 3,224,224), the output shape of dyswin(temporarily ignore the avgpool and later layers) is (4, 49, 768 ) while the standard swin-s is also output this shape tensor. It seems that token length reduction has not been achieved.

   Could you please explain the here? Any advice could be greatly appreciate! :)

The shape of Z_global is (C,) and the shape of Z_local is (N,C).

I have checked your code in line 263 of DynamicViT/models/dyswin.py, I think your idea is to concatenate of Z_local_i with Z_global? Since there is no dimension N in Z_global.

So I think the subscript 'i' of Z_global in equation 4 in your paper maybe a mis-type?

Hello, thank you for your code. I spent some time reading your code carefully. But I still can not understand the following lines(https://github.com/raoyongming/DynamicViT/blob/84b4e2a9b1f11199bd1e2ff506969b0d64e6f55b/models/dyvit.py#L174). Could you please give me sone advices? Why abstract the max value from the atten?

Hi, it is wonderful and solid work. I have several questions about Flops. In your paper, you compute the model's flops (the unit is Gflops). Which package can compute the Gflops? The popular package from ptflops import get_model_complexity_info as I know, its unit is output as MAc not Gflops. Thanks in advance.

Hi, thanks for the great work! I am training the dvit for the deit-small, and the checkpoint it stored has size of 364 MB, while the original model and the weights you shared are ~90MB. Am I making an error?

Tried arch: deit_small, deit_256 Dataset: Imagenet-1k-val File structure:

│ILSVRC2012_val/
├──val/
│  ├── 1(image label)
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── 2(image label)
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......

When I ran python3 infer.py --data-path /home/ubuntu/datasets/ILSVRC2012_val/ --arch deit_small --model-path /home/ubuntu/models/dynamic-vit_384_r0.7.pth --base_rate 0.7, the result is Acc@1 0.080 Acc@5 0.582.

The filenames ( image label ) are decided by ILSVRC2012_validation_ground_truth.txt in the development kit. Is this problem due to the wrong file name, which causes the model to predict a different label than the real one? Should I modify the filename to WNID? But the Val dataset has no WNID, how could I confirm that?

Thanks

Why LVViT_Teacher return aux_head(x[:, 1:]) instead of tokens?

LVViT_Teacher:

x = self.norm(x)
x_cls = self.head(x[:,0])
x_aux = self.aux_head(x[:,1:])
return x_cls, x_aux

VisionTransformerTeacher:

feature = self.norm(x)
cls = feature[:, 0]
tokens = feature[:, 1:]
cls = self.pre_logits(cls)
cls = self.head(cls)
return cls, tokens

And, can I use the LVViT_Teacher to distill deit_small?

Hello,

Thank you for your work. I'm reading your implementation code for computing the attention mask. In lvvit.py line 665, you used F.gumbel_softmax(pred_score) to find the binary mask, but the last layer for computing pred_score is a log_softmax (line 529, same file). Shouldn't it be just a linear unit rather than a log_softmax, since F.gumbel_softmax takes logit rather than probability?

Thank you.

!wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1iZc6d27EuEnlfUpJoNhsZEkt6GVgPy7-' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1iZc6d27EuEnlfUpJoNhsZEkt6GVgPy7-" -O lvvit_m-56M-224-84.0.tar && rm -rf /tmp/cookies.txt I can use the code above download lvvit_m-56M-224-84.0.tar But why I can't use !tar -xvf lvvit_m-56M-224-84.0.tar to extract the file It says

tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors

It would happen randomly in any epoch. And it still appears after setting a lower learning rate or turning off the amp.

Then I found it usually occurs in module 'PredictorLG', if 'policy' is an all-zero matrix, there will be nan in global_x.

Is it ok to add a very small value in the denominator of the global_x, e.g. 1e-6?

Hi, I follow the training command:

python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --output_dir logs/dynamicvit_deit-b --model deit-b --input_size 224 --batch_size 128 --data_path /path/to/ILSVRC2012/ --epochs 30 --base_rate 0.7 --lr 1e-3

and get the final results:

• Acc@1 80.754 Acc@5 94.950 loss 0.888 Accuracy of the model on the 50000 test images: 80.8% Max accuracy: 80.80%

• Acc@1 80.760 Acc@5 94.970 loss 0.887 Accuracy of the model EMA on 50000 test images: 80.8% Max EMA accuracy: 80.86%

fail to reproduce 81.3% for deit-base with 0.7 keeping ratio reported in the paper (~0.5% drop). But I got 79.2% for deit-small (79.3% in the paper). These two experiments were conducted in the same python envrionments.

My envrionments: python 3.10 torch 1.12.1 torchvision 0.13.1 timm 0.3.2

Could you provide the training instructions/checkpoints that achieved ~81.3% accuracy for deit-base?

Thanks!

Hi, thanks for your inspiring work!

I notice that you used the default temperature=1 in all your F.gumbel_softmax implementations, and it didn't anneal to 0. Do you have any suggestions on why should we fix this temperature? Because I thought shouldn't we decrease this temperature during training to make it closer and closer to the real categorical distribution, as indicated in the Gumbel Softmax paper?

Hi, it's a quite solid and promising work but I have some questions. (1) In the paper, you perform an average pooling with kernel size 2 × 2 after the sixth block for the structural downsampling. But in Table 3, you show the results of structural downsampling and static dynamic token sparsification. What is the difference between structural downsampling and static token sparsification since their ACCs are not same? (2) I'm interested in the average pooling with kernel size 2 × 2. Did you do extra experiments in the position of such structural downsampling, like the seventh block or the tenth block in ViT? (3) Could you provide the codes for reproducing the results of structural downsampling and static token sparsification in Table 3 and the probability heat-map in Figure 6?

Thanks for your help!