Hi, I’m wondering if you can provide a recipe to reproduce the results of CoCo detection? I’ve tried to use your pre-trained checkpoint to train the downstream task with Mask R-CNN, but cannot get the results reported in the paper. Not sure if there was something wrong during the training. Could you please provide more details? Thank you!

Hi, I am trying to reproduce the knn results but fail to do so. I am using the pretrained model from the checkpoint on ImageNet-1K following the script provided.

I got the following results:

10-NN classifier result: Top1: 1.876, Top5: 3.462
20-NN classifier result: Top1: 1.872, Top5: 3.912
100-NN classifier result: Top1: 1.85, Top5: 4.884
200-NN classifier result: Top1: 1.834, Top5: 5.352

Is there any chance that the model checkpoint is incorrect?

Thanks!

Hello, I have read your paper and found it very interesting. I was particularly intrigued by Table 1 where you compare the throughput against other methods, including DINO with a deit_tiny and patch size of 16. From the table, EsViT with Swin-T(/W=7) has a throughput of 808 and DINO with DeiT-T/16 has 1007. So I expected EsViT to be +- slower by 20%. Yet, when I run both I do not get this. I attached both logs below.

DINO

arch: deit_tiny
batch_size_per_gpu: 200
clip_grad: 3.0
data_path: /ilsvrc2012/ILSVRC2012_img_train
dist_url: env://
epochs: 100
freeze_last_layer: 1
global_crops_scale: (0.4, 1.0)
gpu: 0
local_crops_number: 8
local_crops_scale: (0.05, 0.4)
local_rank: 0
lr: 0.0005
min_lr: 1e-06
momentum_teacher: 0.996
norm_last_layer: True
num_workers: 24
optimizer: adamw
out_dim: 65536
output_dir: output_dir
patch_size: 16
rank: 0
saveckp_freq: 10
seed: 0
teacher_temp: 0.04
use_bn_in_head: False
use_fp16: True
warmup_epochs: 10
warmup_teacher_temp: 0.04
warmup_teacher_temp_epochs: 0
weight_decay: 0.04
weight_decay_end: 0.4
world_size: 4
Data loaded: there are 1281167 images.
Student and Teacher are built: they are both deit_tiny network.
Loss, optimizer and schedulers ready.
Starting DINO training !

Epoch: [0/100] Total time: 0:38:22 (1.438374 s / it)
Averaged stats: loss: 6.691907e+00 (8.885959e+00)  lr: 1.551861e-04 (7.808108e-05)  wd: 4.008760e-02 (4.002958e-02)

EsViT

aa: rand-m9-mstd0.5-inc1
arch: swin_tiny
aug_opt: dino_aug
batch_size_per_gpu: 48
cfg: experiments/imagenet/swin/swin_tiny_patch4_window7_224.yaml
clip_grad: 3.0
color_jitter: 0.4
cutmix: 1.0
cutmix_minmax: None
data_path: /ilsvrc2012/ILSVRC2012_img_train
dataset: imagenet1k
dist_url: env://
epochs: 100
freeze_last_layer: 1
global_crops_scale: (0.4, 1.0)
gpu: 0
local_crops_number: (8,)
local_crops_scale: (0.05, 0.4)
local_crops_size: (96,)
local_rank: 0
lr: 0.0005
min_lr: 1e-06
mixup: 0.8
mixup_mode: batch
mixup_prob: 1.0
mixup_switch_prob: 0.5
momentum_teacher: 0.996
norm_last_layer: False
num_mixup_views: 10
num_workers: 10
optimizer: adamw
opts: []
out_dim: 65536
output_dir: output_dir
patch_size: 16
pretrained_weights_ckpt: 
rank: 0
recount: 1
remode: pixel
reprob: 0.25
resplit: False
sampler: distributed
saveckp_freq: 5
seed: 0
smoothing: 0.0
teacher_temp: 0.07
train_interpolation: bicubic
tsv_mode: False
use_bn_in_head: False
use_dense_prediction: True
use_fp16: True
use_mixup: False
warmup_epochs: 10
warmup_teacher_temp: 0.04
warmup_teacher_temp_epochs: 30
weight_decay: 0.04
weight_decay_end: 0.4
world_size: 4
zip_mode: False
Data loaded: there are 1281167 images.
=> merge config from experiments/imagenet/swin/swin_tiny_patch4_window7_224.yaml
Unknow architecture: swin_tiny
Student and Teacher are built: they are both swin_tiny network.
Loss, optimizer and schedulers ready.
Starting training of EsViT ! from epoch 0

Epoch: [0/100] Total time: 2:09:19 (1.162958 s / it)
Averaged stats: loss: 4.714716 (6.780889)  lr: 0.000037 (0.000019)  wd: 0.040089 (0.040030)

So EsViT (with swin_tiny W=7) is about 3 times slower than DINO (with deit_tiny and P=16). This is run on a machine with 4xV100 GPUs. In both cases, I set the batch size to the +- highest value I could without having out of memory exceptions.

Is it the case that my run of EsViT should be this row in table 1?

EsViT, Swin-T 28 808 78.1 75.7

If so, do you know why I am getting such contradictory results?

Thank you!

Hi @ChunyuanLI, I've noticed the usage of mixup and cutmix during pre-training, which is not included in DINO. I'm wondering the performance gain brought by applying mixup & cutmix. Have you ever run any related experiments pre-trained w.o. mixup? I'm especially interested in vanilla DINO with Swin-T/Swin-B as the backbone, i.e., EsViT w. only view-level task, w.o. mixup & cutmix. It would be nice if you could inform me of those results.

Hello, Thanks for the code. I have noticed that the multi-crop trick can boost the performance by about 5% top-1 acc (on DINO, SwAV). Since your code base supports disabling this trick, did you conduct the experiments without this multi-crop trick, and would you be so kind that share the results on ImageNet?

Hi!

I am trying to load esvit on Google Colaboratory with the following code:

!git clone https://github.com/microsoft/esvit.git
!pip install -r ./esvit/requirements.txt

import models.vision_transformer as vits

I got the following error:

...
[/usr/local/lib/python3.7/dist-packages/timm/models/layers/helpers.py](https://localhost:8080/#) in <module>
      4 """
      5 from itertools import repeat
----> 6 from torch._six import container_abcs
      7 
      8
ImportError: cannot import name 'container_abcs' from 'torch._six' (/usr/local/lib/python3.7/dist-packages/torch/_six.py)

which seems to be related to the torch version. However, downgrading torch (<1.11.0) I obtain errors on other torch imports.

Is there available a testing notebook?

Hello, I cannot seem to find in the paper which features are used for doing the correspondence matching in the appendix. Is it the last layer features (rough-grained) or the first layer features (fine-grained) or a combination of features at all depths (if so how is the combination?) ? Thanks!

https://github.com/microsoft/esvit/blob/c5d73eba76d76136a5ed162263b934df57ec04dc/models/swin_transformer.py#L300

In this line, should (self.H, self.W) be (H, W)?

A liittle confused that self.head_dense is not explicitly defined in several model files. There is only a None assignment statement in:

https://github.com/microsoft/esvit/blob/main/models/swin_transformer.py#L655 https://github.com/microsoft/esvit/blob/main/models/vision_longformer.py#L518 https://github.com/microsoft/esvit/blob/main/models/vision_transformer.py#L171

Am I missing something?

Hi all,

In table 4 of the arxiv preprint https://arxiv.org/pdf/2106.09785.pdf, the reported AP^bb of Supervised = 46.0 Why is this number lower than the ones reported in the Swin paper ?

• See Table 2 (b) of https://arxiv.org/pdf/2103.14030.pdf
• Swin-S AP^box=51.8

Also, what object detection method are you using? Is it Mask RCNN or Cascade? There is no mention of the detection method used in the paper.

Thanks!

What a custom dataset structure should be like and how to train on it? Let's say I have a dataset of two classes as the folder (binary): 1. Has cat, 2. No cat. In each sub-folder, there are images. What changes to the code and dataset should I make? Thanks in advance.

This pull request adds the JSON schema for cgmanifest.json.

FAQ

Why?

A JSON schema helps you to ensure that your cgmanifest.json file is valid. JSON schema validation is a build-in feature in most modern IDEs like Visual Studio and Visual Studio Code. Most modern IDEs also provide code-completion for JSON schemas.

How can I validate my cgmanifest.json file?

Most modern IDEs like Visual Studio and Visual Studio Code have a built-in feature to validate JSON files. You can also use this small script to validate your cgmanifest.json file.

Why does it suggest camel case for the properties?

Component Detection is able to read camel case and pascal case properties. However, the JSON schema doesn't have a case-insensitive mode. We therefore suggest camel case as it's the most common format for JSON.

Why is the diff so large?

To deserialize the cgmanifest.json file, we use JSON.parse(). However, to serialize the JSON again we use prettier. We found that, in general, it gave smaller diffs than the default JSON.stringify() function.

Hi,

I'm retraining from scratch EsVIT on a custom dataset (1.7M images) with tiny swin, W=14, and a batch size of 64, default lr and wd, and the following hp --teacher_temp 0.04
--warmup_teacher_temp 0.03
--momentum_teacher 0.9996
--warmup_epochs 10
--warmup_teacher_temp_epochs 30
--use_dense_prediction True
--use_fp16 True
--out_dim 65536
--epochs 300 \

The loss does not decrease from epoch 70 onwards.

Which hp would you recommend tuning now resuming from let's say epoch 70 ?

Thanks

Bumps numpy from 1.19.3 to 1.22.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

hi, I use the swin-transformer.py to load the swin-tiny model pretrained by imagenet1k. And the message is here: msg: _IncompatibleKeys(missing_keys=['layers.0.blocks.1.attn_mask', 'layers.1.blocks.1.attn_mask', 'layers.2.blocks.1.attn_mask', 'layers.2.blocks.3.attn_mask', 'layers.2.blocks.5.attn_mask', 'head.weight', 'head.bias'], unexpected_keys=['head.mlp.0.weight', 'head.mlp.0.bias', 'head.mlp.2.weight', 'head.mlp.2.bias', 'head.mlp.4.weight', 'head.mlp.4.bias', 'head.last_layer.weight_g', 'head.last_layer.weight_v']) why is there missing keys here?

Hello.

Thank you for the wonderful work! I have some questions about the learning rate used to pretrain the Swin model in Table 1. As the logs show, the learning rate for the Swin-T model is 0.0005180447994195404 at 201 epoch, while the learning rate for the Swin-S/B model is 0.00025939212681290886 at 201 epoch. however, the parameters shown for the 'args' keyword in the pre-trained model are the same.

Could you please tell me why there is a difference in learning rate in the training log?

Thanks in advance.