PyTorch/GPU re-implementation of the paper Masked Autoencoders Are Scalable Vision Learners

Thanks for your great work. My reproduction result for ViT-B is only 83.3, which is 0.3 lower than the paper result. And I have no idea what may cause this. My exp totally follows this repo, except the following changes: a. I use 32 V100 with batch size 128 (32x128 = 4096). The recommended setting of this repo is 64 V100 with batch size 64(64x64=4096). b. I didn't use submitit_pretrain.py but directly use main_pretrain.py for multi-node training.

Are these two changes the potential causes for performance gap? Any suggestions would be deeply appreciated~

sys.path.append('..') import models_mae dir(models_mae.Block)

The module has been successfully imported. But there is an error happened to Block Class. Use it in jupyter notebook. Any idea of it?

I m trying to pretrain a vit_small model in 1 node with 8 gpus, but the submitit_pretrain.py goes wrong with: Traceback (most recent call last): File "submitit_pretrain.py", line 131, in main() File "submitit_pretrain.py", line 89, in main args.job_dir = get_shared_folder() / "%j" File "submitit_pretrain.py", line 39, in get_shared_folder raise RuntimeError("No shared folder available") RuntimeError: No shared folder available

it seems like that I have no shared folder, so how can I pretrain mae in 1 node with 8 gpus? Thanks for your exciting job!!

Hi,

Thanks for releasing this awesome work!

I notice the code will check the version of timm.

assert timm.__version__ == "0.3.2" # version check

Does this implement support updated timm, e.g. 0.4.12?

Thanks!

Best, Vera

First, thanks for the great work. Unfortunately, here I faced something really weird. I tried to train MAE on my single node with,

python -m torch.distributed.launch --nproc_per_node=4 main_pretrain.py --batch_size 16 --model mae_vit_base_patch16 --norm_pix_loss --mask_ratio 0.75 --epochs 800 --warmup_epochs 0 --blr 1.5e-4 --weight_decay 0.05 --data_path /home/shaoshihao

Something goes weirdly is, it just run at the same speed as i ran that on single gpu with the command, python main_pretrain.py --batch_size 64 --model mae_vit_base_patch16 --norm_pix_loss --mask_ratio 0.75 --epochs 800 --warmup_epochs 0 --blr 1.5e-4 --weight_decay 0.05 --data_path /home/shaoshihao

I tracked the GPU status and it looks good. The 4 GPUs are all ~100% utilization rate when i used 4 gpus.

Any advice will be deeply appreciated.

Why the structure in the pre-trained checkpoint Vit-Base is different in the models_mae.py. The cls_token layer exist in the checkpoint Vit-Base you provide, but not in the models_mae.py.

Thanks for the great work!

I wanted to pre-train a mae_vit_base_patch16 model on imagenet-1k, and I found the loss fluctuates during training. I was wondering if this is normal for pre-training mae.

Also having the same issue as #55 and #57 when running pretrain. I have /checkpoint, /checkpoint/{USER} and /checkpoint/{USER}/experiments. Unclear how to set up --job_dir and getting the following error. Any updates or suggestions? Appreciate any help!

Traceback (most recent call last):
  File "submitit_pretrain.py", line 131, in <module>
    main()
  File "submitit_pretrain.py", line 120, in main
    args.dist_url = get_init_file().as_uri()
  File "/anaconda/envs/mae/lib/python3.6/pathlib.py", line 721, in as_uri
    raise ValueError("relative path can't be expressed as a file URI")
ValueError: relative path can't be expressed as a file URI

Section A.4 of the paper has "We search for the optimal lr for each entry in Table 5 " when referring to segmentation on ADE20k. Could you share these learning rates? My reproduced baselines are about 2 points too low.

Thanks!

Thanks for your awesome work! I am doing some experiments with MAE. Can you provide more results (e.g. mask AP) about these two models (ViT-B Mask R-CNN + MAE 25ep/50ep). I want to cite these data in my paper. Thanks! Looking forward to your reply.

Hey,

Thanks for this awesome code! I am wondering whether you plan to release the linear probing code or not. According to the third party MAE repo, the linear probing results seem hard to be reproduced. It would be great if you plan to do so. Thanks!

Hi, in this image: https://user-images.githubusercontent.com/11435359/147859292-77341c70-2ed8-4703-b153-f505dcb6f2f8.png Why do the patches that aren't masked look worse?

In the code it seems that the patch embeddings are trained on the original image rather than learning only on the patches that survive the masking process. Does this mean the implementation does not follow the paper? From the paper:

"MAE encoder. Our encoder is a ViT [16] but applied only on visible, unmasked patches. Just as in a standard ViT, our encoder embeds patches by a linear projection with added positional embeddings, and then processes the resulting set via a series of Transformer blocks. However, our encoder only operates on a small subset (e.g., 25%) of the full set. Masked patches are removed; no mask tokens are used. This allows us to train very large encoders with only a frac- tion of compute and memory. The full set is handled by a lightweight decoder, described next."

From my interpretation it seems as if the embeddings should also be run only on the small subset of patches.

I have a question that whether it (the pretrained module) works on unlabelled images or other datasets containing different images without being classified before. Thank you!

If you are getting this error, the numpy version is too new. This repository seems to be incompatible with newer numpy. I had to downgrade the numpy with the following command.

pip install "numpy<1.24"