Hi, congratulate on your great work! Could you release label files (train.csv val.csv test.csv) for ssv2? I tried to generate these files following the guidance in DATASET.md and use them for evaluation on ssv2 by running run_class_finetuning.py with --eval, but the accuracy is abnormal(too low), and I guess maybe the csv files I generated were wrong. Could you release the csv files of K400 and ssv2 for everyone to download? Thanks!

Hi, Thanks for sharing the code! I have tested the model on UCF-101 val dataset (split-1) you provided in https://drive.google.com/file/d/1MSyon6fPpKz7oqD6WDGPFK4k_Rbyb6fw/view?usp=sharing. But the accuracy is less than 70%. How can I get the 91.3 accuracy you reported in model_zoo?

Hi VideoMAE team :)

I've implemented VideoMAE as a fork of 🤗 HuggingFace Transformers, and I'm going to add it soon to the library (see https://github.com/huggingface/transformers/pull/17821). Here's a notebook that illustrates inference with it: https://colab.research.google.com/drive/1ZX_XnM0ol81FbcxrFS3nNLkmn-0fzvQk?usp=sharing

The reason I'm adding VideoMAE is because I really like the simplicity of it, it was literally a single line of code change from ViT (nn.Conv2d -> nn.Conv3d).

As you may or may not know, any model on the HuggingFace hub has its own Github repository. E.g. the VideoMAE-base checkpoint fine-tuned on Kinetics-400 can be found here: https://huggingface.co/nielsr/videomae-base. If you check the "files and versions" tab, it includes the weights. The model hub uses git-LFS (large file storage) to use Git with large files such as model weights. This means that any model has its own Git commit history!

A model card can also be added to the repo, which is just a README.

Are you interested in creating an organization on the hub, such that we can store all model checkpoints there (rather than under my user name)?

Let me know!

Kind regards,

Niels ML Engineer @ HuggingFace

Hi Zhan,

Thank you for your excellent work! We are surprised by VideoMAE's data efficiency (paper section 4.3), especially on small datasets like HMDB51. We are trying to use your code to reproduce your experiment on HMDB51. However, we cannot achieve the same finetune accuracy as the table2 reported (61.1 %):

Our experiments show that the model is converged after 2400 pretraining epochs. We are using eight Tesla V100 32GB GPUs. Also, we changed the batch size, learning rate and temporal stride as the appendix described.

I wonder if you can also share your complete experiment configurations for UCF101 and HMDB51? Some settings like warmup epochs number may also be critical for reproducing.

Hi, I have tried to reproduce the VideoMAE performance on SSV2 dataset. I run the experiments on four A100 machines (each includes eight 80G GPUs) and modify --nnodes=4, batch_size 64, such that our total batch size is the same. However, the performance is not consistent with the reported one. I checked the log you provided and notice that the learning rate is different. It seems that your learning rate is not 1.2e-3 (1.5e-4*2048/256) after the warm-up stage. Thanks a lot and I am looking forward to your reply.

First of all, thanks for your excellent work.

I'm re-producing your work by using one node with 8 GPUs.

In your pre-training scripts for K400 and SSv2,

the effective batch size : 32 x 8 nodes x 8 gpus = 2048

but your paper denotes 1024.

Could you let me know which one is correct?

Thanks in advance.

I encontered missing keys error while finetunening, I resume from model pretrained with pretrain model and using it while finetune. how to solve this problem?

Thanks for the great work! But I have one question that was not discussed in the paper, which is the inference speed.

I understand that during training the speed is fast, since you masked out 90% of the patches. However, during test time, I assume you retain all the patches. Since the attention in ViT is quadratic in cost, does it mean you will need 10*10 = 100 times the flops compared to the pretrain phase?

If that is the case, how much time does it take to do a single forward pass?

Hi author: Thanks for your impressive work, I run your code "run_mae_pretraining.py" and it comes an error that "engine_for_pretraining.py", line 79, in train_one_epoch loss_scale_value = loss_scaler.state_dict()["scale"]" , as shown below; p.s. I used the pretrained ssv2's checkpoint;

The NeurIPS camera ready version (v3 on arXiv) has some significantly higher results than the previous paper version (v2 on arXiv). E.g. for ViT-B pretrained on K400 for 1600 epochs, performance on K400 jumps from 80.9% to 81.5%. For ViT-B pretrained on SSv2 for 2400 epochs, performance on SSv2 jumps from 70.6% to 70.8%. Could the authors share the updated finetuning code and configs? I am unable to reproduce the new results. My results are close to what is reported in v2 of the paper

Hi,

I am trying to use data_clean.py for Kinetics 400 but am unable to use petrel_client from the import statement: from petrel_client.client import Client I have installed petrel but that has not worked and I have not been able to find a petrel_client package online.

Hello.

How do I reproduce K400 (pretrain, 1600ep) -> UCF101 (finetune) performance as shown in the paper (96.1 Top-1 in UCF)? Should I use the same config as UCF101 (pretrain) -> UCF101 (finetune)?

Hi, I am reproducing the experiment you conducted because it is very interesting. Among them, a linear probing experiment was conducted, but it failed to obtain the top 1 acc of 38.9% presented in Table 3. The top 1 acc I got was 33.9% in 1x1 view condition and 34.7% in 2x3 view condition. If possible, can you share the code that you used to run the linear-prob experiment? It would be a great help to me if you share

thanks

Thanks for the excellent work!

Could you provide the training configuration for the AVA dataset? I tried to reproduce the results but observed overfitting.

Is there a script or details available for linear probing? The original MAE paper suggests theres a substantial difference between setups for finetuning and linear probing for MAE models and wanted to be able to reproduce results.

Thanks!