Hi, Thanks for the cool work. The paper mentions about learned attention visualization using Attention Rollout scheme presented in Quantifying Attention Flow in Transformers (Abnar and Zuidema, 2020). I wasn't able to locate the corresponding visualization class/files in repo. Could you tell me if the repo contains the mentioned visualization related files? If so, could you please point to me location?

Thanks, Victor

Hello, I want to do transfer learning using the timeSformer model. What I want to achieve is to predict 6 outputs(regression) for every 8 frames. I am a little confused about how exactly to do it. Any help is appreciated.

Thank you

hi,

First of all awesome job bringing transformers to video recognition!! I am comparatively new to the scene. So please bear with me if there are any errors:

My question is how to load the weights of a pretrained model and retrain it for our dataset (with unfreezing the learned weights if necessary) Let's say we have loaded a pretrained model with 400 classes (Kinetics). I want to train it on my dataset with 10 classes. So I create the CSV file and split it into train, val and test. Now I define the model model = TimeSformer(img_size=224, num_classes=10, num_frames=8, attention_type='divided_space_time', pretrained_model='/path/to/pretrained/model.pyth')

But this model doesn't work for my case because, it's not trained on my dataset. What do I do next after this step?

Thanks for releasing the code. In my reproducing, I run your code without modification except for my onw data (~24w training and ~1.975w validation). But get the performance inconsistent of your paper.

I run the code with config 'TimeSformer_divST_8x32_224.yaml', the best validate top-1 accuracy after 15 epochs training only achieve 73.63% which is much lower than the reported ~77%. (8 x V100 GPUs, the running command is python tools/run_net.py \ --cfg configs/Kinetics/TimeSformer_divST_8x32_224.yaml \ DATA.PATH_TO_DATA_DIR path_to_my_dataset \ NUM_GPUS 8 \ TRAIN.BATCH_SIZE 8 \ )

Are there some configuration different or something wrong with my setting?

Hi, first of all, thank you for your amazing work for Video Understanding. I attempt to follow your work, but I can't achieve the best score in Kinetics-400 with the Code in this program. Is there anything that needs to be changed in the current version of the code for the best performance? Thank you again for your work!!!!

Best regards, FightAllDays

Thanks for your code. Could you please show me the .csv file sample？ When I tried to load videos, there is always a bug indicating that

Failed to meta load video idx 24899 from YvYW5eWaNKE; trial 0
Failed to meta load video idx 24899 from YvYW5eWaNKE; trial 1
Failed to meta load video idx 148797 from 5vPDEg8cefE; trial 0
......

Hi,

Thanks for this great work and repo! I'd like to know if you used different training parameters / processing for the HowTo100M task. I did a straightforward adaptation of the code and config used for Kinetics (just changing the number of classes to 1059) but it doesn't seem to work (loss doesn't decrease), both when fine-tuning from ImageNet directly / fine-tuning from Kinetics.

Best, Antoine Yang

Hi,

I was trying to train the Timesformer model from scratch on Kinetics-600 and the estimated time was shown as ~9 days. In the paper it was mentioned that the training time is roughly 440 V100 GPU hours. My setup is 8x Titan V GPUs, so I assumed that the training time would be closer to 50 hours. What am I missing here?

Dear Authors,

Thanks for this great repo for reproducing the results in TimesFormer. I just want to have a quick check whether you have experimented with the two small video classification datasets (i.e., UCF101 and HMDB51) and have some initial results.

This PR refactors renaming lib -> timesformer, this way one can install the library and import it locally into your project.

• reanmes lib to timesformer (lib colissions with pythons lib).
• adds example notebook
• adds environment.yml to create a fresh conda env with depencies.

Would you mind putting simple notebook on how to load a pretrained model minimally? Something like:

import torch
from lib import TimeSformer
simple_config = blablabla

model = TimeSformer(
simple_config,
)

video = torch.randn(2, 8, 3, 224, 224) # (batch x frames x channels x height x width)

pred = model(video,) # (2, 10)

so one can use the model elsewhere? I am having a hard time understanding all the file parsing.

Referring to this part of code:

if cur_global_batch_size >= cfg.GLOBAL_BATCH_SIZE: # Perform the backward pass. optimizer.zero_grad() loss.backward() # Update the parameters. optimizer.step() else: if cur_iter == 0: optimizer.zero_grad() loss.backward() if (cur_iter + 1) % num_iters == 0: for p in model.parameters(): p.grad /= num_iters optimizer.step() optimizer.zero_grad()

Is it due to batchsize being too small?

Hi Team, I am using the driving dataset to understanding road behavior. I would like to train the transformer with the predefine class such as ['left turn', 'right turn', 'U turn']. Then I would like to test my data on the Timesformer. I would like to know how to train the timesformer.

it is my first time to touch with video transformer. I want to know that how long time will take during training process if I have four RTX3080 GPUs. I would appreciate that someone can answer me!

for loading the slow fast models, cfg.TASK is required while building the model. However, it is not defined anywhere in the config file. To load the pretrained models, I am using weights from https://github.com/facebookresearch/SlowFast/blob/main/MODEL_ZOO.md. For them to be loaded properly,, cfg.TEST.CHECKPOINT_TYPE == "caffe2". Can you please let me know if this is the correct way of loading the model?

I ran the following command:

python tools/run_net.py
--cfg configs/Kinetics/TimeSformer_divST_8x32_224_4gpus.yaml
DATA.PATH_TO_DATA_DIR /home/ubuntu/vit/kinetics-dataset/k400/videos_resized
NUM_GPUS 4
TRAIN.BATCH_SIZE 16
\

During training in epoch 1, I observed the following error:

[06/30 01:54:32][INFO] train_net.py: 446: Start epoch: 1 [06/30 01:54:46][INFO] distributed.py: 995: Reducer buckets have been rebuilt in this iteration. [06/30 01:54:59][INFO] logging.py: 95: json_stats: {"_type": "train_iter", "dt": 1.46034, "dt_data": 0.00346, "dt_net": 1.45688, "epoch": "1/15", "eta": "7:33:55", "gpu_mem": "7.68G", "iter": "10/1244", "loss": 6.05343, "lr": 0.00500, "top1_err": 100.00000, "top5_err": 100.00000} [06/30 01:55:14][INFO] logging.py: 95: json_stats: {"_type": "train_iter", "dt": 1.50371, "dt_data": 0.00334, "dt_net": 1.50036, "epoch": "1/15", "eta": "7:47:09", "gpu_mem": "7.68G", "iter": "20/1244", "loss": 6.16927, "lr": 0.00500, "top1_err": 100.00000, "top5_err": 100.00000} ../aten/src/ATen/native/cuda/Loss.cu:271: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed. terminate called after throwing an instance of 'c10::CUDAError' what(): CUDA error: device-side assert triggered

Follow by a lengthy exceptions being raised: Exception raised from createEvent at ../aten/src/ATen/cuda/CUDAEvent.h:166 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f10084d2612 in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: + 0xea8e4a (0x7f1009892e4a in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so) frame #2: + 0x33a968 (0x7f1051d51968 in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

.....

In my instance I have four Tesla T4 GPU with Driver Version: 510.47.03 CUDA Version: 11.6

What does the error I see above means, and how do I fix it?