Hi,

Thanks for sharing this amazing work. I am just wondering where I can get the ImageNet pretrained weights as I see that Kinetics uses pretrained ImageNet weights for training in the paper. I would like to retrain the model.

temporal_aggregation.py第39行对x进行reshape时按照我的理解应该是由B,N,C变为nvids, self.clip_length, NC，为何最后一个维度NC还可以与TransformerEncoderLayer中的embed_dim对齐？ 按理说这里x的输入维度在经过transformer_model.py第179行的embadding后已经变为B,N,C并一直保持到时间聚合模块。这里的代码实在没有看懂，还希望作者如果看到的话能做出一些解答，谢谢

Hi researchers. Great work for getting rid of multi-view inference. Some problems in my experiment: Many recent methods use non-local copies of Kinetics-400 dataset for experiments since more and more YouTube videos are unavailable. While using validation set of non-local copies and torchvion.datasets.Kinetics400 API(in src/utils/utils.py) for loading clips, there are around 39% of the validation data is discarded. In my experiment, top1 accuracy is the same as STAM_16 shows but fewer data is used. Print valid_data.len() at utils.py and it should show there are around 11897 clips if using non-local copies(19761 total). I believe STAM using one clip per video as the paper described. It seems that torchvion.datasets.Kinetics400 API discards same videos due to params settings. I also change the extensions('avi', 'mp4') to extensions('avi','mp4','mkv','webm') to cover all format, but still 11.5% discarded. So could you explain more about your experiment settings like details about dataset source (Kinetics official download links or non-local copies), how many samples in validation set and list of validation file names) or make your validation data public if convenient. Thank you.

Dear researchers,

Thank you for this great work!

I have a confusion about the linear projection. As of the paper, "We design a convolution-free model that is fully based on self-attention blocks for the spatiotemporal domain" So, I was expecting no Conv block in the implementation. But I see a Conv2D in the linear projection. https://github.com/Alibaba-MIIL/STAM/blob/master/src/models/transformer_model.py#L224

Can you provide some explanation on this?

Dear researchers,

Thank you for this very nice piece of of work.

Can you also provide the code you use for the training ?

Without it, it is impossible to reproduce your results, and validate your conclusion.

Best regards,

This is great work and I want to read the code. But I am a rookie on pytorch, I can`t find DropPath, to_2tuple, register_model module, can you tell me where to find them? Thanks a lot!

I want to know hyperparameters in Kinetics400(root=source, step_between_clips=args.step_between_clips, frames_per_clip=args.frames_per_clip, frame_rate=args.frame_rate)

Thank you for your great work! Q1: I can't understand the nvids in the code blew. Does nvids represent batch number? Q2: What is the value of pos_drop should be set ?

  def forward(self, x):
    nvids = x.shape[0] // self.clip_length
    x = x.view((nvids, self.clip_length, -1))

    cls_tokens = self.cls_token.expand(nvids, -1, -1)
    x = torch.cat((cls_tokens, x), dim=1)
    x = x + self.pos_embed
    # x = self.pos_drop(x)

    x.transpose_(1,0)
    # x = x.view((self.clip_length, nvids, -1))
    o = self.transformer_enc(x)
    # o = o.mean(dim=0)

    return o[0]

Quiet promising work which shows the great potential of Video Transformer. Looking forward to the training code and details about training hyperparameters!

Hi, Thanks for this implementation. But I still have some problems. The parameters in model are initialize as follow：

  def _init_weights(self, m):
    if isinstance(m, nn.Linear):
      with torch.no_grad():
        trunc_normal_(m.weight, std=.02)
      if isinstance(m, nn.Linear) and m.bias is not None:
        nn.init.constant_(m.bias, 0)
    elif isinstance(m, nn.LayerNorm):
      nn.init.constant_(m.bias, 0)
      nn.init.constant_(m.weight, 1.0)

Does this mean I should make these parameters trainable when I train this model on my own dataset?

Hi,

Thanks for your great work!

I'm trying to reproduce the results with your code now. However, due to limited computational resources, we can only reach batch_size 32 for the STAM 16 network. Would you have any idea about how good the performance this network can achieve under this setting?

Thanks.

Hello, your work is great, thanks for sharing the code, could you share training-related information? Or the model trained on UCF101, I am very interested in this, thank you very much！

Are there any differences between nn.TransformerEncoder and class Block in transformer_model.py?

Have you ever tried to use class Block instead of nn.TransformerEncoder in aggregate module just like what you do in spatial dimension?

I appreciate for the brilliant model you have created, but I am still confusing about this questions, I would appreciate it if you could reply.

Hello,

This work is really inspiring, and thanks for sharing the code. Meanwhile, could you please also share the training hyper-parameters (e.g., learning rate, optimizer, warmup lr, warmup epochs, etc.)? I would really like to train the model to get a deeper understanding of the model.

Thanks, Steve