I run the motionformer for three different setting on Kinetics datasets.

I run motionformer_224_16x4.yaml with batchsize 8 using 8GPU. Finally, I got 71.48 top-1 on val dataset.

I run configs/K400/joint_224_16x4.yaml. I got 76.63 on val dataset.

I run configs/K400/divided_224_16x4.yaml. I got 76.27 on val dataset.

A typo in the original code meant that the value tensors for the temporal attention step were identical to the input instead of being multiplied by a learned projection matrix (v = x rather than v = Wx). The original code is kept to facilitate replication and can be used by setting use_original_code=True, but is not recommended.

Hello, thanks for this great work and the shared code. A small issue I have been trying to figure out: in vit_helper.py.TrajectoryAttention.forward(), when the temporal attention is applied:

1.         x = rearrange(x, '(b h) s f d -> b s f (h d)', b=B)
2.         x_diag = rearrange(x, 'b (g n) f d -> b g n f d', g=F)
3.         x_diag = torch.diagonal(x_diag, dim1=-4, dim2=-2)
4.         x_diag = rearrange(x_diag, f'b n d f -> b (f n) d', f=F)
5.         q2 = self.proj_q(x_diag)
6.         k2, v2 = self.proj_kv(x).chunk(2, dim=-1)
7.         q2 = rearrange(q2, f'b s (h d) -> b h s d', h=h)
8.         x, k2, v2 = map(
9.             lambda t: rearrange(t, f'b s f (h d) -> b h s f d', f=F,  h=h), (x, k2, v2))
10.         q2 *= self.scale
11.         attn = torch.einsum('b h s d, b h s f d -> b h s f', q2, k2)
12.         attn = attn.softmax(dim=-1)
13.         x = torch.einsum('b h s f, b h s f d -> b h s d', attn, x)
14.         x = rearrange(x, f'b h s d -> b s (h d)')

in line 249 (here 13), why is EINSUM operation is applied on attn and _x? in the paper, the temporal attention is applied as usual: but in the code it seem like v^{\sim}_{stt'} is replaced with the reshaped version x.

In addition, I am confused since x is reshaped in line (here 8) together with k1, v1. So it might seem to be intentional.

Thanks!

Hi!! Thank you for releasing the code.

I tried to load the Epic-kitchens pretrained weights(ek_motionformer_224_16x4.pyth) but seems like there are some missing keys in the pytorch state_dict.

Missing Keys: {'blocks.9.attn.proj_kv.bias', 'blocks.7.attn.proj_q.weight', 'blocks.4.attn.proj_kv.weight', 'head0.weight', 'blocks.5.attn.proj_q.weight', 'patch_embed_3d.proj.weight', 'blocks.1.attn.proj_q.weight', 'head1.weight', 'blocks.10.attn.proj_q.weight', 'temp_embed', 'blocks.1.attn.proj_q.bias', 'blocks.10.attn.proj_kv.bias', 'blocks.1.attn.proj_kv.weight', 'pre_logits.fc.bias', 'blocks.5.attn.proj_q.bias', 'blocks.11.attn.proj_q.bias', 'blocks.2.attn.proj_q.weight', 'blocks.6.attn.proj_kv.weight', 'blocks.2.attn.proj_kv.bias', 'blocks.3.attn.proj_kv.bias', 'blocks.11.attn.proj_q.weight', 'blocks.10.attn.proj_q.bias', 'patch_embed_3d.proj.bias', 'blocks.8.attn.proj_kv.bias', 'blocks.3.attn.proj_q.bias', 'blocks.5.attn.proj_kv.weight', 'blocks.2.attn.proj_kv.weight', 'blocks.3.attn.proj_q.weight', 'blocks.9.attn.proj_kv.weight', 'blocks.9.attn.proj_q.weight', 'pre_logits.fc.weight', 'blocks.10.attn.proj_kv.weight', 'blocks.8.attn.proj_q.bias', 'blocks.5.attn.proj_kv.bias', 'blocks.0.attn.proj_kv.bias', 'blocks.4.attn.proj_kv.bias', 'blocks.0.attn.proj_q.bias', 'blocks.11.attn.proj_kv.weight', 'blocks.6.attn.proj_q.bias', 'head1.bias', 'blocks.3.attn.proj_kv.weight', 'blocks.7.attn.proj_kv.bias', 'blocks.8.attn.proj_q.weight', 'blocks.8.attn.proj_kv.weight', 'blocks.4.attn.proj_q.weight', 'blocks.6.attn.proj_kv.bias', 'blocks.7.attn.proj_q.bias', 'blocks.0.attn.proj_kv.weight', 'blocks.6.attn.proj_q.weight', 'blocks.4.attn.proj_q.bias', 'blocks.2.attn.proj_q.bias', 'blocks.1.attn.proj_kv.bias', 'blocks.9.attn.proj_q.bias', 'blocks.0.attn.proj_q.weight', 'head0.bias', 'blocks.11.attn.proj_kv.bias', 'blocks.7.attn.proj_kv.weight'}

Therefore, I could not reproduce the results with Epic-kitchens validation set. Could you please check the uploaded weights on Epic-kitchens?

Hi! Thanks for opensource the code. I wonder what is the size of the kinetics dataset 400 for training and validation? https://github.com/facebookresearch/SlowFast/issues/42

Hi, I just reproduced the experiment for Sthv2, with the config of motionformer_224_16x4.yaml. Specifically, I trained the model in one node with 8GPUS, which contains 32 samples in a mini-batch. The learning rate I used is 32 / 256 * 1e-4. However, the result I obtained is 64.3%, which is relatively lower than that reported in your paper. I wonder if the aforementioned learning rate is compatible with that setting since the original figure in that config is 1e-4.

This is pull request was created automatically because we noticed your project was missing a Code of Conduct file.

Code of Conduct files facilitate respectful and constructive communities by establishing expected behaviors for project contributors.

This PR was crafted with love by Facebook's Open Source Team.

This is pull request was created automatically because we noticed your project was missing a Contributing file.

CONTRIBUTING files explain how a developer can contribute to the project - which you should actively encourage.

This PR was crafted with love by Facebook's Open Source Team.

Hi. Thanks for the nice work.

I have some questions regarding RGB / BGR standards used by ssv2 and kinetics loaders in this repo. Directly stating, I think RGB / BGR standards are mishandled in the current codebase.

Specifically, SSv2 data loader initially reads frames in BGR standard (using OpenCV), however, the data-loader sometimes incorrectly applies functions that assume RGB input standards (e.g., ToPILImage). The final output is in BGR which is compatible with the pretrained ViT-B that assumes BGR standard.

On the other hand, kinetics data loader initially reads frames in RGB standard (using PyAV), however, the data-loader sometimes incorrectly applies functions that assume BGR input standards (e.g., color_jitter). The final output is in RGB which is incompatible with the pretrained ViT-B that assumes BGR standard.

I will try to point out problems in the order that the data loaders actually processes input files.

1. Kinetics loader (https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/kinetics.py)

1-1. Kinetics loader reads mp4 videos with PyAV backend, using VideoFrame.to_rgb method

reference: https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/kinetics.py#L236-L246 https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/decoder.py#L269-L280

VideoFrame.to_rgb reads mp4 frames in RGB standard

1-2. frames_augmentation is applied, which assumes BGR standards.

Specifically, contrast jitter relies on "BGR to Grayscale" transform, which is sensitive to the channel order. As a result, the augmentation is being incorrectly applied.

1-3. The final output is RGB.

Since ViT-B assumes BGR standards, the performance can be potentially sub-optimal (though we will finetune with video datasets)

2. SSv2 loader (https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/ssv2.py)

I will try to point out problems in the order that the SSv2 actually processes input files.

2-1. SSv2 loader reads jpeg frames using cv2.imdecode method.

reference: https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/ssv2.py#L246-L251 https://github.com/facebookresearch/Motionformer/blob/main/slowfast/datasets/utils.py#L41-L52

cv2.imdecode reads jpeg files in BGR standard

2-2. Frames are converted to PIL images using torchvision.transforms.ToPILImage method.

As stated in the torchvision's documentation, torchvision.transforms.ToPILImage expects RGB standard, and the currently wrong channel order would lead to potentially incorrect color augmentations. Fortunately, I guess the current RandAug profile does not include channel-order sensitive augmentations.

2-3. The final output is BGR

ViT-B also follows BGR standards, hence there is no problem outputting BGR standard frames.

Hi, may I ask the extract learning rate when setting the batch size as 64 in SSV2. Currently, it is set as 1e-4 in the provided config, while it validates the scaling rule, e.g. 32/256 * 1e-4, which has been mentioned for K400.

Hi, First of all, thanks a lot for your work and for providing a clear and documented repository associated with your paper! While reading your paper I wondered how you selected your most orthogonal subset in detail. By looking at the code, I see you provide both keys and queries to the function orthogonal_landmarks. However, it seems you do not use keys to select your subset. Is that an intended behavior?

Thanks !

Hi, Thanks for providing this wonderful model.

I'm trying to reproduce Motionformer-L on SSv2, I see that you use: https://github.com/facebookresearch/Motionformer/blob/6c860614a3b252c6163971ba20e61ea3184d5291/configs/SSV2/motionformer_224_32x3.yaml#L4

But if I understand correctly this is equivalent to BATCH_SIZE=8, could you please clarify?

Thanks, Elad.