Hi, @Atze00 i saved MoViNtes-A0 model to 'pth' and look through model by Netron, but the structure is a little strange. maybe there is something wrong at '_forward_impl' at class 'class MoViNet(nn.Module)'.

My code is as follows:

model = MoViNet(_C.MODEL.MoViNetA0, causal=True, pretrained=False, num_classes=num_class)
...
torch.save(model, '/path/to/*.pth')
path = '/path/to/*.pth'
model = torch.load(path, map_location='cpu')

Please let me know if i did something wrong.

It looks like there is a bug in the current causal padding of the official TF models, I have filed an issue and a pull request to fix it. I am currently waiting for feedbacks on the PR. There is the possibility to obtain the same behavior even without the fix but I'm not considering it at the current moment. Reference to the issue: https://github.com/tensorflow/models/issues/10062

Thank you for your good implementation! Could you provide more detail training code or example?

Originally posted by @zhangyuan1994511 in https://github.com/Atze00/MoViNet-pytorch/issues/2#issuecomment-842952447

Hi @Atze00

I succesfully transfer trained this repos movinets

But i want to deploy it else where into tensorrt format

So i first converted pytorch model to onnx using torch.onnx.export

I wanted to verify whether model is correctly ported . So I ran it in onnxruntime

It is throwing error like in the screenshot

Suggest any solutions

kindly help me here

let me know if any more screenshots or anything else

According to paper, "For all datasets, we train with 64 frames (except when the inference frames are fewer) at various frame-rates, and run inference with the same frame-rate". Frames of training dataset are 64. but frames of one video from my dataset may be less than 32, may be it's 8, 10,13, 17 etc. so is it ok?

Thanks in advance.

Train 'a0' model by clip=1, Tclip=8, and the acc on my custom test dataset is good, but when the model do inference for online video stream, recognition is not good as test dataset. In fact, test dataset if's from the video stream i use, so i believe their domain is same.

Online demo read one frame from video stream and concatente with previous 7 frames, then input to model, so the model do inference with 8 frames too.

code for online video stream is as follows:

if isinstance(new_frame, torch.Tensor):
      torch_inputs = torch.cat((tensor_fifo[:, :, 1:, :, :], new_frame), 2)
else:
      cvt_img = cv2.cvtColor(new_frame, cv2.COLOR_BGR2RGB)
      cvt_img = np.transpose(cvt_img, (2, 0, 1))
      cvt_img = cvt_img[:, np.newaxis, :, :][np.newaxis, :, :, :, :]
      torch_inputs = np.concatenate((tensor_fifo[:, :, 1:, :, :],
                                     cvt_img), 2)

Nit: per the paper, there should be 40 output channels in block 2 for MoViNet-A0, while this implementation currently has 24. https://github.com/Atze00/MoViNet-pytorch/blob/2ad697facd370d01b0f3a6093d38211166e594de/movinets/config.py#L62

https://arxiv.org/pdf/2103.11511.pdf

When I use the model without causalConv, I get the results that I expected. However when I set causal to True, the training/validation loss drops very suddenly and eventually turns into NaN.

This leads to very undesirable results and accuracy, whilst not having any insight into the loss. Are there any ways to solve this?

Did you define you own transforms? I can't import ToFloatTensorInZeroOne() from torchvision.transforms. And I find that type of video returned from torchvision.datasets.HMDB51() is tensor, dtype=torch.uint8, so just divide 255.0 ?

Hello, thanks for the PyTorch implementation! I noticed that the pre-trained models have very low accuracy (0.36%) on the validation set. Have the weights changed? I'm just trying the colab tutorial.

Hi!

Thanks for the work to bring this paper to PyTorch.

I was wondering how can we use MoViNet training with the stream buffer when we don't have videos with the same number of frames.

Authors of the paper claimed, in Section 4, that they used the method in Charades Dataset: "[...] Charades [53], which has variable-length videos with 157 action classes where a video can contain multiple class annotations." However, they don't specify which policy they used when training with variable-length videos. They also report results on the EPIC Kitchens Dataset which is also variable-length.

Do you have any insights into how they may have trained these models? The main issue here is how to build a batch when temporal dimensions are not the same...

Thank you!

Hi, thanks for your great work in movinet. I met a problem when testing hmdb51 videos. For example, the inference result for "brush hair" seems weird, in some frames, the result shows "brush hair", while in other frames, it shows "kick ball". Did you met this problem before? In the code, 16 frames are divided into 2 clips, each clip with 8 frames, but during test phase, the first clip's prediction is different from the second one's, and the ultimate prediction used the second. Is there anything wrong with this?

Hi, thanks for this repo. it's amazing. I am interested in analyzing the stream buffer but couldn't find it. Can someone ponit me in the right direction?

Thanks

Thanks for your Pytorch implementation for MoViNet. Could you please provide a model trained on Kinetics 400 as Table 9 in [1]. It is quite important for our future works. Looking forward to your reply.

[1] MoViNets: Mobile Video Networks for Efficient Video Recognition

Hi, trying to solve a binary classification problem with these movinets and was wondering how I would go about creating a MoVinet with some augmentation with the aim of reducing the TFLOPS required and decreasing the inference time