Hi @hassony2 ,

First of all, thank you for posting your code!

I have a small question, i'm trying to do transfer learning to the model using my own dataset, but the difference is that my input shape is much different from kinetics or ucf101, each sample in my dataset has 64 frames, each frame is 600x600 with 3 channels with 8 classes. I tried to just to finetune the last Unit3Dpy but it didn't do well, do you think i'm missing out something?

Yana

How can I change the Module to adapt it to the UCF101 dataset? If I changed the model (ie changed the out number of classes) , the pretrain weight is still work?

https://github.com/hassony2/kinetics_i3d_pytorch/blob/c2b54db2368e136abe414d24aacd508c37b333a9/src/i3dpt.py#L115

tensorflow SAME padding must ceil https://www.tensorflow.org/api_docs/python/tf/nn/pool If padding = "SAME": output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides[i])

when I ceil_mode=False, I observed

a = torch.autograd.Variable(torch.ones((1,3,16,300,150)), requires_grad=False) out = i3d(a)

torch.Size([1, 64, 8, 150, 75]) torch.Size([1, 64, 8, 75, 37]) torch.Size([1, 64, 8, 75, 37]) torch.Size([1, 192, 8, 75, 37]) torch.Size([1, 192, 8, 37, 18]) torch.Size([1, 256, 8, 37, 18]) torch.Size([1, 480, 8, 37, 18]) torch.Size([1, 480, 4, 18, 9]) torch.Size([1, 512, 4, 18, 9]) torch.Size([1, 512, 4, 18, 9]) torch.Size([1, 512, 4, 18, 9]) torch.Size([1, 528, 4, 18, 9]) torch.Size([1, 832, 4, 18, 9]) torch.Size([1, 832, 4, 18, 9]) torch.Size([1, 832, 4, 18, 9]) torch.Size([1, 1024, 4, 18, 9]) torch.Size([1, 1024, 4, 18, 9])

when I ceil_mode=True a = torch.autograd.Variable(torch.ones((1,3,16,300,150)), requires_grad=False) out = i3d(a)

torch.Size([1, 64, 8, 150, 75]) torch.Size([1, 64, 8, 75, 38]) torch.Size([1, 64, 8, 75, 38]) torch.Size([1, 192, 8, 75, 38]) torch.Size([1, 192, 8, 38, 19]) torch.Size([1, 256, 8, 38, 19]) torch.Size([1, 480, 8, 38, 19]) torch.Size([1, 480, 4, 19, 10]) torch.Size([1, 512, 4, 19, 10]) torch.Size([1, 512, 4, 19, 10]) torch.Size([1, 512, 4, 19, 10]) torch.Size([1, 528, 4, 19, 10]) torch.Size([1, 832, 4, 19, 10]) torch.Size([1, 832, 4, 19, 10]) torch.Size([1, 832, 4, 19, 10]) torch.Size([1, 1024, 4, 19, 10]) torch.Size([1, 1024, 4, 19, 10])

I think ceil_mode=True is correct

Traceback (most recent call last): File "i3d_tf_to_pt.py", line 9, in from src.i3dtf import InceptionI3d File "/home/zhuyisheng/kinetics_i3d_pytorch/src/i3dtf.py", line 32, in class Unit3Dtf(snt.AbstractModule): AttributeError: 'module' object has no attribute 'AbstractModule'

i have searched a lot of info in google about 3d conv. it is easy to understand the 3d conv in vedio segments of single channel. but i do not know 3d conv in vedio segments of rgb channel. what about 3d conv kernel shape ? 333 or input channels * 333.

could you explain it ? thank you very. !

Thanks for sharing the implementation! These are just a few small changes to achieve numerical consistency with the TF version by:

1. Modifying the padding algorithm to match Tensorflow "SAME" for Unit3Dpy and MaxPool3dTFPadding.
2. Updating the batch norm hyperparams.

Tested on PyTorch 0.4.1. Running the i3d_pt_demo.py produces:

Top 5 classes and associated probabilities:
[playing cricket]: 9.999968E-01
[playing kickball]: 1.335340E-06
[catching or throwing baseball]: 4.553116E-07
[shooting goal (soccer)]: 3.143419E-07
[catching or throwing softball]: 1.924329E-07
Top 5 classes and associated probabilities:
[playing cricket]: 9.477569E-01
[hurling (sport)]: 4.068211E-02
[playing tennis]: 4.154132E-03
[playing squash or racquetball]: 2.474060E-03
[hitting baseball]: 1.380014E-03
===== Final predictions ====
logits proba class
4.181368e+01 1.000000e+00 playing cricket
2.149398e+01 1.497148e-09 hurling (sport)
2.013410e+01 3.843058e-10 catching or throwing baseball
1.922558e+01 1.549212e-10 catching or throwing softball
1.891534e+01 1.135999e-10 hitting baseball

Thanks a lot for providing the open-source code!

Could you please provide the code for fine-tuning on HMDB51 or some other datasets such as Moments in Time or UCF101?

Highly appreciate your time and help!

Hi, Thanks a lot for providing the open-source I3D pytorch code! May I ask do you provide the pretrained model on Kinetics?

Highly appreciate your time and help!

Hello! I use your code to train UCF101 dataset, but loss does not go down.Can you show me ｈ-yperparameters when you train the model?Does this code training model achieve the effect of the paper?

Hi,

I was wondering if you have reproduced the preprocessing of the data. I tried to reproduce the preprocessing following the instruction in the official one. But I have no idea why the results could not match with the provided .npy. Here is my code:

    import numpy as np


    from PIL import Image, ImageSequence
    size = 256,256
    new_width = 224
    new_height = 224
    img = Image.open('./data/v_CricketShot_g04_c01_rgb.gif');

    def my_resize(img):
        for frame in ImageSequence.Iterator(img):
            copy_frame = frame.copy()
            copy_frame = copy_frame.convert("RGB")
            resized_frame = copy_frame.resize(size, Image.BILINEAR)
            r, g, b = resized_frame.split()
            r = np.array(r)
            rescaled_r = np.interp(r, (r.min(), r.max()), (-1, +1))
            g = np.array(g)
            rescaled_g = np.interp(g, (g.min(), g.max()), (-1, +1))
            b = np.array(b)
            rescaled_b = np.interp(b, (b.min(), b.max()), (-1, +1))
            rescaled_array = np.zeros((size[0], size[1], 3))
            rescaled_array[..., 0] = rescaled_r
            rescaled_array[..., 1] = rescaled_g
            rescaled_array[..., 2] = rescaled_b

            width, height, color = rescaled_array.shape
            left = (width - new_width)/2
            right = (width + new_width)/2
            top = (height - new_height)/2
            bottom = (height + new_height)/2
            croped_frame = rescaled_array[left:right, top:bottom, :]

            yield croped_frame

    copy_frame = my_resize(img)

    frames = np.array([np.array(frame) for frame in copy_frame])

Do you have any idea about this?

Hello,

I was wondering if you have figured out how to use I3D for higher res videos without going out of memory. I am using I3D for video object segmentation (have modified the network to give a segmentation output) and finding it hard to run all frames at 480p resolution for even a single video. My batch size is one and the max number of frames I can process without filling up a 12GB GPU are 24 or so. If I resize the video down to 224x224 I can manage to run all 100 frames in the video, but I am losing a lot of information this way and my results are not as great.

While training I by-passed the problem by taking random 224x224 crops of the full res image. But for inference I can't use this trick and hence wondering how to solve this issue. Any ideas?

Hi @hassony2,

First of all thanks for the repo.

I wanted to use the pretrained kinetics RGB model to extract features from a dataset I created. Since my application should run in real time with limited computational resources I wanted to use grayscale images, since that way I only have to process one channel. My question was whether that would cause issue with the RGB model. If so, then I will convert my dataset to colour images.

Thank you

Great work @hassony2 . It will be really good if the model weights are moved to some storage and provide a script to download the weights. So that I can again fork the repository.

Reason: Storage limit on GitHub. This repository has approximately 245 MB size, and I have to delete the fork because of the size limit.

The Module of MaxPool3dTFPadding with kernel_size=(1,3,3), stride(1,2,2) can lead to asymmetrical padding. It would influence the output feature map, as the bottom right would be usually higher than other part of the feature map.

When I try to input a all zeros tensor into I3D model pretrained on Kinetics-400, someting strange happen, I average pooling the C and T dim and min-max norm to get a picture as below. The bottom right is much higher than other parts.

By checking each layer output, I find out bottom right is usually have higher activation value than other part but not obvious, until mixed_5b block.

I don't know whether it hurt the model's performance, but at least it hurt the Interpretability.

Hi, I want to ask how can i use this code to extract features from my own video datasets. Your input of your code is .npy. However, how can i get my .npy file from my dataset? I do not find any data process code in this repo.

Thanks

It seems kinetics-600 retrained-model herekinetics-i3d is the same as kinetics-400, but i meet error. Not found error: Key RGB/inception_i3d/Conv3d_1a_7x7/batch_norm/beta not found in checkpoint.