Hi @rohitgirdhar,

Thanks for your great work -- I found it very interesting and plan to use it in my work! I was hoping to clear up exactly how the loss functions are working with the feature decoding since I was a little confused:

The decoder at each timestep from 1..t outputs features (in causal manner), which are then passed through a linear layer to obtain predicted frame features. Another linear layer on top of this then predicts distribution over action classes. Thus, we have t action predictions. Does the predictions for timestep 1 use the action label from timestep 2? The predictions for timestep t from my understanding represent the action at timestep t+1 (next action we want to anticipate). Based on the implementation, I was wondering if the classification loss also does a loss based on predictions for the next frame's labels and the first frame label is not used? Sorry if this is confusing, hope you can help clear my understanding!

Hi @rohitgirdhar, I'm trying to reproduce the experiment '01_ek100_avt.txt'. After training, I read my evaluation results using your example code in README.md, which from ’notebooks.utils‘. And I got these outputs:

[('expts/01_ek100_avt.txt', 0)] Accuracies verb/noun/action: 32.3 77.3 22.3 51.8 13.6 32.8
[('expts/01_ek100_avt.txt', 0)] Mean class top-1 accuracies verb/noun/action: 6.0 7.9 1.5
[('expts/01_ek100_avt.txt', 0)] Recall@5 verb/noun/action: 22.3 28.7 12.0
[('expts/01_ek100_avt.txt', 0)] Recall@5 many shot verb/noun/action: nan nan 12.0
[('expts/01_ek100_avt.txt', 0)] Recall@5 tail verb/noun/action: 13.9 19.3 8.9
[('expts/01_ek100_avt.txt', 0)] Recall@5 unseen verb/noun/action: 27.5 26.0 12.6

My reproduction result is 12.0, lower than your report result 14.9. Am I doing the right evaluation process？

And the only thing I changed in the training is the interface of reading data. Could you please help me to check whether it could be the reason I fail the reproduction? The alteration is as below.

I wrote a PictureReader to read frames in iteration replace of the DefaultReader:

class PictureReader(Reader):
    def forward(self, video_path, start, end, fps, df_row, **kwargs):
        del df_row
        start = int(max(start * fps, 1))
        end = int(end * fps)
        if end <= start: return torch.Tensor()
        
        video_path = video_path[:-4]
        video = []
        for i in range(start, end):
            picture = torchvision.io.read_image(video_path+'/frame_{:010d}.jpg'.format(i))
            video.append(picture.permute(1,2,0))
        return torch.stack(video), {}, {}

As for the get_frame_rate(), I got fps from the annotation df. So I add a line in _init_ of class EPICKitchens. df['fps'] = df['start_frame'] / df['start']

Hi @rohitgirdhar, I'm trying to test the irCSN-152 (IG65M) model for EK-55. I used the model https://dl.fbaipublicfiles.com/avt/checkpoints/expts/10_ek55_avt_ig65m.txt/0/checkpoint.pth and the config expts/10_ek55_avt_ig65m.txt, and added these lines to the config:

test_only=true
train.init_from_model=[[${cwd}/DATA/models/10_ek55_avt_ig65m.pth]]

However, I'm getting

[2021-10-05 12:37:04,999][root][INFO] - Reading from resfiles
[2021-10-05 12:37:11,072][func.train][INFO] - []
[2021-10-05 12:37:11,073][root][INFO] - iter_time: 0.294328
[2021-10-05 12:37:11,073][root][INFO] - data_time: 0.135377
[2021-10-05 12:37:11,074][root][INFO] - loss: 6.164686
[2021-10-05 12:37:11,074][root][INFO] - acc1/action: 7.351763
[2021-10-05 12:37:11,074][root][INFO] - acc5/action: 19.931891
[2021-10-05 12:37:11,074][root][INFO] - cls_action: 6.134162
[2021-10-05 12:37:11,074][root][INFO] - feat: 0.030524

which is far from the 14.4 and 31.7 Top 1/5 performance. Do you know what might be wrong here?

Hi, I am trying to run the code for ek100 dataset. But i notice that it is not able to compute any video clips for dataset creation on as here https://github.com/facebookresearch/AVT/blob/2d6781d5315a4c53bd059b1cd11ee46bd4427648/datasets/data.py#L41

The _dataset variable is EpicKitchen class object since the _target variable defined here is of the same type. https://github.com/facebookresearch/AVT/blob/2d6781d5315a4c53bd059b1cd11ee46bd4427648/conf/dataset/epic_kitchens100/anticipation_train.yaml#L3

Therefore it is not able to execute this line. https://github.com/facebookresearch/AVT/blob/2d6781d5315a4c53bd059b1cd11ee46bd4427648/datasets/data.py#L46 and it execute the except line after this.

Can you help me in this regard. Maybe I am doing something wrong ! Thanks :)

Hi,

Your work is so impressive. Actually, I am a beginner in this field. I try to use the env.yaml file that you provide to create a conda env through conda env create -f env.yaml python=3.7.7. But it reports that found conflicts. Then what should I do?

Thank you in advance!

Hi again :) I have had some wonders regarding the frame sampling method. Would be highly appreciated to clear that out... :) Looking at expt. 09_ek55_avt settings, the following parameters are:

• tau_o: 20sec
• original fps (of EK): 60
• req_fps: 1
• frames_per_clip: 10
• sampling_strategy: "last_clip"

As I see it, in practice out of a 20-second input clip (1200 frames), only 10 are sampled, where each frame represents 1-second sub-clip. As a result (due to req_fps=1 and sampling_strategy='last_clip') the model only looks at the last 10 seconds. Is that correct? If so, what is the actual roll of tau_o?

Thank you very much!

Hi there! I was wondering what is the difference between the ViT ckpt file you supplied instead of simply loading timm.vit_base_patch16_224 with using timm.create_model with the pretrained=True flag. Basically they were both pretrained on IN1k. Am I correct? Thanks!

Hi,

I'm wondering if you have any results regarding performance without data augmentation and end-to-end training? If not, do you know how much augmentations helped in terms of overfitting and performance? Currently, I'm trying to run longer time-horizon experiments without augmentations, but it doesn't seem to perform much better than 2 second horizon experiments. Thank you!

Hi,

I'm trying to re-implement the model from the paper, and for a time horizon of 2 seconds I'm able to reach the same recall, however, increasing that time horizon didn't result in an increase of accuracy or recall. I noticed that here: https://github.com/facebookresearch/AVT/blob/b372773a2fd75e295da3ab737111363b4c860546/models/future_prediction.py#L71, there's this block of code on kmeans/centroids that is not described in the paper, so I'm unsure if this is the insight that I'm missing for longer horizons, or not. Would you be able to help shine some light on what this code is doing? Additionally, do you have any tips on how to see an increase in accuracy / recall when increasing horizon time?

Hi, I was just wondering how exactly the object features are used in the model? At each timestep, does the model consider both image features and object features concatenated? Are you familiar with how to extract these object features from the raw frames (ex: will it work at a lower resolution image)? Thanks!

Hi, the official epic-kitchen provide both the rgb-frame and raw videso source, i want to ask, can i train your model in the rgb-frames, since the raw videos are too large to download for me. if yes, would it lead to lower performance?

thanks!

Hi @rohitgirdhar , Thank you for great idea in video anticipation.

I'm reproducing your work with Breakfast/50salad data, and faced with 2 problem.

1.Do I have to locate breakfast video and 50 salad video at same folder?

2. While training with 50 Salad, ffmpeg returning some error : marker does not match f_code (image below). Although with this error, training goes well but how can i handle this error? I've changed avi with mp4 file, or resizing it, but not worked. Is this because of codec problem?

Thank you

Hi, thanks for your great project! I am wondering the input/output feature dimensions of Transformer Encoder. The description in Section 4.1 of the paper shows the input/output feature dimensions are both 768D, is it right? However, the description in Section 4.4 of the paper shows the input feature dimension of Causal Transformer Decoder is 2048D, what is the output feature dimension of Causal Transformer Decoder? And is there a dimension conversion (768D->2048D) before using Causal Transformer Decoder?

Hi, thanks for repo! Excuse me, could you please tell how to perform long-term anticipation as showed in the paper? Have I missed this part in code, or it's not implemented here yet?

We can only see the accuracy of validation set, when we run python launch.py -c expts/02_ek100_avt_tsn.txt -l -t.

Can you tell us how to generate the submitted file for the challenge ek100?

How can I train/test with video data?

When I run python launch.py -c expts/09_ek55_avt.txt -t -g, I obtain the warning "No video_clips present":

[2022-02-08 17:15:53,778][func.train][INFO] - Computing clips...
[2022-02-08 17:15:53,779][func.train][WARNING] - No video_clips present
[2022-02-08 17:15:53,779][func.train][INFO] - Created dataset with 23430 elts

I have downloaded and cropped the videos and setup the same folder structure as in the README file.

Executing hasattr(_dataset, 'video_clips') results in False. How to add video_clips to _dataset to properly execute compute_clips in datasets/data.py?