In your misc/extract_feats_roi.py code, the return value of functions of coco_demo.run_on_opencv_image() are result, top_preds and top_roi_feats.

But, in the maskrcnn code you linked, only one value (result) is returned as shown in the below.

How do I get the roi_feat for custom video data?

Hi, I used a Titan XP to train or test your code, but I meet the error of 'cuda out of memory', so how can I use multi-gpu to train your code or how can I switch the dataset to MSVD?

Hi, When you calculate the test loss, it uses pred to calculate the test loss. Like this: pred, gt_seq, gt_logseq, _, _, _ = model(feats, bfeats, labels, labels_svo) But in the forward function of the model, it uses: lan_cont = self.embed(torch.cat((svo_it[:,1:2], it.unsqueeze(1)), 1)) So, why here using the ground truth, that is it (but 'it' comes from the label), to get the test loss? Why don't use the predicted last word to calculate the wholely predicted result then to get the loss, because, in the test phase, we don't see the label until the time to calculate the loss?

Thank you very much!

Hello amigo! I have a question about feature extraction for 3D motion features. In the paper Syntax-Aware Action Targeting for Video Captioning, the author declares that he used C3D for motion features extraction. However, I am a little bit confused about the difference between '3D-ResNets' and 'C3D'. Are these two the same thing? Looking forward to your reply!

1. msrvtt_roi_feat.h5
2. msrvtt_roi_box.h5

this two h5 files have 10000 datasets respectively。 you just have 6513 videos ，but why you have 10000 datasets each file ？ what is the meaning of each dataset ? is it for frames ？

Hello！I'm trying to use your project code to test my own video. Thanks for providing the code to extract the features. We import the models in 3D-ResNets-Pytorch demo you linked into the misc folder, but we have the following problems when extracting 3D features:

Traceback (most recent call last): File "./misc/extract_feats_motion.py", line 82, in model, _ = generate_model(opt) File "/media/louyu/DATA/SAAT/misc/model.py", line 51, in generate_model n_classes=opt.n_classes) File "/media/louyu/DATA/SAAT/misc/models/resnext.py", line 66, in generate_model **kwargs) File "/media/louyu/DATA/SAAT/misc/models/resnext.py", line 53, in init shortcut_type, n_classes) File "/media/louyu/DATA/SAAT/misc/models/resnet.py", line 132, in init shortcut_type) File "/media/louyu/DATA/SAAT/misc/models/resnet.py", line 189, in _make_layer downsample=downsample)) TypeError: init() got an unexpected keyword argument 'in_planes'

I wonder how to solve this problem. Thanks!

Hi,

Thanks for the paper and the code. I am wondering that when you used IRV2 to extract 2D frame features and selected 28 uniformly-spaced frames each video how do you aggregate these 28 feature vectors into one vector per video? Concatenate and project? Seemingly it is not clear in the paper.

Thank you!

Hi, when i run this code, after each epoch, it generated the new checkpoint. But in some epoch, there is a best score. Do i need to early stop or just run 200 epoches to get it's final checkpoint. Do the final checkpoint is the best performance checkpoint?

I notice that you use the original feature given by cst in your paper. And you use c3d in h5 format and mean pooling is used to get video level feature. But in your misc folder, you extract features in .npy format. So how to use the 2d and 3d features extracted by ./misc code in SAAT? I wonder wheather many codes should be changed to use the new feature and how to use it?

When I run make -f Makefile_msrvtt_svo train GID=0 EXP_NAME=xe FEATS="irv2 c3d category" BFEATS="roi_feat roi_box" USE_RL=0 CST=0 USE_MIXER=0 SCB_CAPTIONS=0 LOGLEVEL=DEBUG MAX_EPOCH=100 LAMBDA=20

I got make: Nothing to be done for 'train'. What happend ?

When I want to train on MSVD, I got: make: *** No rule to make target 'output/feature/msrvtt_train_resnet_mp1.h5', needed by 'output/model_svo/xe/resnetc3dcategory_msrvtt_concat_CIDEr_64_0.0001_20.pth'. Stop. Can you upload Makefile_msvd_svo ?

The code generates h5 files for which each video has some wierd keys (corresponding to their urls like "HzYtvOYOEoU_21_32"). But SAAT expects keys to be of integer form like 0, 1, 2 and so on. The easiest mapping would have been obtained by replacing vid with cnt here, which means keys (as expected by SAAT) is the index of video while enumerating.

Could you please confirm if this was the actual mapping that is supposed to be used? It is a bit risky to assume anything on my own for key mappings,

when I run ./misc/extract_feats_2D.py, the following error appeared：

Traceback (most recent call last): File "E:/code/video_captioning/SAAT/misc/extract_feats_2D.py", line 7, in from resnet import resnet101 ModuleNotFoundError: No module named 'resnet'

Hey @SydCaption Many thanks for open-sourcing such awesome work !!! 👍

I am unable to understand the steps for inferencing using your pre-trained models on my own dataset. Can you please provide some help here? Many thanks!