Hello, I would like to ask which code base do you use for i3d feature extraction and optical flow feature extraction mentioned in the data set paper? I want to reproduce it and then test my video.

1.How to test highlight_test_release.jsonl text and How to output a prediction like this.

2.Can you explain the metrics that appear in the validation set? MR-long-mAP", "MR-middle-mAP", "MR-short-mAP", "HL-min-Fair-mAP", "HL-min-Fair-Hit1", "HL-min-Good-mAP"

Your paper says "Visual and audio features are temporally aligned at clip level". For example, in the YouTube Highlight dataset, the video is divided into clips every 100 frames, and the overlap is 50%. I extract audio features through the codebase you provided. How to align the audio and video at the clip level? How did you do it?

If the size of video feature is [14, 2048], i need to extract the audio feature which size is [14, 2048].

Follow you, I use the PANN_inference project to extract audio feature from raw wave file. Because of video clips and overlap operation, the first dimension of video feature is 14. How to align the audio feature and video feature?

I found that the size of audio feature is related to the sample rate , window size, hop size and anymore, what i should set the parameter. I want to know more details about how to extract the audio feature, thank you.

if (num_gt := sum(label)) == 0:
           ^

SyntaxError: invalid syntax

I modified the code as follow:

num_gt = sum(label)##add

if num_gt == 0:
    print("????????")
    collected.append(0)
    continue

but The map for each evaluation are different. WHY?

Hello, I am very interested in your research. In the evaluation method, after you sort the predicted scores, you do not use them any more, but only use the real labels corresponding to the sorted scores for prediction. Why? And there is no detailed explanation of the mAP metric method in the paper. Is there any reference?

Hi, first of all, thanks for your great work!

I plan to perform the experiments on charadesSTA dataset with the features you provide, but I notice that there is no text embedding files, although other features (videos, optical flows, audio) are available.

Can you provide the text embedding you used for your experiments?

Also, I have some minor questions about data preprocessing. Similar to the Issue in https://github.com/TencentARC/UMT/issues/29#issue-1410050684, I found the length of optical flow features is different from that of video features.

Did you simply crop the feature using a shorter length, same as the audio features?

Thanks,

Hello, questions about text feature extraction。 1、Is the model loaded when using CLIP to extract text features VIT-B /32? 2、Is the text input when extracting text features using CLIP the value of "query" in the "highligiht_train_release.jsonl" file?

Hello, can this method retrieve a video in real time?

The paper says that "On YouTube Highlights and TVSum, we obtain clip- level visual features using an I3D [4] pre-trained on Kinetics 400 [13] ”, which means that if an unknown video is verified, should audio and video features be extracted offline separately? How to retrieve the highlighted part of a video in real time?

Thanks a lot.

I have a lot of questions？ 1、I want to make a data set similar to QVHighlights in my research direction, What do I need to do?

2、What method was used for feature extraction of QVHIGHLIGHTS text?

THank you for the great work.

However, when I use the features provided by this repo, some video and audio features are misaligned in their context length.

Example is attached below. It is described in the order of "vid, video shape, audio shape". B3yOejNbNks_210.0_360.0 torch.Size([71, 2816]) torch.Size([70, 2048])

Can you provide how to align these features?

Thank you. Best regards