Hi Team, I would like to use both camera and sensor data on transformer. Could you please suggestion to use camera data and sensor both into transformer? I also like to compare with TRN HDD matrix (mAP).

Thanks a lot

Dear authors.

First, thank you for sharing your nice work! It helps me a lot to solve my task.

Btw, your paper states that the best performance is achieved when the number of encoder layers and heads is 5 and 4, respectively, but in your code, they are set to 64 and 8, respectively. Could you explain why?

Hi! I noticed that your code is somewhat different from TRN's in HDD's data segmentation, this maybe because frames are classified in different way in two models. TRN:

for start, end in zip(
    range(seed, target.shape[0] - self.dec_steps, self.enc_steps),
    range(seed + self.enc_steps, target.shape[0] - self.dec_steps, self.enc_steps)):
    enc_target = target[start:end]
    dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    self.inputs.append([
        session, start, end, sensor[start:end],
        enc_target, dec_target,
        ])

Yours:

for start, end in zip(
        range(seed, target.shape[0], 1),  # self.enc_steps
        range(seed + self.enc_steps, target.shape[0]-self.dec_steps, 1)):
    enc_target = target[start:end]
    # dec_target = self.get_dec_target(target[start:end + self.dec_steps])
    dec_target = target[end:end + self.dec_steps]
    distance_target, class_h_target = self.get_distance_target(target[start:end])
    if class_h_target.argmax() != 21:
        self.inputs.append([
            session, start, end, enc_target, distance_target, class_h_target, dec_target
        ])

Will the differences in data segmentation affect the model?

THUMOS14-Kinetics feature is extracted at 4 FPS in IDN. Is your experimental configuration consistent with IDN? In addition, can you provide more details about the feature extraction method?

dear author, I want to use the Repo you gave to generate my own dataset, and then I found this repo. When I generated RGB and flow features separately according to the method of this repo, I found that the RGB feature of each frame is a sequence with a length of 2048, and the flow feature is a sequence with a length of 1024. However, I found that the RGB and flow characteristics of each frame of the THUMOS14 dataset you released are 2048. So, I want to hear how you deal with this problem.

Some supplements: The output location of the rgb feature in "resnet200_anet_2016" model is "caffe.Flatten_673". The output location of the flow feature in "bn_inception_anet_2016_temporal" model is "global_pool".

Hi Thank you for sharing the dataset. But I met some problems when I used the dataset. As you said in the previous issue, the fps of the origin video is 24fps, so the actual fps of the feature is 4, but I calculate the frames and the seconds of the video found that the fps of the feature is 5... If there is any mistakes?

Hi, thanks for your amazing work. When testing the performance of OadTR, I found you seem to miss the first few frames of the video. This may be unfair compared to other methods.

Hi Team , I am using the honda dataset which has 12 classes : ["background", "intersection passing", "left turn", "right turn", "left lane change", "right lane change", "left lane branch", "right lane branch", "crosswalk passing", "railroad passing", "merge", "U-turn"]

When I am trained and getting results the mAP/AP table missing the last label (class):

background: nan intersection passing: nan left turn: na right turn: nan left lane change: nan right lane change: nan left lane branch: nan right lane branch: nan crosswalk passing: nan railroad passing: nan merge: nan

Why it doesn't show the U-turn class

Hi, I am getting this error while running the train.py: FileNotFoundError: [Errno 2] No such file or directory: 'data/THUMOS14/Feature/Anet2016_feature_v2/video_validation_0000690_rgb.npy'

Could you help me with fixing that?

I am getting the following error when I run OadTR train command.

python main.py --num_layers 3 --decoder_layers 5 --enc_layers 64 --output_dir models/en_3_decoder_5_lr_drop_1

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your
module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the 
keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` 
function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel 
module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss 
function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

Could you let me know if I am missing something while running this command ?