Thanks for your sharing!

I search the ActivityNet official code and find there is the get_ava_active_speaker_performance.py file in https://github.com/activitynet/ActivityNet/blob/master/Evaluation/get_ava_active_speaker_performance.py, which looks like to do evaluation for ava dataset. In your code you are using sklearn metric to get the mAP results right? So may I know did you use this code to do evaluation in the process of contest?

If so, could you share the format of the result csv file? My format is looks like that:

video_id,frame_timestamp,entity_box_x1,entity_box_y1,entity_box_x2,entity_box_y2,label,entity_id,score HV0H6oc4Kvs,960.0,0.6042350000000001,0.150289,0.701954,0.349711,NOT_SPEAKING,HV0H6oc4Kvs_0960_1020:1,0.11589885

and the ground truth csv file is "ava_activespeaker_val_augmented.csv" file, which you have shared.

Thanks for your time !

I am training a model from scratch and it is nearly done, however I checked and ASC_forward.py appears to be a copy of the README, do you mind updating it? I am very excited about using your work in a HRI research context but need some help with reproducing your results. Thanks for your great work!

Hi, I am attempting to set up the model for real world use and struggling to reverse engineer your dataset loader. From what I understand, the model should take 11 rgb face crops (144x144) as video_data (which also needs to be normalized), but how do you properly generate the audio input? I understand you are working with Mel Frequency Cepstral Coefficients, but can you give me some insight on why 11 frames worth of audio would match with an input shape of 13x40?

Hi, When I try to slice the audio files following step 2 before training, I am finding the directory is empty, i.e, the audio slices are not extracted at all. Can you please check once from your side?

Thank you for sharing your code.

What does instance_id mean? in csv file? (ex: vL7N_xRJKJU_1440_1500:118:0:0) I need this information to train my own data.

Thanks and looking forward for your reply.

I download the file according the ./scripts/download.sh , but 'wget https://filedn.com/l0kNCNuXuEq70c3iUHsXxJ7/active-speakers-context/ava_activespeaker_test_augment' is not exist. Can you share the test data?

Hi, Thanks for the open source code, I was wondering what time did it take to train the ASC model? Also, Can you please upload the model weight if possible . Thanks!

Hi, Thanks for this very clean and easy-to-use repo!

I was wondering, in STE_postprocessing.py on lines 49 and 50. where do "STE.csv" and "gt.csv" come from?

I have done the forward pass of the STE, and the predictions are stored in "forward_dir" (line 44)

Thanks for your help!

Hi, to capture the audio part from the clip, you are subtracting the audio_offset at the calculation of audio_start and audio_end at the L103. Is it correct way of capturing audio part? I believe, we should not subtract audio_offset.

Hi, Thanks for this fantastic work! Currently, I'm trying to replicate your results and build my own model. When I looked into the way you're dealing with the data, I find two functions in core/dataset.py called: _postprocess_speech_label and _post_process, which seems to transform SPEAKING_NOT_AUDIBLE to NOT_SPEAKING. As far as I can understand, this will change the original 3-category classification task to a 2-category classification during training. Will that influence the results and does it conform to the official guide? Maybe I'm misunderstanding something, please correct me if so. Thanks!

I find this link in the paper "Active Speakers in Context", but this project is empty. So I wonder to know when will publish the code? Thanks!

Hi,

Thanks a lot for sharing your model and your code. Your paper is great!

I intend to perform active speaker detection on a bunch of videos. I just need to apply your pre-trained models , not to train the model.

My understanding is that, in order to do active speaker detection on a video, I first have to perform the face tracking on this video and generate a csv file with similar tracking data as the ones provided with the AVA dataset. I did not find any face tracking script in the code. Could you provide this script (or tell me where it is in the code if I missed it) ? More generally could you provide all the missing scripts or give guidance and pointers to perform active speaker detection on any video ?

Thanks in advance.

In core/dataset.py#L158-L162

# random crop
width, height = video_data[0].size
f = random.uniform(0.5, 1)
i, j, h, w = RandomCrop.get_params(video_data[0], output_size=(int(height*f), int(width*f)))
video_data = [s.crop(box=(j, i, w, h)) for s in video_data]

[Source]

you pass the arguments (left, upper, width, height) into Image.crop() when it should be (left, upper, right, lower). The result is that the training boxes are smaller than intended.

PyTorch's implementation is the following

def crop(img: Image.Image, top: int, left: int, height: int, width: int) -> Image.Image:
    if not _is_pil_image(img):
        raise TypeError('img should be PIL Image. Got {}'.format(type(img)))

    return img.crop((left, top, left + width, top + height))

[Source]

with (top, left, height, width) the output of RandomCrop.get_params(). I would recommend using the following to avoid argument-conversion mistakes.

# random crop
width, height = video_data[0].size
f = random.uniform(0.5, 1)
crop_module = RandomCrop(size=(int(height*f), int(width*f)))
video_data = [crop_module.forward(img) for img in images]

Visualization (first existing code, then fixed code)

Note: the exact position of the crop should be ignored. Only the image size is relevant. Also notice how the old pictures are generally not square crops.

f == 0.98 : negligible

f == 0.52 : significant

I haven't run your code with this fix, so I don't know how much the results would improve (if at all).

Hey,

Thanks again for great and helpful repo - would you be able share the ASCPredictions.csv file that you used to get the mAP scores of 87.1? I would like to analyse some of the results please.

i am getting the following error while runing STE_train.py File "STE_train.py", line 73, in shuffle=True, num_workers=1) File "/home/zain/anaconda3/envs/asc_env/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 213, in init sampler = RandomSampler(dataset) File "/home/zain/anaconda3/envs/asc_env/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 94, in init "value, but got num_samples={}".format(self.num_samples)) ValueError: num_samples should be a positive integer value, but got num_samples=0

is it a pytorch version issue? if yes then, which pytorch version you have used in your experiment

Hi, Thank you for sharing your code and models!

I need to use you code and models on my own video data for other tasks. My own videos are the movie data, I want know how to prepare the utility csv file ava_activespeaker_val_augmented.csv as model's input?

Thanks and looking forward for your reply!