There are some tips when evaluating the model on MUSIC-solo dataset in Stage One:

1. training for 10~15 epoch.
2. when evaluating the localization performance, select the model that aftering localization, e.g. the name of pth file is "location_cluster_net_0xx_xxxx_av_local.pth".
3. For the training of stage two, select the model after classification of stage one as pretrain, e.g. the name of pth file is "location_cluster_net_0xx_xxxx_av_class.pth"

I don't know how to use the function‘visualize’ in the tool.py as follows: def visualize(images, cams, e): #images (batchsize, h, w, 3) #cams (batchsize, 1, h, w) images = images.cpu().numpy() cams = cams.detach().cpu().numpy() for i in range(images.shape[0]): if i % 2 == 0: cor = 'cor' else: cor = 'not' cam = cams[i, 0] image = images[i] #recover image to 0-1 image = image * np.array([0.229, 0.224, 0.225]) + np.array([0.485, 0.456, 0.406]) image = np.clip(image, 0, 1) #generate heatmap cam = cam * 255 cam = cv2.applyColorMap(cam.astype(np.uint8), cv2.COLORMAP_JET) cam = cam[:, :, ::-1] / 255 plt.imsave('vis/img'+str(e)+''+str(i)+cor+'.jpg', 0.5cam+0.5image) I don't see the function called in tool.py ,and I'm so confused ,I will truly appreciate it if you can explain it for me!

As mentioned in the paper, Faster RCNN was used for detecting 15 instruments as per AudioSet-instrument dataset but MUSIC dataset contains some other instruments that are not available in MUSIC. How are the extra objects annotated?

I run training_stage_two.py with 5 epochs learning_rate 1e-4 batch_size 32, the train acc is nearly 0.875 and eva acc is nearly 0.8 then I run : match_cluster.py test_stage_two.py(use the pretrained weight aquired in training_stage_two.py) after this,I run eval.py, but the AUC is quite lower than you, which is about 0.05, and the visualize result is very inaccurate. I have trained so many times but the AUC is always so low,I cannot find out the reason, I am so sorry to bother you ,but can you give me some advice for this ?

I will appreciate it if you can contact me in my wechat(L18355043548) or my mailbox([email protected])

Dear authors,

Thank you so much for releasing the code! I encountered a problem when doing evaluation in stage one, music-exp. I found the annotation name in solotest.json is not consistent with the processed data.

For example: in solotest.json, video "1Ju4S0qeDqw", the original fps is 25pfs, so frame_116 should corresponds to around 2900th frame (116*25). In my data folder (I followed cut_video.py), it is true, but in solotest.json, it shows a frame of 3480.jpg. Based on my understanding, 3480.jpg should corresponds to 139th seconds of the video (3480/25=139.2). Other annotations are in similar situations. Can you give some tips on how to deal with it? Thanks!

When I run test_stage_two.py I need a variable path, but the "path" doesn't contain in dataloader syn_dataset.py line 86. I would like to ask about the meaning of the "path" and how can I get it.

I wonder if you could provide the timestamp of the videos that are used to synthesized Synthetic-Music Dataset. I want to test it use different audio length and different image scale. Thanks.

Could you please offer the website for downloading the MUSIC dataset because I find it too common to search the name MUSIC to get this databset. Thank you so much.

Hi, thanks for your excellent work. Now, i want to reproduce the experiment about AudioSet-instrument dataset. I find there are only video ids about train and val set. Could you share the video ids about the test set? Additionally, if convenient, could you share the AudioSet-instrument dataset to me? Looking for your help. Thanks a lot!

For our NeurIPS2021 paper, we firstly detected all the candidate objects emerged in the visual scene, then manually filter out the silent ones but only remain the sounding ones. Hence, all the annotated objects in the .json file are all the sounding ones.

Great thanks Triantafyllos for pointing out this!