Hi Filip,

Thank you for making your code and dataset available! It's really good.

I have a one quick question. I tested "R-[37]-GEM+DFS" approach on the both ROxf and RPar datasets using the below git repository code and realized that these performance worked better than the performance you reported in your Revisiting paper published at CVPR 18.

https://github.com/ducha-aiki/manifold-diffusion

Here are the evaluation results.

Plain

roxford5k: mAP E: 84.81, M: 64.67, H: 38.47 roxford5k: mP@k[ 1 5 10] E: [97.06 92.06 86.49], M: [97.14 90.67 84.67], H: [81.43 63. 53. ] Diffusion cg roxford5k: mAP E: 88.36, M: 74.4, H: 50.57 roxford5k: mP@k[ 1 5 10] E: [95.59 93.82 91.39], M: [95.71 92.86 88.14], H: [88.57 70.29 61.29] Diffusion trunkated roxford5k: mAP E: 88.18, M: 74.33, H: 50.7 roxford5k: mP@k[ 1 5 10] E: [95.59 93.82 91.39], M: [95.71 92.86 88.14], H: [88.57 70.29 61.29] Spectral R=2000 roxford5k: mAP E: 86.5, M: 72.0, H: 45.7 roxford5k: mP@k[ 1 5 10] E: [94.12 92.65 91.3 ], M: [94.29 91.14 87.86], H: [81.43 72.43 61.29]

Plain

rparis6k: mAP E: 92.12, M: 77.2, H: 56.32 rparis6k: mP@k[ 1 5 10] E: [100. 97.14 96.14], M: [100. 98.86 98.14], H: [94.29 90.29 89.14] Diffusion cg rparis6k: mAP E: 94.72, M: 89.63, H: 80.01 rparis6k: mP@k[ 1 5 10] E: [100. 97.43 95.86], M: [100. 99.43 98.57], H: [95.71 91.14 91.43] Diffusion trunkated rparis6k: mAP E: 94.73, M: 89.64, H: 80.03 rparis6k: mP@k[ 1 5 10] E: [100. 97.43 95.86], M: [100. 99.43 98.57], H: [95.71 91.14 91.29] Spectral R=2000 rparis6k: mAP E: 93.61, M: 85.42, H: 73.31 rparis6k: mP@k[ 1 5 10] E: [98.57 96.29 95.95], M: [100. 98.86 98.86], H: [92.86 90.86 91.71]

Could you please let me know why the results are difference between the results you reported and the results that I got using the code?

Thanks you.

Best, Jangwon

Appreciate in making this dataset available.

I was wondering if you plan to release the other evaluations codebase to the community. Currently, I see that R-[37]-GeM is released. Is there any plan to release other?

If not, perhaps, can you do release DELF-ASMK*+SP, HesAff-rSIFT-ASMK*+SP, R-[10]-R-MAC+DFS?

Hello @filipradenovic,

Thank you for providing the tool for accessing Oxford5k, Paris6k, and the revisited ones. However, there is a bug in loading the distractor in the example file 'example_process_distractor.py'.

# Error
Traceback (most recent call last):
  File "example_process_distractors.py", line 42, in <module>
    cfg = configdataset(distractors_dataset, os.path.join(data_root, 'datasets'))
  File "/store/tsunyi/revisitop-master/python/dataset.py", line 11, in configdataset
    raise ValueError('Unknown dataset: {}!'.format(dataset))
ValueError: Unknown dataset: revisitop1m!

It is because 'configdataset' is adopted for getting the names.

config file for the dataset
cfg = configdataset(distractors_dataset, os.path.join(data_root, 'datasets'))

for i in np.arange(cfg['n']):
    im = pil_loader(cfg['im_fname'](cfg, i))
    ##------------------------------------------------------
    ## Perform image processing here, eg, feature extraction
    ##------------------------------------------------------
    print('>> {}: Processing image {}'.format(distractors_dataset, i+1))

However, revisitop1m does not share the same thing as ROxford5k, RParis6k, and it does not need the label as well.

Therefore I provide an alternative to load them.

import glob
from tqdm import tqdm

img_list = glob.glob('../data/datasets/revisitop1m/**/*.jpg', recursive=True)

for filename in tqdm(img_list):

    im = pil_loader(filename)
    ##------------------------------------------------------
    ## Perform image processing here, eg, feature extraction
    ##------------------------------------------------------

Thank you again for such an amazing project.

I can see that he has not updated the datasets for a long time. But, where is the pre-trained network mat file. I have checked every link and cannot find the file. When I loaded the file in the ftp folder. it does not contain X and Q variables. Where are they? any help is appreciated.

Hi Filip, I am new to these datasets and hence the confusion. Usually we have training data with landmark ids , their GT (positive samples) and then separate query images and corresponding positive samples for evaluation.

In these datasets like Oxford5k or ROxford5k, I find the landmark images, and other images for that landmark. For ex, everything starting with all_souls correspond to the all_souls building. But when I see other images containign this tag, it contain people and indoor images which are possibly junk. In the gt files, I see the structure having, all_souls_1_query.txt, all_souls_1_good.txt, all_souls_1_ok.txt, all_souls_1_junk.txt and so on.

While training do we use all_souls_1_good.txt, all_souls_ok.txt and ignore the junk? and cannot use all_souls_1_query.txt?

I just want to make sure I get the standard practice of how to train on this dataset and evaluate properly. I looked in to the PhD thesis too which you linked in another issue, but this training protocol is I am unable to grasp.

Thanks a lot for your patience.

Hi,

I use the python file to download distractor images (1M), but the download failed at the first file. I also change the start id and try again, they all failed. Could you please help to check, or is there any other links that I can download the distractor images? Thank you very much.

@filipradenovic Hi filipradenovic, I find there is a bug when pos is empty in evaluate.py#L80. If there is no positive result in the ranking result, such as rank@10, the pos is empty, then it will throw an error. How you fix the bug? I'm on the way to try to understand the MAP in the file of evaluate.py.

Thanks.

Hi, @filipradenovic, I'm quite interested in your paper "Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking". Table 4 in your paper said DELF feature extraction plus ASMK* aggregation using GPU cost 0.41s per 1024x768 image, right? So I'm curious about the time of feature extraction and aggregation respectively. Do you remember how much time does it cost to aggregate (ASMK*) an 1024768 image using GPU and CPU? And can you provide the code of ASMK aggregation (GPU) or how can I have access to it?

Looking forward to your reply. Thank you in advance!

Thanks for making your work available.

This is a report for a possible bug in computation of AP.

Let say that I have ranks of [0, 3, 4, 5]. If I pass it to compute_ap() with nres = 4, and inspected precision_0 and precision_1 in the function, their values in the for-loop are

rank 0: (precision_0, precision_1) = (1.000, 1.000)
rank 3: (precision_0, precision_1) = (0.333, 0.500)
rank 4: (precision_0, precision_1) = (0.500, 0.600)
rank 5: (precision_0, precision_1) = (0.600, 0.667)

However, the values at rank 3 (zero-based) is odd to me, because the precision of ranks at each k is 1.0, 0.5 (2/4), 0.6 (3/5) and 0.667 (4/6). Therefore, the value of precision_0 and precision_1 must be 1.0 and 0.5 respectively at rank 3.

I could be wrong, but it seems to me that there is a bug in conpute_ap().