In this toolbox, homographies on hpatches are evaluated with accuracy, while recent papers such as LoFTR and SuperGlue both use AUC, which makes comparisons difficult.

It further complicates matters that pydegensac is used here, while the opencv version is used in most papers. Adding the option of calculating AUC and opencv would make comparison easier. See e.g. https://github.com/PruneTruong/DenseMatching/blob/40c29a6b5c35e86b9509e65ab0cd12553d998e5f/validation/utils.py#L209 for an AUC implementation.

Hi @GrumpyZhou Thanks for your great works!

I use patch2pix + superglue as matcher for aachen evaluation, and use eval_aachen.py script. I can get keypoints.h5 and matches.h5, but an error was encountered while running triangulation, the error as follows: "Keypoint format not supported"

I find the format slightly different from the hloc, any suggestion? Did you test patch2pix + superglue on aachen ? Thanks!

Hi~ I'm trying to evaluate my method on robotcar with your amazing image-matching-toolbox. I notice that there should be a queries folder under RobotCar dataset, which is missing from the dataset download page https://data.ciirc.cvut.cz/public/projects/2020VisualLocalization/RobotCar-Seasons/ . Would you please provide the download link of this folder?

RuntimeError: Tried to access nonexistent attribute or method 'call' of type 'torch.kornia.geometry.boxes.Boxes3D'. Did you forget to initialize an attribute in init()?: File "/home/szw/anaconda3/envs/image-matching-toolbox-main/lib/python3.7/site-packages/kornia/geometry/boxes.py", line 601 # Due to some torch.jit.script bug (at least <= 1.9), you need to pass all arguments to init when # constructing the class from inside of a method. return cls(hexahedrons, raise_if_not_floating_point=False, mode=mode) ~~~ <--- HERE

Thanks for this great project. According to the installation doc the toolbox should be install using ssh, which is impossible for users interested to add ssh-key to this project. And offcourse, nonsense. and I have checked the .gitmodules file [submodule "third_party/SuperGluePretrainedNetwork"] path = third_party/superglue url = [email protected]:magicleap/SuperGluePretrainedNetwork.git [submodule "third_party/caps"] path = third_party/caps url = [email protected]:GrumpyZhou/caps.git [submodule "third_party/d2-net"] path = third_party/d2net url = [email protected]:mihaidusmanu/d2-net.git [submodule "third_party/r2d2"] path = third_party/r2d2 url = https://github.com/naver/r2d2 ignore = untracked [submodule "third_party/sparse-ncnet"] path = third_party/sparsencnet url = [email protected]:ignacio-rocco/sparse-ncnet.git [submodule "third_party/hloc"] path = third_party/hloc url = [email protected]:GrumpyZhou/Hierarchical-Localization.git branch = extend_base_dev would you change the url to https, so we can test and use this great project.

I copied the loftr code to add the required files, but when running my code failed: KeyError: 'block_type' Self. block_type = config['block_type'], which is the argument in my code

I am unable to execute the quick testing code shown in readme.md I've imported all the files as a PyCharm project, then I installed all the required packages using Conda, I created the following file to execute the image-matching-toolbox, but I have this error... (in addition, there is NO file plot_matches.py inside immatch.utils)

Waiting your answer, thank you in advance

Thank you very much for this toolbox.

Here I would like to ask the question of how to set the evaluation for Aachen Night only with the default retrieved image list? Could you please guide how to do this? I am very new to the system.

bash download.sh： requests.exceptions.ConnectionError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=1T2c3DHJlTdRpo_Y3Ad4AH8p4tS7cOyjP (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0d00812cd0>: Failed to establish a new connection: [Errno 101] Network is unreachable')) unzip: cannot find or open pairs.zip, pairs.zip.zip or pairs.zip.ZIP.

Hi!

Thank you so much for releasing this toolbox and for the awesome paper on Patch2Pix. Here, I would like to kindly ask for your help.

I have tried to run your example for Patch2Pix and another one of SuperPoint + NCNet ...

The following code lines are in ./test_patch3pix.sh...

python -m immatch.eval_hpatches --gpu 0 \
    --config 'patch2pix' --match_thres 0.25 0.5 0.9  \
    --task 'both' --save_npy \
    --root_dir . 

However, I keep getting following error ValueError: zero-size array to reduction operation minimum which has no identity ... I am not sure how to fix it. So, please give the guidance...

Below here is the full error report.

Also, I have captured the screen to confirm at the bottom to confirm

(immatch) gabby-suwichaya@gabby-suwichaya:/mnt/HDD4TB1/image-matching-toolbox$ ./test_patch3pix.sh Can not import sparsencnet

>>>> Method=Patch2Pix Default config: {'class': 'Patch2Pix', 'ckpt': 'pretrained/patch2pix/patch2pix_pretrained.pth', 'ksize': 2, 'imsize': 1024, 'match_threshold': 0.25} Thres: [0.25, 0.5, 0.9]

Load model method:patch2pix 
Ckpt:pretrained/patch2pix/patch2pix_pretrained.pth
Initialize Patch2Pix: backbone=ResNet34 cstride=True upsample=8
Init regressor Namespace(conv_dims=[512, 512], conv_kers=[3, 3], conv_strs=[2, 1], fc_dims=[512, 256], feat_comb='pre', feat_dim=259, panc=1, pshift=8, psize=[16, 16], shared=False)
FeatRegressNet:  feat_comb:pre psize:16 out:5 feat_dim:518 conv_kers:[3, 3] conv_dims:[512, 512] conv_str:[2, 1] 
FeatRegressNet:  feat_comb:pre psize:16 out:5 feat_dim:518 conv_kers:[3, 3] conv_dims:[512, 512] conv_str:[2, 1] 
Xavier initialize all model parameters
Initialize ResNet using pretrained model from https://download.pytorch.org/models/resnet34-333f7ec4.pth
Reload all model parameters from weights dict
Initialize Patch2Pix
Matching thres: 0.25  Save to: ./outputs/hpatches/cache/Patch2Pix.im1024.m0.25.npy

>>Eval hpatches: method=Patch2Pix rthres=2 thres=[1, 3, 5, 10] 
Save results to ./outputs/hpatches/cache/Patch2Pix.im1024.m0.25.npy
/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3441: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
  File "/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/HDD4TB1/image-matching-toolbox/immatch/eval_hpatches.py", line 98, in <module>
    print_out=args.print_out
  File "/mnt/HDD4TB1/image-matching-toolbox/immatch/eval_hpatches.py", line 74, in eval_hpatches
    save_npy=result_npy
  File "/mnt/HDD4TB1/image-matching-toolbox/immatch/utils/hpatches_helper.py", line 323, in eval_hpatches
    lprint_(eval_summary(results, thres, save_npy=save_npy))
  File "/mnt/HDD4TB1/image-matching-toolbox/immatch/utils/hpatches_helper.py", line 28, in eval_summary
    summary += '# Features: mean={:.0f} min={:d} max={:d}\n'.format(np.mean(n_feats), np.min(n_feats), np.max(n_feats))
  File "<__array_function__ internals>", line 6, in amin
  File "/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2880, in amin
    keepdims=keepdims, initial=initial, where=where)
  File "/home/gabby-suwichaya/anaconda3/envs/immatch/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

exec the command according to the evaulation.md

python -m immatch.eval_hpatches --gpu 0 \ 
    --config  'cotr' \
    --task 'both' --save_npy \
    --root_dir .

and output:

>>>> Method=COTR Default config: {'class': 'COTR', 'ckpt': 'pretrained/cotr/cotr_default.pth.tar', 'backbone': 'resnet50', 'hidden_dim': 256, 'dilation': False, 'dropout': 0.1, 'nheads': 8, 'layer': 'layer3', 'backbone_layer_dims': {'layer1': 256, 'layer2': 512, 'layer3': 1024, 'layer4': 2048}, 'enc_layers': 6, 'dec_layers': 6, 'position_embedding': 'lin_sine', 'max_corrs': 100, 'match_threshold': 0.0, 'imsize': -1, 'batch_size': 32} Thres: [0.0]
Matching thres: 0.0  Save to: ./outputs/hpatches/cache/COTR.npy

>>>>Eval hpatches: task=matching+homography method=COTR scale_H=False rthres=2 thres=[1, 3, 5, 10] 
>>Finished, pairs=0 match_failed=580 matches=0.0 match_time=nans
==== Image Matching ====
#Features: mean=0 min=0 max=0
#(Old)Matches: a=0, i=0, v=0
#Matches: a=0, i=0, v=0
MMA@[ 1  3  5 10] px:
a=[0. 0. 0. 0.]
i=[0. 0. 0. 0.]
v=[0. 0. 0. 0.]

==== Homography Estimation ====
Hest solver=degensac est_failed=580 ransac_thres=2 inlier_rate=0.00
Hest Correct: a=[0. 0. 0. 0.]
i=[0. 0. 0. 0.]
v=[0. 0. 0. 0.]
Hest AUC: a=[0. 0. 0. 0.]
i=[0. 0. 0. 0.]
v=[0. 0. 0. 0.]

and i set print function in match_pairs function but no infomation output

def match_pairs(self, im1_path, im2_path, queries_im1=None):
        im1 = imageio.imread(im1_path, pilmode='RGB')
        im2 = imageio.imread(im2_path, pilmode='RGB')
        engine = SparseEngine(self.model, self.batch_size, mode='tile')
        matches = engine.cotr_corr_multiscale(
            im1, im2, np.linspace(0.5, 0.0625, 4), 1,
            max_corrs=self.max_corrs, queries_a=queries_im1,
            force=True
        )

        print(matches.shape) # add print

        # Fake scores as not output by the model
        scores = np.ones(len(matches))
        kpts1 = matches[:, :2]
        kpts2 = matches[:, 2:4]
        return matches, kpts1, kpts2, scores

hi，i used image-matching-toolbox to eval loftr for inloc, config: default: &default class: 'LoFTR' ckpt: 'pretrained/loftr/outdoor_ds.ckpt' match_threshold: 0.2 imsize: 1024 no_match_upscale: False example: <<: *default match_threshold: 0.5 imsize: -1 hpatch: <<: *default imsize: 480 no_match_upscale: True inloc: <<: *default pairs: 'pairs-query-netvlad40-temporal.txt' rthres: 48 skip_matches: 20

and then get result as below:

image-matching-toolbox/immatch/utils/data_io.py:31: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

Here is the offending line https://github.com/GrumpyZhou/image-matching-toolbox/blob/main/immatch/utils/data_io.py#L31

The Resampling enum was added in Pillow 9.1.0. Prior to this, to resided as a constant at top level.

More changes related to #20 .

Running python -m immatch.eval_hpatches --config 'loftr' --task 'both' --rot_dir . --ransac_thres=3 --h_solver cv now yields

>>>>Eval hpatches: task=matching+homography method=LoFTR_outdoor_ds_noms scale_H=True rthres=3.0 thres=[1, 3, 5, 10] 
>>Finished, pairs=540 match_failed=0 matches=913.1 match_time=0.12s
==== Image Matching ====
#Features: mean=913 min=7 max=1000
#(Old)Matches: a=913, i=975, v=856
#Matches: a=913, i=975, v=856
MMA@[ 1  3  5 10] px:
a=[0.73 0.95 0.96 0.97]
i=[0.8  0.99 0.99 1.  ]
v=[0.66 0.91 0.94 0.95]

==== Homography Estimation ====
Hest solver=cv est_failed=0 ransac_thres=3.0 inlier_rate=0.93
Hest Correct: a=[0.65 0.88 0.92 0.95]
i=[0.82 0.98 0.99 1.  ]
v=[0.49 0.79 0.86 0.9 ]
Hest AUC: a=[0.4  0.67 0.76 0.85]
i=[0.56 0.81 0.88 0.94]
v=[0.25 0.53 0.65 0.77]

These numbers (at 3px) are actually higher than in the LoFTR paper. Take what you find useful from this pull request. I suppose you might want to wait with merging this until there is a reply here: https://github.com/zju3dv/LoFTR/issues/136

Hi,

There is a sample repository for the submission to IMC https://github.com/ducha-aiki/imc2021-sample-kornia-submission

If you want, I can adapt it for your toolbox. Caveat - it would be quite hard to do for the methods, not based on keypoints now.