This is an differentiable pytorch implementation of SIFT patch descriptor.

Dmytro Mishkin

Last update: Dec 24, 2022

Related tags

Pytorch Utilities descriptor cnn pytorch sift image-matching local-features

Overview

This is an differentiable pytorch implementation of SIFT patch descriptor. It is very slow for describing one patch, but quite fast for batch. It can be used for descriptop-based learning shape of affine feature.

UPD 08/2019 : pytorch-sift is added to kornia and available by kornia.features.SIFTDescriptor

There are different implementations of the SIFT on the web. I tried to match Michal Perdoch implementation, which gives high quality features for image retrieval CVPR2009. However, on planar datasets, it is inferior to vlfeat implementation. The main difference is gaussian weighting window parameters, so I have made a vlfeat-like version too. MP version weights patch center much more (see image below, left) and additionally crops everything outside the circular region. Right is vlfeat version

descriptor_mp_mode = SIFTNet(patch_size = 65,
                        sigma_type= 'hesamp',
                        masktype='CircularGauss')

descriptor_vlfeat_mode = SIFTNet(patch_size = 65,
                        sigma_type= 'vlfeat',
                        masktype='Gauss')

Results:

OPENCV-SIFT - mAP 
   Easy     Hard      Tough     mean
-------  -------  ---------  -------
0.47788  0.20997  0.0967711  0.26154

VLFeat-SIFT - mAP 
    Easy      Hard      Tough      mean
--------  --------  ---------  --------
0.466584  0.203966  0.0935743  0.254708

PYTORCH-SIFT-VLFEAT-65 - mAP 
    Easy      Hard      Tough      mean
--------  --------  ---------  --------
0.472563  0.202458  0.0910371  0.255353

NUMPY-SIFT-VLFEAT-65 - mAP 
    Easy      Hard      Tough      mean
--------  --------  ---------  --------
0.449431  0.197918  0.0905395  0.245963

PYTORCH-SIFT-MP-65 - mAP 
    Easy      Hard      Tough      mean
--------  --------  ---------  --------
0.430887  0.184834  0.0832707  0.232997

NUMPY-SIFT-MP-65 - mAP 
    Easy     Hard      Tough      mean
--------  -------  ---------  --------
0.417296  0.18114  0.0820582  0.226832

Speed:

0.00246 s per 65x65 patch - numpy SIFT
0.00028 s per 65x65 patch - C++ SIFT
0.00074 s per 65x65 patch - CPU, 256 patches per batch
0.00038 s per 65x65 patch - GPU (GM940, mobile), 256 patches per batch
0.00038 s per 65x65 patch - GPU (GM940, mobile), 256 patches per batch

If you use this code for academic purposes, please cite the following paper:

@InProceedings{AffNet2018,
    title = {Repeatability Is Not Enough: Learning Affine Regions via Discriminability},
    author = {Dmytro Mishkin, Filip Radenovic, Jiri Matas},
    booktitle = {Proceedings of ECCV},
    year = 2018,
    month = sep
}

Comments

The shape of input image can only be square?

Hi, this is a very very nice work, but it seems that the shape of input image can only be square? I mean the input image should be h x h, can it be h x w (h!=w)?

opened by coolbeam 0

This is an differentiable pytorch implementation of SIFT patch descriptor.

Related tags

Overview

You might also like...

PyTorch implementation of TabNet paper : https://arxiv.org/pdf/1908.07442.pdf

An implementation of Performer, a linear attention-based transformer, in Pytorch

PyTorch implementation of Glow, Generative Flow with Invertible 1x1 Convolutions

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

Tacotron 2 - PyTorch implementation with faster-than-realtime inference

A Pytorch Implementation for Compact Bilinear Pooling.

A pure Python implementation of Compact Bilinear Pooling and Count Sketch for PyTorch.

Pytorch implementation of Distributed Proximal Policy Optimization

A PyTorch implementation of L-BFGS.

Comments

The shape of input image can only be square?

Owner

Dmytro Mishkin

Differentiable ODE solvers with full GPU support and O(1)-memory backpropagation.

Differentiable SDE solvers with GPU support and efficient sensitivity analysis.

OptNet: Differentiable Optimization as a Layer in Neural Networks

Unofficial PyTorch implementation of DeepMind's Perceiver IO with PyTorch Lightning scripts for distributed training

Tez is a super-simple and lightweight Trainer for PyTorch. It also comes with many utils that you can use to tackle over 90% of deep learning projects in PyTorch.

ONNX Runtime for PyTorch accelerates PyTorch model training using ONNX Runtime.

A lightweight wrapper for PyTorch that provides a simple declarative API for context switching between devices, distributed modes, mixed-precision, and PyTorch extensions.

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

PyTorch framework A simple and complete framework for PyTorch, providing a variety of data loading and simple task solutions that are easy to extend and migrate

A PyTorch implementation of EfficientNet