Self-supervised Product Quantization for Deep Unsupervised Image Retrieval - ICCV2021

Last update: Dec 27, 2022

Overview

Self-supervised Product Quantization for Deep Unsupervised Image Retrieval

Pytorch implementation of SPQ
Accepted to ICCV 2021 - paper
Young Kyun Jang and Nam Ik Cho

Abstract

Supervised deep learning-based hash and vector quantization are enabling fast and large-scale image retrieval systems. By fully exploiting label annotations, they are achieving outstanding retrieval performances compared to the conventional methods. However, it is painstaking to assign labels precisely for a vast amount of training data, and also, the annotation process is error-prone. To tackle these issues, we propose the first deep unsupervised image retrieval method dubbed Self-supervised Product Quantization (SPQ) network, which is label-free and trained in a self-supervised manner. We design a Cross Quantized Contrastive learning strategy that jointly learns codewords and deep visual descriptors by comparing individually transformed images (views). Our method analyzes the image contents to extract descriptive features, allowing us to understand image representations for accurate retrieval. By conducting extensive experiments on benchmarks, we demonstrate that the proposed method yields state-of-the-art results even without supervised pretraining.

Concept

By maximizing cross-similarity between the deep descriptor of one view and the product quantized descriptor of the other, both codewords and deep descriptors are jointly trained to contain discriminative image content representations in SPQ.

An illustration of training procedure in SPQ

Training

Install requirements on your environment.

PyTorch=1.7.1
kornia=0.5.10
packaging=21.0
torchvision=0.8.2
tqdm=4.62.2

Documentation

The explanation of arguments to reproduce the models presented in our paper can be found in the args, and by simply run:

python main_SPQ.py --help

Vanilla SPQ training

We utilize CIFAR-10 provided by torchvision in this work, and if not installed, please set the --if_downlad=True.
We will provied pretrained models in the near futurue.
To obtain the retrieval results reported in our paper, you need to train the model over 2,000 epochs with default setup. In order to train the model for 32-bit and compute mAP for every 100-th epoch, please run as:

python main_SPQ.py --gpu_id=0 --batch_size=256 --N_books=8 --N_words=16 --eval_epoch=100

Citation

@inproceedings{SPQ,
  title={Self-supervised Product Quantization for Deep Unsupervised Image Retrieval},
  author={Young Kyun Jang, and Nam Ik Cho},
  booktitle={Proceedings of the International Conference on Computer Vision (ICCV)},
  year={2021}
}

Comments

To reproduce results for large-sized image datasets (FLICKR, NUS-WIDE)

Hi there, Thank you for sharing this awesome code. The performance on CIFAR is great, however, the performance on FLICKR and NUS-WIDE seems to be quite low (e.g., ~0.67 on FLICKR 32bit vs. ~0.75 reported on the paper). Do you have any suggestions about how to reproduce your results on large-sized image datasets (e.g., FLICKR, NUS-WIDE)? Any specific hyper-parameter settings? Thanks.

opened by solucky-95 6
Still can not reproduce results
Hi @youngkyunJang , I've face the similar issue as in #6 . At first, I took the following packages as my environment:

PyTorch: 1.10.1 torchvision: 0.11.2 kornia: 0.6.2

And I got mAP = 0.7673 at 2800 epochs. Then, I downgrade to following versions that exactly match your suggestions in README.md:

PyTorch: 1.7.1 torchvision: 0.8.2 kornia: 0.5.10

But Still have mAP = 0.7842 at 3800 epochs. There's still a gap with original paper. Do you have any idea?
opened by xiaosu-zhu 4

tau_q is multiplied instead of divison

Hi. Thanks for your work @youngkyunJang ! I noticed that to implement equation 1 of the paper i.e to compute the quantization vector using Soft quantization, tau_q is multiplied instead of division.

def Soft_Quantization(X, C, N_books, tau_q):
    L_word = int(C.size()[1]/N_books)
    x = T.split(X, L_word, dim=1)
    c = T.split(C, L_word, dim=1)
    for i in range(N_books):
        soft_c = F.softmax(squared_distances(x[i], c[i]) * (-tau_q), dim=-1)
        if i==0:
            Z = soft_c @ c[i]
        else:
            Z = T.cat((Z, soft_c @ c[i]), dim=1)
    return Z

Please let me know if there is a correction.

Thanks Aakash

opened by aakash-saboo 2

Code about PQN

Thanks for your great work!

I am looking for the code of this paper "Product Quantization Network for Fast Image Retrieval". I emailed the author but didn't receive a reply.

I noticed that your paper cite this paper and your network is similar with that paper. So I put my hopes on you!

opened by aruba01 1
Pretrained models

Hi, Are there any pretrained models available? I am interested in evaluating performance on some other datasets but I do not have access to resources to do a lot of training myself!

opened by Benjamin-Hansson 1
Missing LICENSE

This repository is missing a license, which means that others cannot build on this code. Can you please add a license, if your intent is for others to use the code? (e.g. MIT License). Thanks!

opened by seanbell 1
How to realize the retrieval function

I'm sorry to disturb you. At present, I have realized the training of the model. Now I want to use the model to realize the retrieval effect instead of simply calculating MAP, that is, input some images and return the index of topk images.I have no idea at present and hope to get your help

opened by Breeze-Zero 1

RuntimeError raised in function `pqDist_one` in Retrieval.py during evaluation

def pqDist_one(C, N_books, g_x, q_x):
    l1, l2 = C.shape
    L_word = int(l2/N_books)
    D_C = T.zeros((l1, N_books), dtype=T.float32)

    q_x_split = T.split(q_x, L_word, 0)
    g_x_split = np.split(g_x.cpu().data.numpy(), N_books, 1)
    C_split = T.split(C, L_word, 1)
    D_C_split = T.split(D_C, 1, 1)

    for j in range(N_books):
        for k in range(l1):
            D_C_split[j][k] = T.norm(q_x_split[j]-C_split[j][k], 2) # ERROR OCCURRENCE
        if j == 0:
            dist = D_C_split[j][g_x_split[j]]
        else:
            dist = T.add(dist, D_C_split[j][g_x_split[j]])
    Dpq = T.squeeze(dist)
    return Dpq

With out any modification during my recurrence, a runtime error occurred at the commented line above:

RuntimeError: A view was created in no_grad mode and is being modified inplace with grad mode enabled. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.

I tried to add .detach() to the end of that line:

def pqDist_one(C, N_books, g_x, q_x):
    l1, l2 = C.shape
    L_word = int(l2/N_books)
    D_C = T.zeros((l1, N_books), dtype=T.float32)

    q_x_split = T.split(q_x, L_word, 0)
    g_x_split = np.split(g_x.cpu().data.numpy(), N_books, 1)
    C_split = T.split(C, L_word, 1)
    D_C_split = T.split(D_C, 1, 1)

    for j in range(N_books):
        for k in range(l1):
            D_C_split[j][k] = T.norm(q_x_split[j]-C_split[j][k], 2).detach() # SOLVED
        if j == 0:
            dist = D_C_split[j][g_x_split[j]]
        else:
            dist = T.add(dist, D_C_split[j][g_x_split[j]])
    Dpq = T.squeeze(dist)
    return Dpq

and it worked correctly, at least for my machine.

BTW: I'm working with Pytorch-1.9 + Cuda-11.1.

opened by KARLSZP 1

Datasets for other datasets .txt

Use a different dataset. I would like to know about ./data/* DB.txt the data in the text file is composed of: image name + binary tag code? How do I make binary labels when working with my own dataset? How do I get datasetDB .txt?

opened by yayaYsmile 0
Training time

I want to know how long does it take to use cifar10 training? I use cifat10 spend too much time for training, 24 hours just to run over 200 epoch. Thank you for your reply!

opened by Lisachinasd 0

Self-supervised Product Quantization for Deep Unsupervised Image Retrieval - ICCV2021

Related tags

Overview

Self-supervised Product Quantization for Deep Unsupervised Image Retrieval

Abstract

Concept

An illustration of training procedure in SPQ

Training

Install requirements on your environment.

Documentation

Vanilla SPQ training

Citation

Comments

Owner

Young Kyun Jang

DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

Quantization library for PyTorch. Support low-precision and mixed-precision quantization, with hardware implementation through TVM.

Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation. In CVPR 2022.

Optimal space decomposition based-product quantization for approximate nearest neighbor search

Streamlit App For Product Analysis - Streamlit App For Product Analysis

Product-based-recommendation-system - A product based recommendation system which uses Machine learning algorithm such as KNN and cosine similarity

This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021.

Image-retrieval-baseline - MUGE Multimodal Retrieval Baseline

PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation

Perturbed Self-Distillation: Weakly Supervised Large-Scale Point Cloud Semantic Segmentation (ICCV2021)

The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Supervised Contrastive Learning for Product Matching

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

QKeras: a quantization deep learning library for Tensorflow Keras

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

QTool: A Low-bit Quantization Toolbox for Deep Neural Networks in Computer Vision

deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)