[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Related tags

Deep Learning zero-shot-learning nus-wide multi-label-learning self-attention openimages-v4 channel-attention generalized-zero-shot-learning multi-label-zsl iccv2021
Overview

PWC PWC

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021)

[arXiv][Project page >> coming soon]

Sanath Narayan*, Akshita Gupta*, Salman Khan, Fahad Shahbaz Khan, Ling Shao, Mubarak Shah

( 🌟 denotes equal contribution)

Installation

The codebase is built on PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.6, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

conda create -n mlzsl python=3.6
conda activate mlzsl
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image scikit-learn opencv-python yacs joblib natsort h5py tqdm pandas

Install warmup scheduler

cd pytorch-gradual-warmup-lr; python setup.py install; cd ..

Attention Visualization

Results

Our approach on NUS-WIDE Dataset.

Our approach on OpenImages Dataset.

Training and Evaluation

NUS-WIDE

Step 1: Data preparation

  1. Download pre-computed features from here and store them at features folder inside BiAM/datasets/NUS-WIDE directory.
  2. [Optional] You can extract the features on your own by using the original NUS-WIDE dataset from here and run the below script:
python feature_extraction/extract_nus_wide.py

Step 2: Training from scratch

To train and evaluate multi-label zero-shot learning model on full NUS-WIDE dataset, please run:

sh scripts/train_nus.sh

Step 3: Evaluation using pretrained weights

To evaluate the multi-label zero-shot model on NUS-WIDE. You can download the pretrained weights from here and store them at NUS-WIDE folder inside pretrained_weights directory.

sh scripts/evaluate_nus.sh

OPEN-IMAGES

Step 1: Data preparation

  1. Please download the annotations for training, validation, and testing into this folder.

  2. Store the annotations inside BiAM/datasets/OpenImages.

  3. To extract the features for OpenImages-v4 dataset run the below scripts for crawling the images and extracting features of them:

## Crawl the images from web
python ./datasets/OpenImages/download_imgs.py  #`data_set` == `train`: download images into `./image_data/train/`
python ./datasets/OpenImages/download_imgs.py  #`data_set` == `validation`: download images into `./image_data/validation/`
python ./datasets/OpenImages/download_imgs.py  #`data_set` == `test`: download images into `./image_data/test/`

## Run feature extraction codes for all the 3 splits
python feature_extraction/extract_openimages_train.py
python feature_extraction/extract_openimages_test.py
python feature_extraction/extract_openimages_val.py

Step 2: Training from scratch

To train and evaluate multi-label zero-shot learning model on full OpenImages-v4 dataset, please run:

sh scripts/train_openimages.sh
sh scripts/evaluate_openimages.sh

Step 3: Evaluation using pretrained weights

To evaluate the multi-label zero-shot model on OpenImages. You can download the pretrained weights from here and store them at OPENIMAGES folder inside pretrained_weights directory.

sh scripts/evaluate_openimages.sh

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Citation

If you find this repository useful, please consider giving a star ⭐ and citation 🎊 :

@article{narayan2021discriminative,
title={Discriminative Region-based Multi-Label Zero-Shot Learning},
author={Narayan, Sanath and Gupta, Akshita and Khan, Salman and  Khan, Fahad Shahbaz and Shao, Ling and Shah, Mubarak},
journal={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
publisher = {IEEE},
year={2021}
}

Contact

Should you have any question, please contact 📧 [email protected]

Comments
  • About download_imgs. Py

    About download_imgs. Py

    No such file or directory: 1:labelmap_path=path+'classes-trainable.txt' 2:dict_path=path+'class-descriptions.csv' 3:df_label = pd.read_csv(path+data_set+'/annotations-human.csv') 4:df_image = pd.read_csv(path+data_set+'/images.csv')

    opened by sorrowyn 14
  • About Pretrained Weights

    About Pretrained Weights

    Thanks for your working.

    I noticed that you mentioned pre trained model weights in the code. I have some interest in this,Could you realse the pretained weight in the GitHub or tell me how to pre-train the model.

    thank you very much.

    opened by luoyixi924208423 2
  • Why multi-headed self-attention?

    Why multi-headed self-attention?

    Dear author, thanks for your papper and code.However,I've had a problem for a long time.Why the multi-headed attention were used in RCB?Can we not use multi-head? Like just using the normal self-attention mechanism in RCB.Looking forward to your reply.Thank you so much.

    opened by hugh920 1
  • Missing 81_only_full_nus_wide_train.json and 81_only_full_nus_wide_test.json files

    Missing 81_only_full_nus_wide_train.json and 81_only_full_nus_wide_test.json files

    Thanks for your great work and the code release. I am now using your code (extract_nus_wide.py) to extract the features of the NUS-WIDE dataset, but I did not find the 81_only_full_nus_wide_train.json and 81_only_full_nus_wide_test.json files. Could you please provide these two json files?Thank you very much.

    opened by MPR666 1
  • Training with Resnet50

    Training with Resnet50

    Thank you for your great work and the code release, I've learned a lot from your work. I just wonder if you tried BiAM with different backbone networks such as Resnet or VIT? I try to train BiAM with features extracted from Resnet50, but it is very easy to get 'nan' loss value after several training epochs. Do you have any suggestions to fix this?

    opened by SonDaoDuy 1
  • ImportError: cannot import name 'Net'

    ImportError: cannot import name 'Net'

    In the data preparation step for NUS-WIDE, I met the import error as in the title. I couldn't find the model called "Net" inside the model.py.

    I would like to know which network model is this "Net" refers to.

    opened by Kashu7100 1
  • Strange Loss

    Strange Loss

    Hello, @akshitac8 , I want to reproduce the results with the pre-computed features you provide. But I got a strange loss, it seems that the model didn't converge well. 企业微信截图_20210901154910

    opened by sunanhe 0
  • Misisng Files

    Misisng Files

    Hey Will you please help me with the following:

    1. How can I get the "/images.csv"
    2. Where to find 'full_train_partition_idxs.npy' and 'full_train_files.npy'
    opened by jswapnil10 0
Owner
Akshita Gupta
Sem @IITR | Outreachy @mozilla | Research Engineer @IIAI
Akshita Gupta
GitHub https://akshitac8.github.io/BiAM/
Where-Got-Time - An NUS timetable generator which uses a genetic algorithm to optimise timetables to suit the needs of NUS students

Where Got Time(table)? A timetable optimsier which uses an evolutionary algorith

Nicholas Lee 3 Jan 9, 2022
Official Pytorch Implementation of: "Semantic Diversity Learning for Zero-Shot Multi-label Classification"(2021) paper

Semantic Diversity Learning for Zero-Shot Multi-label Classification Paper Official PyTorch Implementation Avi Ben-Cohen, Nadav Zamir, Emanuel Ben Bar

null 28 Aug 29, 2022
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
Shared Attention for Multi-label Zero-shot Learning

Shared Attention for Multi-label Zero-shot Learning Overview This repository contains the implementation of Shared Attention for Multi-label Zero-shot

dathuynh 26 Dec 14, 2022
Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation (AAAI 2021) Official pytorch implementation of our paper: Discriminative

Beom 74 Dec 27, 2022
Official implementation of "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise" (NeurIPS 2021)

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise NeurIPS 2021: This repository is the official implementation of ODNL. Require

Hongxin Wei 12 Dec 7, 2022
A PyTorch implementation of ICLR 2022 Oral paper PiCO: Contrastive Label Disambiguation for Partial Label Learning

PiCO: Contrastive Label Disambiguation for Partial Label Learning This is a PyTorch implementation of ICLR 2022 Oral paper PiCO; also see our Project

王皓波 83 May 11, 2022
Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

null 150 Dec 26, 2022
Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation

Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation The code of: Cross-Image Region Mining with Region Proto

LiuWeide 16 Nov 26, 2022
Label Mask for Multi-label Classification

LM-MLC 一种基于完型填空的多标签分类算法 1 前言 本文主要介绍本人在全球人工智能技术创新大赛【赛道一】设计的一种基于完型填空(模板)的多标签分类算法：LM-MLC，该算法拟合能力很强能感知标签关联性，在多个数据集上测试表明该算法与主流算法无显著性差异，在该比赛数据集上的dev效果很好，但是由

null 52 Nov 20, 2022
PyTorch implementation of 1712.06087 "Zero-Shot" Super-Resolution using Deep Internal Learning

Unofficial PyTorch implementation of "Zero-Shot" Super-Resolution using Deep Internal Learning Unofficial Implementation of 1712.06087 "Zero-Shot" Sup

Jacob Gildenblat 196 Nov 27, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Official PyTorch Implementation of Embedding Transfer with Label Relaxation for Improved Metric Learning, CVPR 2021

Embedding Transfer with Label Relaxation for Improved Metric Learning Official PyTorch implementation of CVPR 2021 paper Embedding Transfer with Label

Sungyeon Kim 37 Dec 6, 2022
EMNLP 2021 Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections Ruiqi Zhong, Kristy Lee*, Zheng Zhang*, Dan Klein EMN

Ruiqi Zhong 42 Nov 3, 2022
Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

null 184 Dec 11, 2022
(ICCV'21) Official PyTorch implementation of Relational Embedding for Few-Shot Classification

Relational Embedding for Few-Shot Classification (ICCV 2021) Dahyun Kang, Heeseung Kwon, Juhong Min, Minsu Cho [paper], [project hompage] We propose t

Dahyun Kang 82 Dec 24, 2022
SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model

SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frede

Edresson Casanova 92 Dec 9, 2022
Zero-shot Synthesis with Group-Supervised Learning (ICLR 2021 paper)

GSL - Zero-shot Synthesis with Group-Supervised Learning Figure: Zero-shot synthesis performance of our method with different dataset (iLab-20M, RaFD,

Andy_Ge 62 Dec 21, 2022
Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"

Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds Björn Michele1), Alexandre Boulch1), Gilles Puy1), Maxime Bucher1) and Rena

valeo.ai 15 Dec 22, 2022