Reproducing-BowNet: Learning Representations by Predicting Bags of Visual Words

Last update: Mar 16, 2022

Related tags

Deep Learning Reproducing-BowNet

Overview

Reproducing-BowNet

Our reproducibility effort based on the 2020 ML Reproducibility Challenge. We are reproducing the results of this CVPR 2020 paper: Learning Representations by Predicting Bags of Visual Words by Gidaris et al S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord, “Learning Representations by Predicting Bags of Visual Words,” ArXiv, 27-Feb-2020. [Online]. Available: https://arxiv.org/abs/2002.12247. [Accessed: 15-Nov-2020].

Group project for UWaterloo course SYDE 671 - Advanced Image Processing by Harry Nguyen, Stone Yun, Hisham Mohammad

Code base is implemented with PyTorch. Dataloader is adapted from Github released by authors of the RotNet paper: https://github.com/gidariss/FeatureLearningRotNet

Our model definitions are in model.py. Custom loss and layer class definitions are in layers.py

See dependencies.txt for list of libraries that need to be installed. Pip install or conda install both work

Before running the experiments:

Inside the project code, create a folder ./datasets/CIFAR, download the dataset CIFAR100 from https://www.cs.toronto.edu/~kriz/cifar.html and put in the folder.

For running the code:

Pretrained weights of BowNet and RotNet from our best results are in saved_weights directory. To generate your own RotNet checkpoint, running rotation_prediction_training.py will train a new RotNet from scratch. The checkpoint is saved as rotnet1_checkpoint.pt

To run rotnet_linearclf.py or rotnet_nonlinearclf.py, you need to have the checkpoint file of pretrained RotNet, download here (eg. saved_weights/rotnet.pt). These scripts load the pretrained RotNet and use its feature maps to train a classifier on CIFAR-100 prediction.

$python rotnet_linearclf.py --checkpoint /path/to/checkpoint

$python rotnet_nonlinearclf.py --checkpoint /path/to/checkpoint

bownet_plus_linearclf_cifar_training.py takes pretrained BowNet and uses feature maps to train linear classifier on CIFAR-100. kmeans_cluster_and_bownet_training.py loads pretrained RotNet, performs KMeans clustering of feature map, then trains BowNet on BOW reconstruction. Thus, you'll need pretrained BowNet and RotNet checkpoints respectively.

We also include a pre-computed RotNet codebook for K = 2048 clusters. If you include the path to it for kmeans_cluster_and_bownet_training.py the script will skip the codebook generation step and go straight to BOW reconstruction training

$python bownet_plus_linearclf_cifar_training.py --checkpoint /path/to/bownet/checkpoint

$python kmeans_cluster_and_bownet_training.p --checkpoint /path/to/rotnet/checkpoint [optional: --rotnet_vocab /path/to/rotnet/vocab.npy]

Comments

Some bug fixes in model.py. Created some helper functions that reduce redundancy

Put accuracy and load_checkpoint into utils.py. Created a get_dataloader() function so that we can stop copy/pasting the Dataset and Dataloader instantiations to every training module we have.

opened by StoneY1 0
Updated dataloader transformations for bow-training and implemented a KMeans codebook-loading function.

Fixed dataloader used for RotNet codebook generation. Needed to be dataloader with mode = 'rotation'. Implemented KMeans initialization function so that we can load codebook from a .NPY and use for bow-training. Updated some of the transformation parameters in dataloader.

opened by StoneY1 0
Fixed some errors in BowNet related to BOW training. Also reworked the tensor shapes for softmax.

Corrected an error in bow_training == True behaviour of BowNet. NormalizedLinearLayer should have been receiving the GlobalAvgPooled vector from resblock3_256b_fmaps. Fixed the softmax shapes so that we don't have to do that '[:, 0]' indexing any more. Updated all relevant scripts (don't worry, this time I made sure to merge your latest changes before continuing to add my work).

opened by StoneY1 0
New changes related to Bow reconstruction training

Finished implementing BOW reconstruction training. Added standardization for label images that are returned by dataloader during bow training. Implemented SoftCrossEntropyLoss in model.py (Pytorch's CrossEntropyLoss is only usable for classification tasks).

opened by StoneY1 0
Changes made to Dataloader and train_test.py

Added CIFAR-100 specific standardization (gets applied to all data). Although I haven't fully inspected the order of transformations for Supervised mode.

Also corrected the epoch accuracy calculation in train_test.py train_test.py currently has other training policies that were tried. I didn't remove the previous ones though, just commented out.

opened by StoneY1 0

Owner

GitHub

Repository for reproducing `Model-Based Robust Deep Learning`

Model-Based Robust Deep Learning (MBRDL) In this repository, we include the code necessary for reproducing the code used in Model-Based Robust Deep Le

16 Sep 19, 2022

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

239 Jan 2, 2023

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Deep Continuous Clustering Introduction This is a Pytorch implementation of the DCC algorithms presented in the following paper (paper): Sohil Atul Sh

197 Nov 29, 2022

Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

StackGAN-v2 StackGAN-v1: Tensorflow implementation StackGAN-v1: Pytorch implementation Inception score evaluation Pytorch implementation for reproduci

809 Dec 16, 2022

PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

PySlowFast PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficie

5.3k Jan 3, 2023

Reproducing code of hair style replacement method from Barbershorp.

Barbershorp Reproducing code of hair style replacement method from Barbershorp. Also reproduces II2S, an improved version of Image2StyleGAN. Requireme

1 Dec 24, 2021

PyTorch framework, for reproducing experiments from the paper Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks. Code, based on the PyTorch framework, for reprodu

3 Dec 27, 2022

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations This repo contains the Pytorch implementation of our paper: Revisit

80 Nov 20, 2022

PyTorch implementation of SimCLR: A Simple Framework for Contrastive Learning of Visual Representations

1.7k Dec 28, 2022

Conformer: Local Features Coupling Global Representations for Visual Recognition

Conformer: Local Features Coupling Global Representations for Visual Recognition (arxiv) This repository is built upon DeiT and timm Usage First, inst

378 Jan 8, 2023

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

The ImageNet-CoG Benchmark Project Website Paper (arXiv) Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalizatio

23 Oct 9, 2022

This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).

Predicting Patient Outcomes with Graph Representation Learning This repository contains the code used for Predicting Patient Outcomes with Graph Repre

76 Dec 22, 2022

A geometric deep learning pipeline for predicting protein interface contacts.

44 Dec 30, 2022

Predicting path with preference based on user demonstration using Maximum Entropy Deep Inverse Reinforcement Learning in a continuous environment

Preference-Planning-Deep-IRL Introduction Check my portfolio post Dependencies Gym stable-baselines3 PyTorch Usage Take Demonstration python3 record.

9 Oct 26, 2022

Predicting Tweet Sentiment Maching Learning and streamlit

Predicting-Tweet-Sentiment-Maching-Learning-and-streamlit (I prefere using Visual Studio Code ) Open the folder in VS Code Run the first cell in requi

1 Nov 20, 2021

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

310 Dec 28, 2022