Contrastive Learning with Non-Semantic Negatives

Last update: Jul 31, 2022

Overview

Contrastive Learning with Non-Semantic Negatives

This repository is the official implementation of Robust Contrastive Learning Using Negative Samples with Diminished Semantics. Contrastive learning utilizes positive pairs which preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples to make the model more robust, where only the superfluous instead of the semantic features are preserved.

Preparation

Install PyTorch and check preprocess/ for ImageNet-100 and ImageNet-Texture preprocessing scripts.

Training

The following code is used to pre-train MoCo-v2 + patch / texture-based NS. The major code is developed with minimal modifications from the official implementation.

python moco-non-sem-neg.py -a resnet50 --lr 0.03 --batch-size 128 --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed --world-size 1 --rank 0 \
  --mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
  --robust nonsem --num-nonsem 1 --alpha 2 --epochs 200 --patch-ratio 16 72 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200  \
  /path/to/imagenet-100/ 

python moco-non-sem-neg.py -a resnet50 --lr 0.03 --batch-size 128 --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed --world-size 1 --rank 0 \
  --mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
  --robust texture_syn --num-nonsem 1 --alpha 2 --epochs 200 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs128_lr0.03_texture_nn1_alpha2_epoch200 \
  /path/to/imagenet-100-texture/

Change /path/to/imagenet-100/ with the ImageNet-100 dataset directory.
Change --alpha and -moco-k to reproduce results with different configurations.

Linear Evaluation

Run following code is used to reproduce MoCo-v2 + patch-based NS model reported in Table 1.

python main_lincls.py -a resnet50 --lr 10.0 --batch-size 128 --epochs 60 \
  --pretrained ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200/checkpoint_0199.pth.tar \
  --dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
  --ckpt_path ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200 \
  /path/to/imagenet-100/

Pre-trained Models

You can download pretrained models here:

	moco-k	alpha	ImageNet-100	Corruption	Sketch	Stylized	Rendition	Checkpoints
MoCo-v2	16384	-	77.88±0.28	43.08±0.27	28.24±0.58	16.20±0.55	32.92±0.12	Run1, Run2, Run3
+ Texture	16384	2	77.76±0.17	43.58±0.33	29.11±0.39	16.59±0.17	33.36±0.15	Run1, Run2, Run3
+ Patch	16384	2	79.35±0.12	45.13±0.35	31.76±0.88	17.37±0.19	34.78±0.15	Run1, Run2, Run3
+ Patch	16384	3	75.58±0.52	44.45±0.15	34.03±0.58	18.60±0.26	36.89±0.11	Run1, Run2, Run3
MoCo-v2	8192	-	77.73±0.38	43.22±0.39	28.45±0.36	16.83±0.12	33.19±0.44	Run1, Run2, Run3
+ Patch	8192	2	79.54±0.32	45.48±0.20	33.36±0.45	17.81±0.32	36.31±0.37	Run1, Run2, Run3

You might also like...

Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Siamese Deep Neural Networks for Semantic Text Similarity PyTorch A repository c

32 Dec 15, 2022

Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Semantic-NeRF: Semantic Neural Radiance Fields Project Page | Video | Paper | Data In-Place Scene Labelling and Understanding with Implicit Scene Repr

243 Jan 7, 2023

Dense Contrastive Learning (DenseCL) for self-supervised representation learning, CVPR 2021.

Dense Contrastive Learning for Self-Supervised Visual Pre-Training This project hosts the code for implementing the DenseCL algorithm for se

491 Jan 3, 2023

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning This repository contains the code and relevant instructions

5 Aug 19, 2022

[ICLR'21] FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization This is the PyTorch implemention of our paper FedBN: Federated Learning on

156 Dec 15, 2022

PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

Memory In Memory Networks It is based on the paper Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spati

12 May 30, 2022

Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

Comments

Where is ShufflePatches?

https://github.com/SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives/blob/5fbdc4d84346afa7a17ae2632cab38a6291d69fb/moco/loader.py#L217

Sorry, but I cannot find ShufflePatches in the code.

opened by le4m 1

Contrastive Learning with Non-Semantic Negatives

Related tags

Overview

Contrastive Learning with Non-Semantic Negatives

Preparation

Training

Linear Evaluation

Pre-trained Models

You might also like...

Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Build upon neural radiance fields to create a scene-specific implicit 3D semantic representation, Semantic-NeRF

Dense Contrastive Learning (DenseCL) for self-supervised representation learning, CVPR 2021.

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning

[ICLR'21] FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

This is a template for the Non-autoregressive Deep Learning-Based TTS model (in PyTorch).

Curvlearn, a Tensorflow based non-Euclidean deep learning framework.

Comments

Where is ShufflePatches?

Owner

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

A non-linear, non-parametric Machine Learning method capable of modeling complex datasets

A Robust Non-IoU Alternative to Non-Maxima Suppression in Object Detection

SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING - The Facebook paper about fine tuning RoBERTa with contrastive loss

Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

PyTorch implementation of Federated Learning with Non-IID Data, and federated learning algorithms, including FedAvg, FedProx.

A curated list of awesome resources related to Semantic Search🔎 and Semantic Similarity tasks.

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)