ReSSL: Relational Self-Supervised Learning with Weak Augmentation

Last update: Oct 25, 2022

Overview

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

This repository contains PyTorch evaluation code, training code and pretrained models for ReSSL.

For details see ReSSL: Relational Self-Supervised Learning with Weak Augmentation by Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang and Chang Xu

Reproducing

To run the code, you probably need to change the Dataset setting (dataset/imagenet.py), and Pytorch DDP setting (util/dist_init.py) for your own server enviroments.

The distribued training of this code is base on slurm enviroments, we have provide the training scrips under the script folder.

We also provide the pretrained model for ResNet50 (single crop and 5 crops)

	Arch	BatchSize	Epochs	Crops	Linear Eval	Download
ReSSL	ResNet50	256	200	1	69.9 %	ressl-200.pth
ReSSL	ResNet50	256	200	5	74.7 %	ressl-multi-200.pth

If you want to test the pretained model, please download the weights from the link above, and move it to the checkpoints folder (create one if you don't have .checkpoints/ directory). The evaluation scripts also has been provided in script/train.sh

Citation

If you find that ReSSL interesting and help your research, please consider citing it:

@misc{zheng2021ressl,
      title={ReSSL: Relational Self-Supervised Learning with Weak Augmentation}, 
      author={Mingkai Zheng and Shan You and Fei Wang and Chen Qian and Changshui Zhang and Xiaogang Wang and Chang Xu},
      year={2021},
      eprint={2107.09282},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Comments

Implementation details about linear evaluation

Thanks for your great work and code sharing! I downloaded the code and logs for small-sized dataset but can't reproduce the linear evaluation accuracy in table 1 following page 6 settings, can you provide some detailed hyperparameter settings? If I'm not wrong, we use 100 epochs linear eval after 200 epochs pertaining?

opened by MrChenFeng 11
Weight decay and Resnet18
Hi!

In my last issue, I forgot to congratulate to your exceptional paper. I was also looking for a relational method since PAWS, but couldn't really find one that could achieve such high performance on imagenet. Also, this method works with small batch size and very low computational resources due to the frozen target network and single-view backprop. Nice work!

Reading the code I noticed two minor differences to the paper though. Can you please double-check these and clarify which one reflects the results published?

(1) Weight decay

Paper: I didn't find any mentions on weight decay when training on Imagenet, but found 5e-4 for small and medium datasets.

Code: link You use 1e-4 weight decay and 0 for bias. Is it the default Imagenet settings?

(2) Resnet18 7x7 conv

Paper:

We adopt the ResNet18 [25] as our backbone network. Because most of our dataset contains low-resolution images, we replace the first 7x7 Conv of stride 2 with 3x3 Conv of stride 1 and remove the first max pooling operation for a small dataset.

Code: link I see no sign of these changes, it looks you kept the original imagenet-resnet setup. Didn't you?

Thank you.

PS.: I am about to reproduce your results from the paper, but currently hanging around 65% on Imagenet.
opened by gergopool 9
What is the final loss size?

Hi @KyleZheng1997,

Thanks for your contribution, could you provide the pre-training log? I really want to know the pre-train loss when training is over.

opened by mitming 4
Pretrained models for CIFAR10, 100 and STL10

Hi @KyleZheng1997 , Thanks for the nice work and code. Would it be possible to provide the pretrained models for CIFAR-10, 100 and STL-10. Thanks in Advance.

opened by dmlpt 3
Pretrain on CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet

Thank you for your great work! I notice that results on small and medium datasets (i.e. CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet) are provided in your paper. Can you provide pretraining configs on these datasets?

opened by gyfastas 2
CIFAR100 accuracy

After downloading your tiny-ressl implementation and evaluating your pretrained cifar100 network, I get 66.7%, while you reported 63.8% in the paper. When you have the time, can you please double-check which percentage is right? Thanks.

opened by gergopool 1
Momentum for imagenet

Hi!

In the paper I found no reference for the momentum parameter when training on Imagenet. However, I noticed 0.999 is the default in your code.

Did you use 0.999 momentum on Imagenet?

Thanks.

opened by gergopool 1
KNN evaluation

Hey, Great paper! By the design of the loss and architecture , I would guess the learned representation should be good or even SOTA in KNN evaluation (that was also showed In DINO paper). I did not see such results on imagenet in the paper, did you try it? And if yes, can you share your results? Thanks!

opened by YoadTew 3

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

Related tags

Overview

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

Reproducing

Citation

Comments

Implementation details about linear evaluation

Weight decay and Resnet18

What is the final loss size?

Pretrained models for CIFAR10, 100 and STL10

Pretrain on CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet

CIFAR100 accuracy

Momentum for imagenet

KNN evaluation

Owner

mingkai

Pytorch implementation of "A simple neural network module for relational reasoning" (Relational Networks)

The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Self-training with Weak Supervision (NAACL 2021)

A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''

Weak-supervised Visual Geo-localization via Attention-based Knowledge Distillation

[CVPR 2022] Back To Reality: Weak-supervised 3D Object Detection with Shape-guided Label Enhancement

Self-supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021)

Pytorch codes for "Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation"

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Official implementation of "Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision" ECCV2020

Open source implementation of AceNAS: Learning to Rank Ace Neural Architectures with Weak Supervision of Weight Sharing

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

[ICCV 2021] Official PyTorch implementation for Deep Relational Metric Learning.

ReLoss - Official implementation for paper "Relational Surrogate Loss Learning" ICLR 2022