A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Related tags

Deep Learning pytorch-SimSiam
Overview

Exploring simple siamese representation learning

This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that reported in the paper. The implementation is based on the codes of MOCO.

Unsupervised pre-training

To run unsupervised pre-training on ImageNet,

sh train_simsiam.sh

This is to do the unsupervised pre-training for 100 epochs. Please modify the path to your ImageNet data folder.

Note 1: I try to follow the setting in the paper, which is bs=512 and lr=0.1 on 8-GPU, but somehow I can not fit it. So I used the max batch_size that I can fit (432) while kept the lr unchaged (0.1).

Note 2: In pre-training, I didn't fix the lr of prediction MLP. According to the paper (Table. 1), fixing the lr of prediction MLP can give slightly improvements (67.7% -> 68.1%). You can try it if interested.

Linear evaluation

To run linear evaluation,

sh train_lincls.sh

The linear evaluation is done using NVIDIA LARC optimizer by setting trus_coefficient=0.001 and clip=False. The batch size is 4096.

Note: I first followed the setting in the paper, which is Lr=0.32 (0.02*4096/256). But I can only got a result of 66.0%. Then I increased the learning rate to Lr=1.6 (0.1*4096.256) and achieved the result of 67.8%. The results and models are given below.

SimSiam pretrained batchsize lincls Lr Top-1 Acc
Reported 512 0.32 67.7%
Reproduced 432 (Model) 1.6 67.8% (Model)
Reproduced 432 0.32 66.0%

Acknowledgement

Thank Xinlei for his help on some implementation details.

Comments
  • Training loss collapse

    Training loss collapse

    I am trying to pretrain the SimSiam model on mscoco dataset.. but the loss collapses to -1 very quickly.. What are the possible reasons behind and some suggestions to solve the same?

    opened by aisagarw 0
  • Is the bias term in the predictor/projector in the Linear layer necessary?

    Is the bias term in the predictor/projector in the Linear layer necessary?

    Hi, thanks for your work. It's wonderful. But I noticed that the bias arg in all nn.Linear layer in models/simsiam.py is set as its default value True. To my knowledge, the bias in linear/conv before batch_norm will take no effect as BN will remove all biases. So I'd like to know whether the bias term in Linear before batch_norm could be removed during training.

    opened by flishwang 0
  • transferring performance on coco dataset

    transferring performance on coco dataset

    Hello, Have you ever try to transfer the simsiam model to object detection task. I have trained a 200-epoch model on ImageNet through simsiam（following the optimal config in paper), Then I take it as pre-trained model of Faster-RCNN and use the transferring config of moco-v2, but I got bad performance. I wonder whether there are different training schedule when transferring self-supervised pre-trained model to object detection task? Hope to get your suggestiones!

    opened by UcanSee 1
  • collapse on other dataset

    collapse on other dataset

    firstly,it is a great job to do ddp(distributed_data_paralel) train on huge dataset(other code is on cifar10), i could get the result you published. however ,when i train the simsiam on other dataset(not published) the simsiam model tend to collapse, i wonder whether it is the problem of not suitable lr or not including the procedure of warm up, i really hope Dr.tao could share your suggestions.

    opened by knightyxp 3
  • about fixing the lr of prediction MLP

    about fixing the lr of prediction MLP

    Thank you for the implementation! Nice work!

    For me I just did not get what the lr decay of prediction MLP means. Does it mean the lr decay in the pretraining stage as we normally use?

    opened by danielchyeh 5
Owner
Taojiannan Yang
PhD student at University of North Carolina at Charlotte.
Taojiannan Yang
GitHub
PyTorch implementation of SimSiam: Exploring Simple Siamese Representation Learning

SimSiam: Exploring Simple Siamese Representation Learning This is a PyTorch implementation of the SimSiam paper: @Article{chen2020simsiam, author =

Facebook Research 834 Dec 30, 2022
Exploring Simple Siamese Representation Learning

G-SimSiam A PyTorch implementation which refers to repo for the paper Exploring Simple Siamese Representation Learning by Xinlei Chen & Kaiming He Add

zhuyun 1 Dec 19, 2021
A PyTorch implementation of "Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning", IJCAI-21

MERIT A PyTorch implementation of our IJCAI-21 paper Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning. Depen

Graph Analysis & Deep Learning Laboratory, GRAND 32 Jan 2, 2023
[CVPR 2022 Oral] Crafting Better Contrastive Views for Siamese Representation Learning

Crafting Better Contrastive Views for Siamese Representation Learning (CVPR 2022 Oral) 2022-03-29: The paper was selected as a CVPR 2022 Oral paper! 2

null 249 Dec 28, 2022
[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.

MiVOS (CVPR 2021) - Mask Propagation Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang [arXiv] [Paper PDF] [Project Page] [Papers with Code] This repo impleme

Rex Cheng 106 Jan 3, 2023
"Segmenter: Transformer for Semantic Segmentation" reproduced via mmsegmentation

Segmenter-based-on-OpenMMLab "Segmenter: Transformer for Semantic Segmentation, arxiv 2105.05633." reproduced via mmsegmentation. We reproduce Segment

EricKani 22 Feb 24, 2022
Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

EgoNet Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo inclu

Shichao Li 138 Dec 9, 2022
The official implementation of paper Siamese Transformer Pyramid Networks for Real-Time UAV Tracking, accepted by WACV22

SiamTPN Introduction This is the official implementation of the SiamTPN (WACV2022). The tracker intergrates pyramid feature network and transformer in

Robotics and Intelligent Systems Control @ NYUAD 28 Nov 25, 2022
PyTorch implementation of Asymmetric Siamese (https://arxiv.org/abs/2204.00613)

Asym-Siam: On the Importance of Asymmetry for Siamese Representation Learning This is a PyTorch implementation of the Asym-Siam paper, CVPR 2022: @inp

Meta Research 89 Dec 18, 2022
Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning By Zhenda Xie*, Yutong Lin*, Zheng Zhang, Yue Ca

Zhenda Xie 293 Dec 20, 2022
Exploring Visual Engagement Signals for Representation Learning

Exploring Visual Engagement Signals for Representation Learning Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie and Ser-Nam Lim C

Menglin Jia 9 Jul 23, 2022
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper

ImageNet-21K Pretraining for the Masses Paper | Pretrained models Official PyTorch Implementation Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelni

null 574 Jan 2, 2023
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
From this paper "SESNet: A Semantically Enhanced Siamese Network for Remote Sensing Change Detection"

SESNet for remote sensing image change detection It is the implementation of the paper: "SESNet: A Semantically Enhanced Siamese Network for Remote Se

null 1 May 24, 2022
Siamese-nn-semantic-text-similarity - A repository containing comprehensive Neural Networks based PyTorch implementations for the semantic text similarity task

Siamese Deep Neural Networks for Semantic Text Similarity PyTorch A repository c

Shahrukh Khan 32 Dec 15, 2022
(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

Res2Net Applications 928 Dec 29, 2022
Pytorch implementation of "Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet"

Token Labeling: Training an 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet (arxiv) This is a Pytorch implementation of our te

蒋子航 383 Dec 27, 2022
PyTorch implementation of PNASNet-5 on ImageNet

PNASNet.pytorch PyTorch implementation of PNASNet-5. Specifically, PyTorch code from this repository is adapted to completely match both my implemetat

Chenxi Liu 314 Nov 25, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 32 Dec 26, 2022