Implementation of Barlow Twins paper

Related tags

Deep Learning self-supervised-learning
Overview

barlowtwins

PyTorch Implementation of Barlow Twins paper: Barlow Twins: Self-Supervised Learning via Redundancy Reduction

This is currently a work in progress. The code is a modified version of the SimSiam implementation here

  • Time per epoch is around 40 seconds on a V100 GPU
  • GPU usage is around 9 GBytes
  • The current version reaches around 84.7% test accuracy

Todo:

  • warmup learning rate from 0
  • report results on cifar-10
  • create PR to add to lightly

Installation

pip install -r requirements.txt

Dependencies

  • PyTorch
  • PyTorch Lightning
  • Torchvision
  • lightly

Benchmarks

We benchmark the BarlowTwins model on the CIFAR-10 dataset following the KNN evaluation protocol. Currently, the best effort achieved a test accuracy of 84.7%.

Accuracy Loss

Paper

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Comments
  • Fix buggy loss

    Fix buggy loss

    The off-diagonal elements were not really multiplied by the lambda_param. I fixed it and additionally upgraded the lightly requirement to 1.0.8. With this setup I'm at 56% accuracy after 48 epochs on CIFAR-10.

    opened by philippmwirth 0
  • Parameters to reproduce example charts

    Parameters to reproduce example charts

    Hi @IgorSusmelj, thanks for having this open source! I was wondering if you have the parameters used for the "best effort" that yielded the charts posted in the README? Running main.py under the stock configuration has the kNN accuracy stabilize at around 60% by epoch 70.

    opened by mx781 0
  • Do you do linear evaluation?

    Do you do linear evaluation?

    I've seen some implementations of SimCLR, MoCo, BYOL and they have a python file that does linear evaluation. Simply, they froze the parameters of the trained encoder (resnet18 or resnet50) and only train a FC to see the top1 accuracy on test dataset. Do you do this testing? I'm asking because I'm not sured what does the kNN-acc do in this project. Thank you and look forward to your reply.

    opened by cucutone 0
  • about the projection dimension

    about the projection dimension

    First of all, thank you for sharing this project. It's nice to understand BarlowTwins more with your code. I've read some unsupervised learning papers like SimCLR and MoCo. The projection there is kind of decreasing the dimension of representations. (2048->512->128, something like this). On the other hand, BarlowTwins uses the projection layer that increases the dimension of representations. Do you have any thought about the reasons? (I know the loss function and the concept of algorithms are quite different between SimCLR and BarlowTwins but still quite confused about the projection layers) Look forward to your reply. Again, thank you for this project.

    opened by cucutone 0
  • Projection_mlp

    Projection_mlp

    Hi,

    Are you using a projection network with following dimensions resnet (output) -> 512 -> 2048 -> 2048 ?

    If that is the case then I am curious to know that why you decided to do it like this as in the paper they are having projection layer of 8192 dimensions.

    Thanks

    opened by Mushtaqml 1
Owner
IgorSusmelj
Co-founder at Lightly Degree from ETH Zurich with a focus on embedded computing and machine learning.
IgorSusmelj
GitHub
PyTorch implementation of Barlow Twins.

Barlow Twins: Self-Supervised Learning via Redundancy Reduction PyTorch implementation of Barlow Twins. @article{zbontar2021barlow, title={Barlow Tw

Facebook Research 839 Dec 29, 2022
Barlow Twins and HSIC

Barlow Twins and HSIC Unofficial Pytorch implementation for Barlow Twins and HSIC_SSL on small datasets (CIFAR10, STL10, and Tiny ImageNet). Correspon

Yao-Hung Hubert Tsai 49 Nov 24, 2022
Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

null 482 Dec 18, 2022
The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

This repository contains the software implementation of most algorithms used or developed in my research. The LaTeX and Python code for generating the

João Fonseca 3 Jan 3, 2023
Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow 18 Oct 6, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.

Regularized Greedy Forest Regularized Greedy Forest (RGF) is a tree ensemble machine learning method described in this paper. RGF can deliver better r

RGF-team 364 Dec 28, 2022
Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"

Description: This is the official implementation of our AAAI-21 accepted paper Label Confusion Learning to Enhance Text Classification Models. The str

null 101 Nov 25, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

null 49 Nov 23, 2022
A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Exploring simple siamese representation learning This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that

Taojiannan Yang 72 Nov 9, 2022
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting.

Non-AR Spatial-Temporal Transformer Introduction Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series For

Chen Kai 66 Nov 28, 2022
This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

null 212 Dec 25, 2022
Official implementation of the ICLR 2021 paper

You Only Need Adversarial Supervision for Semantic Image Synthesis Official PyTorch implementation of the ICLR 2021 paper "You Only Need Adversarial S

Bosch Research 272 Dec 28, 2022
Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

Phil Wang 95 Jan 2, 2023
Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

zhaohu xing 112 Dec 16, 2022
Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow 201 Dec 21, 2022
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023