PyTorch implementation of Barlow Twins.

can you provide the config to train with batch size 256? how to set the lambd and scale loss? and does the LARS is still necessary when training with small batch sizes?

This line https://github.com/facebookresearch/barlowtwins/blob/main/main.py#L208 use torch.distributed.all_reduce to sum the correlation matrices across all gpus. However as I know this op is not dedicated for forward computation where backward computation would run later. Instead, to apply "correctly differentiable" distributed all reduce, the official PyTorch document recommends using torch.distributed.nn.*: https://pytorch.org/docs/stable/distributed.html#autograd-enabled-communication-primitives

Hi,

Thanks for making it painless for others to use and build upon your work.

I'm curious to know what the loss function value looks like during the training process on the val set of images. In my case of using spectrogram images on a resnet-18 backbone, the network trains well but the val loss value is very noisy and shows no clear trend. For example, I'm not sure what to make of this:

In the original experiments, was the BT loss value on the val set tracked every N epochs (without any classifier involved)? As far as I can tell, the loss reported in the 'val logs' txt file is of the classifier that was trained on top of the embeddings generating using a single training checkpoint.

The loss curve on the val set should be expected to follow typical behavior, right? Any pointers as to what might be causing a noisy val loss?

Hello, I really enjoyed reading the paper and thought about the intentions of the loss.

However, I was wondering if setting the target matrix as identity matrix is eligible.

As far as I understand, each element of cross correlation matrix is matrix multiplication on each feature element. Barlow Twins loss aims to have correlation of 1 on the diagonal and 0(no correlation) on the non-diagonal elements.

So, if two identical representation vector were fed to the loss, I thought it should give loss of zero, but it didn't.

For the sake of simplicity, let's say we have 2 pairs of representation vectors with identical values. (that's 4 vectors) However, when I take two identical 1d vectors for 2 data, took the batch norm and computed Barlow Twins loss with them, I got 1 on the diagonal but not 0 on the non-diagonal elements.

Same thing goes for the case when the batch size is 1. (batch norm makes the value to be normalized to zero though)

I'm not sure how the loss will learn invariance and redundancy with the target of identity matrix, especially on the redundancy term. Can you please elaborate on how representation vector learns within redundancy term?

Here's a simple example I tried. (I followed the code implementation)

Thank you!

Thanks for your great work. If there are two machines (each with 8 V100 GPUs) connected with ethernet, without slurm management, then how to run the code with your stated 16 V100 config?

Hi. Thanks for the great work! I'm trying to reproduce your paper's results in Fig. 4 (effect of the dimensionality of the last layer of the projector network on performance). I have two questions about this:

• Could you teach me what hyperparameters you used in the experiments?
• Did you run main.py with --projector 8192-8192-16384? not --projector 16384-16384-16384, right?

Hi, thank you for your work, it inspires me a lot. I want to apply BT to my data set about one-dimensional pulse signals instead of images, the data length is 500, do you think it needs to be changed to 224? I did the following work: I changed the number of channels in resnet50 from 3 to 1, and conv2d to conv1d, but the output dimension is still 2048. Good results can be obtained with supervised networks, but the loss of self-supervised learning processes does not drop significantly, and the downstream classification tasks are not as good as supervised networks. mlp also raised the dimension to 8192, but that didn't work either. What do you think might be the reason? Or can you give me some advice. Thanks again!

Hi, Thanks a lot for the very clear implementation and the paper is so easy to read!. I had a quick question on the Barlow Twins loss.

Since Barlow Twins relies on the statistics of a batch of data, if there are batch norm layers in the encoder network, is it possible that the parameters of the BN layers be updated/affected more than any other parameters in the network to optimize the BT loss? Did you experiment without any batchnorm layers in the encoder to see if that affects the learned representations?

Thanks for this great work!

I am a bit confused about the computation of the Barlow Twins loss in the multi-gpu setting. If I understand it correctly, each batch is split into smaller minibatches and these are then processed on separage GPUs. Each GPU computes the cross correlation matrix corresponding to its minibatch. The cross correlations between samples on different GPUs are not computed. It is not clear to me, why the different cross correlation matrices are averaged out across GPUs. This creates a mean correlation matrix and this one is then used for loss computation.

Why not compute the loss for each correlation matrix separately and only average out the final loss? Or even better, why not compute the full cross correlation matrix (i.e. gather all embedding vectors onto one device and computing the cross correlation there?)

I fail to see why summing up correlation matrices is a valid mathematical operation - or is it just an implementation "hack" that makes things easier? I guess since all cross correlation matrices are ideally converging towards identity matrices (as forced by the loss function), avereging them out does not strictly break the convergence - is that the case?

I am not very experienced with distributed deep learning so there may be technical things I don't understand. Thanks for your help.

https://github.com/facebookresearch/barlowtwins/blob/a655214c76c97d0150277b85d16e69328ea52fd9/main.py#L206-L223

Hi, I'm very glad to thank your guys share this amazing project. it training very easy. but when I use it to test or predict, it blocked and can not be killed

here is my predictor code:

` import torch import argparse from PIL import Image from torch import nn, optim from main import BarlowTwins, Transform

""" BarlowTwins Predictor """

def load_model(args): args.ngpus_per_node = torch.cuda.device_count() args.rank = 0 args.dist_url = 'tcp://localhost:58472' args.world_size = args.ngpus_per_node gpu = 0 model = BarlowTwins(args).cuda(gpu) model = nn.SyncBatchNorm.convert_sync_batchnorm(model) torch.distributed.init_process_group( backend='nccl', init_method=args.dist_url, world_size=args.world_size, rank=args.rank) model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[gpu]) best_cp = torch.load("model/barlowtwins/bt_face.pth") # model.module.backbone.state_dict(best_cp) model.load_state_dict(best_cp) model.eval() return model

def predict(t1, t2, model, device): model.to(device) with torch.no_grad(): t1=t1.to(torch.device("cuda:1" if torch.cuda.is_available() else "cpu")) t2=t2.to(torch.device("cuda:2" if torch.cuda.is_available() else "cpu")) out_data = model(t1, t2) return out_data

def predict_img(img_url, args): img = Image.open(img_url) bt_trans = Transform() t1, t2 = bt_trans(img) t1 = t1.unsqueeze(0) t2 = t2.unsqueeze(0) model = load_model(args) device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") output = predict(t1, t2, model, device) return output

if name=="main": parser = argparse.ArgumentParser(description='Barlow Twins Predict') parser.add_argument('--workers', default=8, type=int, metavar='N', help='number of data loader workers') parser.add_argument('--projector', default='8192-8192-8192', type=str, metavar='MLP', help='projector MLP') parser.add_argument("-d", default="data/test/1.jpg") args = parser.parse_args() img_url = str(args.d) result = predict_img(img_url, args) print(result) `

would you please tell me which part is wrong? I'll very hopefully for your answer!

hi, thanks for your contribution, I am guess that the additional parameter scale-loss cannot affect the gradient of loss, so in practice, we can ignore it, please tell me that i am right...

Hi,

I am trying to train a pre-training model on the CIFAR-10 and CIFAR-100 dataset, but I could not achieve high top-1 classification accuracy while evaluating the pre-trained model using KNN. Does anyone have similar experience of using this official implementation for the CIFAR datasets?

Thanks,

are the learned linear classifier weights for ImageNet classification on torch hub? It seems that when I try to use the barlowtwins model there to classify, the performance is random so I'm assuming the fc layers in that model still have the random weights. Could you provide the linear classifier weights?

Thank you!

I'm seeing NaN's being introduced whilst training with the default configuration on ImageNet. The loss spiked suddenly, after which the loss was returned as NaN. This has occurred twice in a row. Similar issues exist in other facebookresearch repos https://github.com/facebookresearch/vissl/issues/543, the common link perhaps being the near identical implementation of the LARS optimiser. The linked issue suggests setting the optimiser to exclude bias and norm to avoid NaNs, however this is different to the published BarlowTwins method.

EDIT: The BarlowTwins paper does in fact state that the bias and batch normalization parameters are excluded from the LARS optimiser and this is the case in the code. Perhaps given the spike, this is an exploding gradients problem?

Loss graph:

Hello,

thanks for your publishing your amazing work. Have you done any investigations on the quality of the embeddings for downstream tasks rather than the representations? I think the SimCLR paper did some, where they showed that the representations were better, but since the embeddings are nicely disentangled in your work, I was wondering if barlowTwins' embeddings might be better, especially when disentanglement is needed?

Greetings

Hi, I added an implementation of the Barlow Twins loss to dlib, and it's now part of dlib, since v19.23.

Again, great paper! It was a pleasure to read and implement.