An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Ramin Nakhli

Last update: Dec 4, 2022

Related tags

Deep Learning Decoupled-Contrastive-Learning

Overview

Decoupled-Contrastive-Learning

This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Requirements

Pytorch
Numpy

Usage Example

import torch
import torchvision.models as models

from loss import dcl

resnet18 = models.resnet18()
random_input = torch.rand((10, 3, 244, 244))
output = resnet18(random_input)

# for DCL
loss_fn = dcl.DCL(temperature=0.5)
loss = loss_fn(output, output)  # loss = tensor(-0.2726, grad_fn=
   

# for DCLW
loss_fn = dcl.DCLW(temperature=0.5, sigma=0.5)
loss = loss_fn(output, output)  # loss = tensor(38.8402, grad_fn=
   
    )

Results

Will be added shortly.

Comments

Implementation of DCLW

Hi,

I saw your implementation of the nominator of DCLW weight_fn, which uses element multiplication for the z1 and z2 calculation. But in the paper, the nominator of DCLW weight_fn formula is: exp(<z1, z2> / sigma). Can you tell me why you use element multiplication for the z1 and z2 calculation instead of torch.mm(z1, z2) or dot multiplication?

Thanks.

opened by wqtwjt1996 2
About the optimization setting

Hello, Thanks for your good work! I'd like to know if the cosine annealing schedule is also applied to the small-scale dataset experiments of CIFAR and STL10?

opened by YEARNLL 2
Was trying to work with the formula

Hi there,

Great work! I am trying to learn from the formula and walk through the proof of propositions. I was trying to find the derivatives of the loss function and encountered finding the partial derivative of the cosine similarity term. By comparing my answer and steps in the paper I kind of get this result (as the image shows).

I am not quite sure if that is correct and how that works. Would you mind help me a bit with this?

opened by mikelmh025 2

DCLW bug?

In DCLW, your code is like:

weight_fn = lambda z1, z2: 2 - z1.size(0) * torch.nn.functional.softmax((z1 * z2).sum(dim=1) / sigma, dim=0).squeeze()

I think the right way shall be like:

weight_fn = lambda z1, z2: 2 - torch.nn.functional.softmax((z1 * z2).sum(dim=1) / sigma, dim=0).squeeze()

z1.size(0) is not a variable introduced in the origin paper.

What do you think of it?

opened by tangzhy 2

Follow-up Issue of DCLW Implementation
Hi @raminnakhli Thanks for the reply.

As you can see from the two pictures, the formulas are similar which all include: exp(<z_1, z_2>). But in your code, for formula(5) (6) you use matrix multiplication, and for the formula of w(z_1, z_2), you use element multiplication.

Could you please explain to me why? Thanks!

Originally posted by @wqtwjt1996 in https://github.com/raminnakhli/Decoupled-Contrastive-Learning/issues/10#issuecomment-1293916353
opened by wqtwjt1996 1

An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Related tags

Overview

Decoupled-Contrastive-Learning

Requirements

Usage Example

Results

Comments

Implementation of DCLW

About the optimization setting

Was trying to work with the formula

DCLW bug?

Follow-up Issue of DCLW Implementation

Owner

Ramin Nakhli

SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING - The Facebook paper about fine tuning RoBERTa with contrastive loss

The official implementation of the CVPR2021 paper: Decoupled Dynamic Filter Networks

Code for paper Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting

Implement Decoupled Neural Interfaces using Synthetic Gradients in Pytorch

Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.

Implementation of the method proposed in the paper "Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation"

Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

A PyTorch implementation of Mugs proposed by our paper "Mugs: A Multi-Granular Self-Supervised Learning Framework".

PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Implementation of "A Deep Learning Loss Function based on Auditory Power Compression for Speech Enhancement" by pytorch

Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

Code and data of the Fine-Grained R2R Dataset proposed in paper Sub-Instruction Aware Vision-and-Language Navigation

PyTorch implementation for Partially View-aligned Representation Learning with Noise-robust Contrastive Loss (CVPR 2021)

Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

Multi-scale discriminator feature-wise loss function

clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation

HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images