"# SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING"
in this code, I've implemented sentiment analysis task with sst-2 dataset.
the below results are for 100 training samples:
cross entropy loss:
cross entropy + contrastive loss:
cross entropy heatmap on test dataset:
Accuracy on test dataset: 90.13
cross entropy + contrastive loss heatmap:
Accuracy on test dataset: 92.20
Adaptive Segmentation Mask Attack This repository contains the implementation of the Adaptive Segmentation Mask Attack (ASMA), a targeted adversarial
t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that
InversePrompting Paper: Controllable Generation from Pre-trained Language Models via Inverse Prompting Code: The code is provided in the "chinese_ip"
Knowledge-Inheritance Source code paper: Knowledge Inheritance for Pre-trained Language Models (preprint). The trained model parameters (in Fairseq fo
Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization [Paper] accepted at the EMNLP 2021: Vision Guided Genera
Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses
Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin
UniSpeech The family of UniSpeech: UniSpeech (ICML 2021): Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR UniSpeech-
# Compute loss if scl == True: cross_loss = loss_fn(logits, b_labels) contrastive_l = contrastive_loss(tem, hiden_state.cpu().detach().numpy(), b_labels) loss = (lam * contrastive_l) + (1 - lam) * (cross_loss) if scl == False: loss = loss_fn(logits, b_labels)
when calculating contrastive_loss, you use "detach". Can gradients be updated？
53 Jul 4, 2022
1 Dec 13, 2021
101 Dec 16, 2022
31 Nov 19, 2022
42 Jan 7, 2023
22 Dec 8, 2022
32 Sep 21, 2022
11 Jul 28, 2022
282 Jan 9, 2023
for i in range(len(embedding)):
n_i = label.tolist().count(label[i]) - 1
inner_sum = 0
# calculate inner sum
for j in range(len(embedding) - 1):
if label[i] == label[j]:
inner_sum = inner_sum + np.log(dis[i][j] / row_sum[i])
if n_i != 0:
contrastive_loss += (inner_sum / (-n_i))
else:
contrastive_loss += 0
for example, for label=[1,2,0,1,2]
tensor([[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 1.],
[0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.]])
when i=1,j=4,inner_sum should add a number,but j is less then 4.
ERICA Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive L
CLIP4CMR A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval The original data and pre-calculate
Decoupled-Contrastive-Learning This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper. Requirements P
Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval This repository contains source code and pre-trained/fine-tun
CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi
Black-Box-Tuning Source code for paper "Black-Box Tuning for Language-Model-as-a
Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a
PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's
DziriBERT DziriBERT is the first Transformer-based Language Model that has been pre-trained specifically for the Algerian Dialect. It handles Algerian
75 Nov 2, 2022
9 Jan 12, 2022
71 Dec 4, 2022
18 Dec 17, 2022
65 Dec 26, 2022
17 Mar 10, 2022
149 Jan 4, 2023
1.1k Dec 24, 2022
1.4k Jan 5, 2023
117 Jan 7, 2023