Implementation of soft embeddings from https://arxiv.org/abs/2104.08691v1 using Pytorch and Huggingface transformers
@misc{lester2021power,
title={The Power of Scale for Parameter-Efficient Prompt Tuning},
author={Brian Lester and Rami Al-Rfou and Noah Constant},
year={2021},
eprint={2104.08691},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
learned_embedding will be incorrectly initialized if initialize_from_vocab==False.
The dim of learned_embedding should be (n_tokens, n_embd) while n_embd is the dimensionality of the embeddings and hidden states.
I wonder if the following lines are redundant.
# need to pad attention_mask and input_ids to be full seq_len + n_learned_tokens
# even though it does not matter what you pad input_ids with, it's just to make HF happy
inputs['input_ids'] = torch.cat([torch.full((1,n_tokens), 50256), inputs['input_ids']], 1)
inputs['attention_mask'] = torch.cat([torch.full((1,n_tokens), 1), inputs['attention_mask']], 1)
Everything seems to be working fine without adding those for transformers==4.23.1 in the sense that I am not getting any error. Are they needed for something else though?
Thank you for sharing this great work. I am using a similar code and added soft prompt tuning to encoder. However, my training loss is super strange, I keep getting training loss > 40 (the regular training loss should be small for my case, usually >0.001). Did you have the same issue? Thanks 🙏
Could someone please share some example code for how to generate text using a model with a soft prompt?
I have finetuned a soft prompt model (as implemented in this repo), however when I try to use the .generate(...) method from the Huggingface transformers library, I get an error in the forward pass of the model about mismatched tensor sizes.
In the case of multi task mixing, do different source tasks use different prompt parameters? In the code, there is no such process? Can you briefly describe how to implement it? Thanks for answer my question
I dont find any backpropagation process in the code. I'm curious about how the stochastic embedding be optimized. I wonder Am I misunderstanding the paper or the code is uncompeleted? Thanks for answer my question
T-Few This repository contains the official code for the paper: "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learni
GEP (GDB Enhanced Prompt) GEP (GDB Enhanced Prompt) is a GDB plug-in which make your GDB command prompt more convenient and flexibility. Why I need th
Based on Yolo's low-power, ultra-lightweight universal target detection algorithm, the parameter is only 250k, and the speed of the smart phone mobile terminal can reach ~300fps+
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections Ruiqi Zhong, Kristy Lee*, Zheng Zhang*, Dan Klein EMN
KnowPrompt Code and datasets for our paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction" Requireme
EntLM The source codes for EntLM. Dependencies: Cuda 10.1, python 3.6.5 To install the required packages by following commands: $ pip3 install -r requ
Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular
Black-Box-Tuning Source code for paper "Black-Box Tuning for Language-Model-as-a
"# SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING" i
HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re
NodePiece - Compositional and Parameter-Efficient Representations for Large Knowledge Graphs NodePiece is a "tokenizer" for reducing entity vocabulary
Self-Tuning for Data-Efficient Deep Learning This repository contains the implementation code for paper: Self-Tuning for Data-Efficient Deep Learning
OpenDelta is a toolkit for parameter efficient methods (we dub it as delta tuning), by which users could flexibly assign (or add) a small amount parameters to update while keeping the most paramters frozen. By using OpenDelta, users could easily implement prefix-tuning, adapters, Lora, or any other types of delta tuning with preferred PTMs.
pgmpy pgmpy is a python library for working with Probabilistic Graphical Models. Documentation and list of algorithms supported is at our official sit
hyperopt-sklearn Hyperopt-sklearn is Hyperopt-based model selection among machine learning algorithms in scikit-learn. See how to use hyperopt-sklearn
PGPElib A mini library for Policy Gradients with Parameter-based Exploration [1] and friends. This library serves as a clean re-implementation of the
An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.
extrinsic2pyramid Visualize Camera's Pose Using Extrinsic Parameter by Plotting Pyramid Model on 3D Space Intro A very simple and straightforward modu
?? Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.⭐⭐⭐
220 Dec 31, 2022
23 Dec 21, 2022
567 Dec 26, 2022
42 Nov 3, 2022
137 Dec 31, 2022
77 Dec 27, 2022
18 Dec 17, 2022
149 Jan 4, 2023
28 Dec 12, 2022
189 Nov 22, 2022
107 Jan 4, 2023
101 Dec 11, 2022
386 Dec 26, 2022
2.2k Jan 3, 2023
1.4k Jan 1, 2023
56 Jan 1, 2023
45 Dec 8, 2022
106 Dec 28, 2022
7.7k Jan 5, 2023