Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

（1）在根据频次优化时(Frequency Refinement)，体现在代码中，是否为： myverbalizer = KnowledgeableVerbalizer(tokenizer, classes=class_labels, candidate_frac=cutoff, pred_temp=args.pred_temp, max_token_split=args.max_token_split).from_file(f"{args.openprompt_path}/scripts/{scriptsbase}/knowledgeable_verbalizer.{scriptformat}")，其中的cutoff是否为实验中提到的阈值呢？不知道理解的对不对。 （2）在few-shot实验中，为何没有将support set的label值remove掉呢？ 我看论文中提到的是“ . Our proposed Contextualized Calibration utilizes a limited amount of unlabeled support data to yield significantly better results”， 但是，在实验中是，却将其注释掉了，这里有些不解。 **

OSError: Can't load the configuration of '../plm_cache/roberta-large'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '../plm_cache/roberta-large' is the correct path to a directory containing a config.json file

for the up-to-date code of KPT, I get the bellow result on dataset Agnews for Zeroshot:

KPT: Acc: 0.5125 Manual: Acc: 0.6589473684210526

Is it correct ?

我下载解压roberta-large,但是依然提示: OSError: ../plm_cache/roberta-large is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

Thank you for sharing your work. In my understanding, the knowledgeable_verbalizer.json is coming from the "OpenPrompt/scripts/TextClassification/yahoo_answers_topics/knowledgeable_verbalizer.json" (yahoo for instance).

I think the high-quailty knowledgeable_verbalizer is very important. Could you generate knowledgeable_verbalizer.json automanticlly? When I searched "Society & Culture" in https://relatedwords.org/, I found that the returned words are slightly different with the verbalizer in knowledgeable_verbalizer.json of yahoo dataset. So I just want to confirm that knowledgeable_verbalizer.json is from the original returned results from https://relatedwords.org/? How to obatain the relevance scores from https://relatedwords.org/?

Maybe it is an unimportant part in your method. I will be appreciate if you can answer my simple question.

Thanks for your excellent work! As I'm a totally beginer about prompt learning, I want to know how to do the ablation experiment "-RR""-LR" and "PT+CC"

Thanks to anyone who answer my stupid question

test fewshot.py, get the below fatal error.

pytorch 1.10.0

for step, inputs in enumerate(train_dataloader):
            if use_cuda:
                inputs = inputs.cuda()
            logits = prompt_model(inputs)
            labels = inputs['label']
            loss = loss_func(logits, labels)
            loss.backward()  # it causes the fatal error.
            torch.nn.utils.clip_grad_norm_(prompt_model.parameters(), 1.0)
        
            tot_loss += loss.item()
            
            optimizer1.step()

one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [20, 1076]], which is output 0 of SoftmaxBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I have already installed the latest openprompt package. And the script for KPT is

PYTHONPATH=python3
BASEPATH="./"
DATASET=dbpedia #agnews dbpedia imdb amazon yahoo
TEMPLATEID=0 # 1 2 3
SEED=144 # 145 146 147 148
SHOT=0 # 0 1 10 20
VERBALIZER=kpt #
CALIBRATION="--calibration" # ""
FILTER=tfidf_filter # none
MODEL_NAME_OR_PATH="roberta-large"
RESULTPATH="results_zeroshot"
OPENPROMPTPATH="../OpenPrompt"

cd $BASEPATH

CUDA_VISIBLE_DEVICES=0 $PYTHONPATH zeroshot.py \
--model_name_or_path $MODEL_NAME_OR_PATH \
--result_file $RESULTPATH \
--openprompt_path $OPENPROMPTPATH \
--dataset $DATASET \
--template_id $TEMPLATEID \
--seed $SEED \
--verbalizer $VERBALIZER $CALIBRATION \
--filter $FILTER

And I got the error:

##Num of label words for each label: [217, 256, 214, 211, 324, 294, 215, 766, 345, 408, 880, 181, 246, 666]
Traceback (most recent call last):
  File "/mnt/data2/xuandong/prompt/KPT/zeroshot.py", line 113, in <module>
    support_dataloader = PromptDataLoader(dataset=dataset["support"], template=mytemplate, tokenizer=tokenizer,
  File "/mnt/data2/xuandong/prompt/KPT/openprompt/pipeline_base.py", line 100, in __init__
    self.wrap()
  File "/mnt/data2/xuandong/prompt/KPT/openprompt/pipeline_base.py", line 126, in wrap
    wrapped_example = self.template.wrap_one_example(example)
  File "/mnt/data2/xuandong/prompt/KPT/openprompt/prompt_base.py", line 207, in wrap_one_example
    text = self.incorporate_text_example(example)
  File "/mnt/data2/xuandong/prompt/KPT/openprompt/prompt_base.py", line 107, in incorporate_text_example
    text[i] = d["add_prefix_space"] + d.get("post_processing", lambda x:x)(getattr(example, d['placeholder']))
TypeError: 'str' object is not callable

Could you please check about it?