Hi there,

I am recently rebuilding your work in fairseq. Your model is really impressive.

I am able to rebuild your results in Table 8, with different templates, I can get 78.41 scores on average (RoBERTa_base as backbone model).

However, when I try to reproduce your default method, which is different templates with denoising, the highest score I can get is 78.54 (RoBERTa_base as backbone model).

I tried using either 1) MASK token's representation to represent the template, or 2) cls token's representation to represent the template at the Template Denoising step.

Can you clarify which method you use as the template biases?

Many thanks!

About ./run.sh bert-optiprompt or ./run.sh sup-roberta How do i learn from scratch, readme is bash evaluation only. After downloading the model via sh [unsup-bert|unsup-roberta|sup-berta], I can running ./run.sh sup-roberta. I'd appreciate it if you could give me a hint. I'm fascinated by your thesis.

Hi, I met some questions that when I want to fine-tuned with the command ./run.sh unsup-bert 0 ,there is an error that OSError: Can't load config for 'result/unsup-bert_s0'. I want to know that if I need to download the model under the Results on STS Tasks title? and I am confused that Shouldn't these models be obtained after this command is executed? What should I put in the result directory？ thanks!

Hello, I download the "unsup-bert" model, and evaluate it. I only get 77.80 on avg. Can you add an evaluate procedure in readme or a bash file ?

Hello, first congratualations on your great work.

I want to ask the details about the performance of the unfine-tuned prompt bert using manual prompt, cause I use "The sentence of "[X]" means [MASK]." only get an result of avg 59 on sts.

Did you use template denoising also on unfine-tuned version?

I understand that the concept of contrastive learning represents closer to each other and farther away. When learning unsuperviced contrastive learning, do you use the template to bring the closest closer, but does it include the concept of further away?

Hi, thank u for your excellent work. Please forgive me for asking a simple question. These days, I am training a unsup-bert model. I set random seed=0 and get a standard result with avg score 78.54, which is same as the paper reported (78.54±0.15), both the results are lower than your provided model (78.87). So I wonder if you have updated the training strategy or provide the best model for us.

Hello, I have just read the excellent paper. And I have a question. I want to know if the "frequency" in Figure1 is calculated from the datasets where 「5.1Dataset」 mentioned?

Hello， In the "model.py" , I finded the loss function of supervised training is CrossEntropyLoss. With hard negatives, why not select TripletMarginLoss? Looking forward to your reply, thanks!

I noticed that you add periods during evaluation (in eval.py). I am not so sure if this operation is valid because I didn't find the same operation in SimCSE's evaluation code. I notice results under certain settings drop if I remove this operation.

Hi,

Thanks for your brilliant work!

I have a question about the results. I noticed that only unsupervised models are reported with standard deviation. Why supervised models are not reported with the standard deviation?

Also, would you mind to share about the random seeds that you used?

Thanks a lot!

First of all, thanks for your inspiring paper.

When I read the introduction part of your paper (I haven't read all yet), I wondered one thing.

In the paper, removing biased tokens shows improvements. However, it doesn't really have any relationships with anisotropy?

I guess tokens which appear frequently in one sentence, have high probabilities that such tokens appear in other sentences. So, I think removing such tokens makes sentence embeddings distinguished each other. And finally, I expect it makes sentence embeddings have different directions and be isotropic.

My opinion can be wrong ! I am just curious of your opinion.

Thank you.

Hi, I'm trying to adopt Prompt-Bert into the public Chinese dataset.

using prompt like following: cls下面这句话"sent_0"对应的语义是mask。sep+

The spearman score is much worse than the rare SimCSE.

Is there any advice for the Chinese prompts?

Hello,

I was very fortunate to read your paper, and the experimental results are exciting. The paper mentions two observations: Observation 1: Original BERT layers fail to improve the performance. Observation 2: Embedding biases harm the sentence embeddings performance. Based on your experimental results, these two phenomena do exist and can be improved. But I don't see a connection between these two observations and prompts.

How does prompt solve the bias problem?

Looking forward to your reply, thanks!