Hi, thanks for your nice work. However, I meet a problem when I run the code following the readme.

allennlp.common.checks.ConfigurationError: huggingface not in acceptable choices for dataset_reader.type: ['conll2003', 'interleaving', 'sequence_tagging', 'sharded', 'babi', 'text_classification_json']. You should either use the --include-package flag to make sure the correct module is loaded, or use a fully qualified class name in your config file like {"model": "my_module.models.MyModel"} to have it imported automatically.

I do not use the allennlp before, could you help me?

File "/home/hanlin/warp/allennlp/allennlp/common/params.py", line 488, in from_file file_dict = json.loads(evaluate_file(params_file, ext_vars=ext_vars)) File "/home/hanlin/warp/allennlp/allennlp/commands/train.py", line 169, in train_model_from_file params = Params.from_file(parameter_filename, overrides) File "/home/hanlin/warp/allennlp/allennlp/commands/train.py", line 119, in train_model_from_args file_friendly_logging=args.file_friendly_logging, File "/home/hanlin/warp/allennlp/allennlp/commands/init.py", line 119, in main args.func(args) File "/home/hanlin/warp/allennlp/allennlp/main.py", line 34, in run main(prog="allennlp") File "/home/hanlin/warp/allennlp/allennlp/main.py", line 38, in run()

Hi, thanks for your nice work. However, when I ran the follow code you provide: for DATASET in 'cola' 'sst2' 'mrpc' 'qqp' 'stsb' 'mnli' 'rte' 'wnli' 'qnli' do export HPARAMS='{ "dataset": "'$DATASET'", "lr": 0.0001, "num_epochs": 20, "prompts": [], "reorder_optimized": false, "max_batch_size": 8, "max_tokens_sq": 262144, "on_logits": false, "pooling_index": null, "seed": 1}' python3 -m allennlp train
-s .aim/baseline-linear-${DATASET} configs/warp.jsonnet done , I encountered the follow problem: ModuleNotFoundError: No module named 'aim.sdk.session'. And I check the version of package "aim = 3.0.2" and the functions in aim.sdk. I did not find the function "session". Do I get the wrong aim version?

Hi, thanks for your job for the paper and code. I can train the model smoothly. However, when I run

python -m allennlp predict \
  --silent --use-dataset-reader --cuda-device 0 \
  --batch-size 50 \
  --predictor glue --output-file v0.1/MNLI-m.tsv /data/arp/.aim/H-93ae5ae9 test_matched

I'm confused about the parameters on the last line, as well as the data organization, cause the datasets were downloaded automatically to /home/dqx/.cache/huggingface/datasets/glue/mnli/train.arrow，but I'm not clear about the data in tsv format. Are they downloaded manually and just stored under WARP/ ? And how can I evaluate the model on my own testing data? If I upload rte.dev as the testing data to WARP/ and run the following command

python -m allennlp predict \
  --silent --use-dataset-reader --cuda-device 0 \
  --batch-size 50 \
  --predictor super_glue --output-file test_output.tsv .aim/t-rte  dev.tsv

the terminal output will be a list like "[...fixed_cross_validation_79', 'few_wsc.fixed_cross_validation_80', 'few_wsc.fixed_cross_validation_81', 'few_wsc.fixed_cross_validation_82', 'few_wsc.fixed_cross_validation_83', 'few_wsc.fixed_cross_validation_84', 'few_wsc.fixed_cross_validation_85', 'few_wsc.fixed_cross_validation_86', 'few_wsc.fixed_cross_validation_87', 'few_wsc.fixed_cross_validation_88', 'few_wsc.fixed_cross_validation_89', 'few_wsc.fixed_cross_validation_90', 'few_wsc.fixed_cross_validation_91', 'few_wsc.fixed_cross_validation_92', 'few_wsc.fixed_cross_validation_93', 'few_wsc.fixed_cross_validation_94', 'few_wsc.fixed_cross_validation_95', 'few_wsc.fixed_cross_validation_96', 'few_wsc.fixed_cross_validation_97', 'few_wsc.fixed_cross_validation_98', 'few_wsc.fixed_cross_validation_99', 'few_wsc.fixed_cross_validation_100']" but there's nothing in test_output.tsv.

I'm really confused about this, looking forward to your response, thanks😊✨

Hi,

Thank you for your work and for releasing the code! I just have a few questions regarding the ordering of the tokens.

From the script that you shared for training WARP, the prompt should be in the format of [-10,-11,-12,-13,-14,null,-15,-16,-17,-18,-19,[MASK],-20,-21,-22,-23,-24,null,-25,-26,-27,-28,-29]. If I understood correctly, the hypothesis and the premise should be placed in the null position and the other tokens should be considered as the prompt tokens. However, as I looked through the code I found that the input is given as [[MASK], -10,-11,-12,-13,-14,-15,-16,-17,-18,-19,-20,-21,-22,-23,-24,-25,-26,-27,-28,-29, null, null] with the mask token at the first position and the hypothesis and the premise at the end of the prompt. Did I understand the code correctly? Could you clarify the positions of the prompt tokens?

Thanks again! 😄

This PR adds GLUE and SuperGLUE datasets in HuggingFace's datasets format. SuperGLUE also features FewGLUE - a few short version consisting of 32 examples.

Those withs were like a Winograd schema challenge. Let me know if I got the interpretation right

(Look at only the second commit to see the diff of the wording changes without the formatting changes)

Hi,

Thank you for your work and for releasing the code! After reading your paper, I am confused about the manual prompts.

""" In addition to the regular models where we initialize with [MASK] tokens, we performed a run on the GLUE datasets with the same prompt [CLS] "S1"? [MASK]. "S2"! [SEP] for all the tasks """ In the manual prompts, I want to know where to insert the prompts. Wouldn't the original WARP also have [CLS], [SEP] and [MASK] special tokens? What is the difference between WARPinit and WARP8 in the insertion position of prompts?

I don't know much about this field, thank you very much for answering my question

Hi, thanks for your nice work. When I read the source code, I have a simple question for the position id used in the code as follow,

parameters['position_ids'][0]

tensor([ 2, 47,  3,  4,  5,  6,  7, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 89,
        90, 91, 92, 93,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
        22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
        40, 41, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
        69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86,
        87, 88, 94,  1,  1,  1], device='cuda:0')

I find that the position id is not ordered, and what are the benefits of such a position ID