Thanks for the very interesting paper and code. I'm trying to train adapet on custom data (in Google Colab). I've prepared the data in the format as indicated in the documentation. my task is a binary sentiment classification task "positive" vs. "negative".

!python cli.py --data_dir data/sentiment-news-econ \
               --pattern 'The quote: "[TEXT1]". The quote is overall [LBL]' \
               --dict_verbalizer '{"positive": "verbalizer_1", "negative": "verbalizer_2"}'

I get the following error:

Traceback (most recent call last):
  File "cli.py", line 54, in <module>
    train(config)
  File "/content/drive/MyDrive/Colab Notebooks/Few-shot-experiments/ADAPET/src/train.py", line 93, in train
    loss, dict_val_update = model(sup_batch)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/MyDrive/Colab Notebooks/Few-shot-experiments/ADAPET/src/adapet.py", line 213, in forward
    pet_disc_loss = self.get_decoupled_label_loss(batch)
  File "/content/drive/MyDrive/Colab Notebooks/Few-shot-experiments/ADAPET/src/adapet.py", line 178, in get_decoupled_label_loss
    lbl_logits = self.get_single_logits(pet_mask_ids, mask_idx, list_lbl) # [bs, num_lbl]
  File "/content/drive/MyDrive/Colab Notebooks/Few-shot-experiments/ADAPET/src/adapet.py", line 70, in get_single_logits
    lbl_ids[i, :len(i_lbl_ids)] = i_lbl_ids
ValueError: could not broadcast input array from shape (4,) into shape (1,)

The error seems to come from the following part in the source code. Where the token IDs for the labels are retrieved. https://github.com/rrmenon10/ADAPET/blob/6d2bfef5405ca53112235021d9fb8b36f641be9d/src/adapet.py#L70

I couldn't figure out what it causing it though :/

Hi,

I want to use ADAPET on my own data. So, I was trying to follow your recommendation with a toy dataset. I took a small subset of data/BoolQ/train.jsonl and removed the "idx" key, so it looks like my own data.

So, the file lines look like this {"question": "is ghost in the shell based on the anime", "passage": "Ghost in the Shell -- Animation studio Production I.G has produced ....", "label": false} . . . .

I used the command you provided in the README.md as follows: python cli.py --data_dir $data_dir --pattern '"[TEXT1]" has an answer in "[TEXT2]"? "[LBL]"' --dict_verbalizer '{"true": "yes", "false": "no"}'

and that command throws this error: Traceback (most recent call last): File "cli.py", line 52, in <module> train(config) File "~/ADAPET/src/train.py", line 59, in train batcher = Batcher(config, tokenizer, config.dataset) File "~/ADAPET/src/data/Batcher.py", line 21, in __init__ self.dataset_reader = DatasetReader(config, tokenizer, dataset) File "~/ADAPET/src/data/DatasetReader.py", line 44, in __init__ self.dataset_reader = GenericReader(self.config, tokenizer) File "~/ADAPET/src/data/GenericReader.py", line 24, in __init__ self.check_pattern(self.config.pattern) File "~/ADAPET/src/data/GenericReader.py", line 45, in check_pattern raise ValueError("Need at least one text ") ValueError: Need at least one text

I would highly appreciate guiding me on what I am doing wrong.

Thank you!

I am training an ADAPET model on 800gb of text. However, I cannot load this data into a variable since I don't have nearly enough RAM to do that. Is there any way to keep loading the data chunk-by-chunk and training it bit-by-bit? Online learning of some sorts, maybe? How to do this with an ADAPET model?

I am trying to evaluate your fine-tuned model on BoolQ task. I am using this command :

python src/run_pretrained.py -m {finetuned_model_dir}/{task_name} -c config/{task_name}.json -k pattern={best_pattern_for_task}

when trying to run it I get the following warning:

Token indices sequence length is longer than the specified maximum sequence length for this model (737 > 512). Running this sequence through the model will result in indexing errors

I want to know why is this happening? and how can I fix it to get the evaluation results?

Note: I tried with another task (CB), and it is working fine.

For WSC, the Batcher sometimes produces batches of size 0. Do you have any idea why that is? In particular, on this line in WSCReader.py, I sometimes find that len(list_input_ids) == 0.

I don't think this happens with the original FewGLUE WSC data, but it does happen for me on other versions of FewGLUE that I have sampled myself, for example the WSC train file below: train.txt

I have made some additional small changes on top of the ADAPET repo, so it's possible that those changes are causing the issue, but I just wanted to check if you knew if something in the original ADAPET repo that could be causing the issue.

Hi,

In your paper, you tried training the model with multiple patterns at once (C.2), my question is how can I do that with your code, if I want to train a personalised ADAPET.

Thanks!

Thanks for the great repo! two quick questions I am hoping you can help me with.

1. How do I evalutate on the test data for a novel dataset?
2. I do not believe multilabel classification is supported, is that correct?

Thanks so much!

For the model checkpoint I've saved, I'm getting an error when loading them with src/dev.py -- basically the model_state_dict field doesn't appear to exist in the *.pt saved model checkpoint file. The issue is caused by the line below (line number may be different in this repo):

Traceback (most recent call last):
  File "src/dev.py", line 35, in <module>
    model.load_state_dict(torch.load(os.path.join(args.exp_dir, checkpoint))["model_state_dict"])
KeyError: 'model_state_dict'

The model loads successfully if I remove ["model_state_dict"], but just wanted to check that that's the correct fix here.

Does the code automatically use the single pattern that performed best on dev (used to report results in Table 1)? Also, were those patterns the ones used for the SuperGLUE test evaluation (Table 2)?

Hi,

in your paper you trained the sPET and ADAPET models using the same experimental setups, I wonder how did you set sPET's batch number? Setting batch number for ADAPET is straight forward, but if I remember right in PET's code they don't set up for it, by contrast, they have flags for number of train examples and train batch size, and also some settings related to numbers and batch sizes of unlabelled data.

Fan.

Hi,

I am trying to train ADAPET with my own data. If I want to use other pretrained language models, how do I do that? Can I just change the line of pretrained_weight argument in cli.py file?

Thanks.

Hi,

Thanks for sharing this work. I am wondering if there is a simple way to run this code without having a validation set? Also, if I run ADAPET on generic dataset, do you recommend doing hyperparameter search?

Thanks.

The COPA training process fails with the following exception.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/adapet/src/train.py", line 135, in <module>
    train(config)
  File "/workspace/adapet/src/train.py", line 118, in train
    dev_acc, dev_logits = dev_eval(config, model, batcher, batch_idx, dict_avg_val)
  File "/workspace/adapet/src/eval/eval_model.py", line 65, in dev_eval
    eval(config, model, train_iter, train_scorer)
  File "/workspace/adapet/src/eval/eval_model.py", line 28, in eval
    pred_lbl, lbl_logits = model.predict(batch)
  File "/workspace/adapet/src/adapet.py", line 392, in predict
    return self.predict_helper(batch, self.get_pattern())
  File "/workspace/adapet/src/adapet.py", line 364, in predict_helper
    lbl_logits = self.get_eval_multilbl_logits(pet_mask_ids, mask_idx, list_lbl)
  File "/workspace/adapet/src/adapet.py", line 282, in get_eval_multilbl_logits
    tok_pos = torch.min(torch.nonzero(mask_idx[idx] == mask_pos)[0])
TypeError: nonzero() received an invalid combination of arguments - got (bool), but expected (Tensor input, *, bool as_tuple)

It causes the problem that the code checks data type on the wrong variable.