Running

/gpfsscratch/rech/six/commun/experiments/muennighoff/bloomckpt/6b3t0/tr13f-6b3-ml-t0-lmtoks168b-t0toks13b-prefix on super_glue,copa,None,"best_option" resulted in:

With prefix: Result: {'accuracy': 0.54} Without prefix: Result: {'accuracy': 0.53}

There are a mix of single-token multiple choice and multi-token multiple choice in the prompt dataset. In the run_eval.py code, it appears to only be written for single-token multiple choice. I only see a single call to forward:

https://github.com/bigscience-workshop/t-zero/blob/master/evaluation/run_eval.py#L348

How do you calculate the probability of each multi-token/phrase option? Is that code in this repo?

Thanks.

This script extends run_eval.py to support few-shot evaluation, which naturally also works for non-few-shot full fine-tuning (when num_shots is None, which is actually the default.) Let me know if I should name this few_shot_eval.py or fine_tune.py. (The former was my own intended use, but the latter may be used by more people?)

In theory, this script could be further extended to support training T0 from scratch (that is, from T5 LM Adapt), but more work needs to be done for handling mixtures of datasets and prompts.

I replaced run_eval.py's output JSON with a CSV where each row is an epoch (will add support for eval per x steps soon). I also added optional support for Weights & Biases for loss and metric graphs plus template metadata.

Hello,

What method did you use for truncation in T0?

The experimental setup section of T0 paper states, "We feed the model input and target sequences of 1024 and 256 tokens, respectively." However, I have not been able to find whether truncation is done at the beginning or at the end of texts--or maybe another method is used.

I'm specifically wondering about this in respect to the wikihop original dataset, where many of the inputs are >1024 tokens.

Thank you!

added template_config_name arg so that the dataset and prompt template source can be different, example: prompt from XNLI En but data to eval from XNLI Fr

Hi and thanks for open-sourcing this!

I'm trying to run the evaluation script for the ANLI dataset with the following command:

python run_eval.py \
    --dataset_name anli \
    --template_name "should assume" \
    --model_name_or_path bigscience/T0pp \
    --output_dir ./debug \
    --parallelize

I get the following error:

Traceback (most recent call last):
  File "radt5-dev/evaluate_t0_big_science.py", line 389, in <module>
    main()
  File "radt5-dev/evaluate_t0_big_science.py", line 385, in main
    run(args)
  File "radt5-dev/evaluate_t0_big_science.py", line 289, in run
    preprocess_function, batched=True, remove_columns=column_names
  File "/azureml-envs/azureml_030d8ea9f3a01ad1b81e1990cbdde727/lib/python3.7/site-packages/datasets/dataset_dict.py", line 782, in map
    for k, dataset in self.items()
  File "/azureml-envs/azureml_030d8ea9f3a01ad1b81e1990cbdde727/lib/python3.7/site-packages/datasets/dataset_dict.py", line 782, in <dictcomp>
    for k, dataset in self.items()
  File "/azureml-envs/azureml_030d8ea9f3a01ad1b81e1990cbdde727/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2329, in map
    f"Column to remove {list(filter(lambda col: col not in self._data.column_names, remove_columns))} not in the dataset. Current columns in the dataset: {self._data.column_names}"
ValueError: Column to remove ['train_r1', 'dev_r1', 'test_r1', 'train_r2', 'dev_r2', 'test_r2', 'train_r3', 'dev_r3', 'test_r3'] not in the dataset. Current columns in the dataset: ['uid', 'premise', 'hypothesis', 'label', 'reason']

Is there a known fix for this?

System info:

I used Google Colab free to test the evaluation code.

Reproduction:

!git clone https://github.com/bigscience-workshop/t-zero.git
cd ./t-zero/
pip install -e .
!python ./evaluation/run_eval.py  --dataset_name super_glue --dataset_config_name cb --template_name "GPT-3 style" --model_name_or_path gpt2 --output_dir ./debug --per_device_eval_batch_size 1
!python ./evaluation/run_eval.py  --dataset_name super_glue --dataset_config_name cb --template_name "GPT-3 style" --model_name_or_path gpt2 --output_dir ./debug --per_device_eval_batch_size 2

Expected behavior:

accuracy scores for the two executions of the evaluation script should be the same. i.e. regardless of the batch size, the script should spit out the same number for the accuracy

However, I get Result: {'accuracy': 0.39285714285714285} for the batch size=1, but Result: {'accuracy': 0.4107142857142857}

I suspect this has to do with how the padding is handled in DecoderModel. https://github.com/bigscience-workshop/t-zero/blob/master/t0/model.py#L93

When batch > 1, the shorter texts will be padded to the longest text in the batch. i.e. some elements in batch["input_ids"] will contain pad tokens. Therefore, when concatenating input_ids and labels with "input_ids": torch.cat([batch["input_ids"], batch["labels"]], dim=-1), some of final input to the DecoderModel will look like "T-zero is awesome. Is this true or false? <pad><pad><pad>True". I see that this is supposed to be handled by appropriately setting the position_ids position_ids = torch.cumsum(model_inputs["attention_mask"].to(torch.long), dim=-1) - 1.

What I found was that this pad middle and set position_ids strategy does NOT give the same result as when there is no padding in the middle. Please see: https://colab.research.google.com/drive/1-Bw3-ODDLrEvP75xIzC8wlJmvB7mQqTg?usp=sharing

In a short summary, it looks like the logits for the first token of the label sentence will be different if you have pad tokens in the middle.

Note: When batch==1, there will no padding tokens in batch["input_ids"] since all the sentences in batch["input_ids"] are the same. Therefore, no special handling is done here. So I assume that the value when batch_size=1 gives the correct number.

Hi, I also want to conduct multi-task learning on T5 using serveral datasets, and I notice you use a batch size of 1024 to fine-tune T5. So I want to ask how many steps did you fine-tune it during multi-task learning?

First, thanks for the great work!

Does it make sense to request instructions for fine-tuning a trained t-zero? That would be really great. If it is outside the scope of this repo, feel free to close this issue.

If yes, I would suggest providing the computing power required, thanks!

Fixes: #29

As @kkawamu1 noticed, <pad> between inputs and targets is bad. Right now I've chose to use padding_side="left" as a fix. The current fix does:

<INPUT> <INPUT_PAD> <TARGET> <TARGET_PAD> -> <INPUT_PAD> <INPUT> <TARGET> <TARGET_PAD>

The more permanent fix is to to do:

<INPUT> <INPUT_PAD> <TARGET> <TARGET_PAD> -> <INPUT> <TARGET> <INPUT_PAD> <TARGET_PAD>

This becomes especially useful when you start to leave the text input | text target paradigm so multiple step one.

Hi, thank you for your work.

From what I understand a large portion of the evaluation was done on the big-bench benchmark. How would we run evaluation to reproduce these results? It is unclear from the evaluation README.

Thank you!

Hello Maintainer of Github repo bigscience-workshop/t-zero!

Thank you for your work on bigscience-workshop/t-zero. This GitHub project is interesting, and we think that it would be a great addition to make this work instantly discoverable & available as an API for all your users, to quickly try and use it in their applications.

The list of model card(s) covered by this PR are:

• https://console.tiyaro.ai/explore/bigscience-T0_3B-1756165
• https://console.tiyaro.ai/explore/bigscience-T0pp-1795776

On Tiyaro, every model in bigscience-workshop/t-zero will get its own:

• Dedicated model card (e.g. https://console.tiyaro.ai/explore/bigscience-T0_3B-1756165
• Model demo (e.g. https://console.tiyaro.ai/explore/bigscience-T0_3B-1756165/demo)
• Unique Inference API (e.g. https://api.tiyaro.ai/explore/huggingface/1//bigscience/T0_3B)
• Sample code snippets and swagger spec for the API

Users will also be able to compare your model with other models of similar types on various parameters using Tiyaro Experiments (https://tiyaro.ai/blog/ocr/)

—- I am from Tiyaro.ai (https://tiyaro.ai/). We are working on enabling developers to instantly evaluate, use and customize the world’s best AI. We are constantly working on adding new features to Tiyaro EasyTrain, EasyServe & Experiments, to make the best use of your ML model, and making AI more accessible for anyone.

Sincerely, I-Jong Lin

Hi @VictorSanh

Thanks for releasing the code and data. I am trying to retrain it in pytorch Some questions , in your paper you have p=1 vs p=5.7 results

Say for p=1 we take one random prompt per example of a dataset. This is fine perfectly

I have some doubts about the

1) Sampling strategy: proportional to the number of examples in each dataset (we treated any dataset with over 500'000 examples as having 500'000/num_templates examples) -  
Does this mean for big datasets like gigaword you include  422661 examples instead of  3803957



2) On huggingface T0 it says Fine-tuning steps: 12'200  but in your script says 
export TRAIN_STEPS=1112200. Any idea how many epochs you trained ?



3) Can you tell the total number of samples included for p=1  given tasks ['commonsense_qa', 'dream', 'quail', 'quartz', 'social_i_qa', 'wiqa', 'cosmos_qa', 'qasc', 'quarel', 'sciq', 'wiki_hop', 'adversarial_qa_dbert', 'adversarial_qa_dbidaf', 'adversarial_qa_droberta', 'quoref', 'duorc_ParaphraseRC', 'duorc_SelfRC', 'ropes', 'wiki_qa', 'common_gen', 'wiki_bio', 'app_reviews', 'amazon_polarity', 'imdb', 'rotten_tomatoes', 'gigaword', 'cnn_dailymail', 'multi_news', 'samsum', 'xsum', 'ag_news', 'dbpedia_14', 'trec', 'paws_labeled_final', 'glue_mrpc', 'glue_qqp', 'yelp_review_full', 'kilt_tasks_hotpotqa']

I have Num examples = 3068602 , which was done by taking p=1 from individual datasets , for datasets bigger than 500k dividing num of samples by num_of_prompts. If you have the file for T0 ( p=1 ) or (p=5.7) do you mind sharing them 



4) Example grouping: We use packing to combine multiple training examples into a single sequence to reach the maximum sequence length . Not sure whats this ? Is it necessary and how can we do it ?

The readme says that For reproducibility, we have released an already cached version of the data(https://huggingface.co/datasets/bigscience/P3), which means you don't need to cache the data yourself. The only exception is Story Cloze (https://cs.rochester.edu/nlp/rocstories/). However upon inspecting I see that there are 92 config names that are missing in P3 but present in the seqio.mixture. Here is my code to find out the missing dataset config names:

import datasets
import seqio
import t0.seqio_tasks

ds = datasets.get_dataset_config_names("bigscience/P3")

t0_train_task_names = [task.name for task in seqio.MixtureRegistry.get("t0_train").tasks]
t0p_train_task_names = [task.name for task in seqio.MixtureRegistry.get("t0+_train").tasks]
t0pp_train_task_names = [task.name for task in seqio.MixtureRegistry.get("t0++_train").tasks]
t0_eval_score_eval = [task.name for task in seqio.MixtureRegistry.get("t0_eval_score_eval").tasks]
t0_train_score_eval = [task.name for task in seqio.MixtureRegistry.get("t0_train_score_eval").tasks]

all_tasks = set()
for tasks in [t0_train_task_names, t0p_train_task_names, t0pp_train_task_names, t0_eval_score_eval, t0_train_score_eval]:
    all_tasks.update(tasks)

missing = []
for t in all_tasks:
    if t not in ds:
        missing.append(t)
print(missing)

The missing pieces are: wiki_qa_found_on_google_score_eval quartz_having_read_above_passage_score_eval ag_news_classify_score_eval cosmos_qa_no_prompt_text_score_eval glue_qqp_duplicate_or_not_score_eval wiki_qa_Decide_good_answer_score_eval cosmos_qa_context_description_question_answer_text_score_eval rotten_tomatoes_Writer_Expressed_Sentiment_score_eval qasc_qa_with_separated_facts_1_score_eval sciq_Direct_Question_score_eval qasc_qa_with_separated_facts_2_score_eval quartz_paragraph_question_plain_concat_score_eval cosmos_qa_no_prompt_id_score_eval story_cloze_2016_Novel_Correct_Ending_score_eval wiki_qa_Is_This_True__score_eval quail_context_question_answer_description_id_score_eval quartz_given_the_fact_answer_the_q_score_eval sciq_Multiple_Choice_Question_First_score_eval rotten_tomatoes_Reviewer_Expressed_Sentiment_score_eval quail_context_question_description_answer_id_score_eval qasc_qa_with_separated_facts_3_score_eval yelp_review_full_format_rating_score_eval quail_context_question_answer_description_text_score_eval story_cloze_2016_Story_Continuation_and_Options_score_eval yelp_review_full_on_a_scale_score_eval rotten_tomatoes_Movie_Expressed_Sentiment_2_score_eval yelp_review_full_this_place_score_eval ag_news_which_section_choices_score_eval cos_e_v1.11_question_option_description_text_score_eval glue_mrpc_want_to_know_score_eval cos_e_v1.11_description_question_option_text_score_eval wiki_qa_exercise_score_eval cos_e_v1.11_question_description_option_text_score_eval qasc_qa_with_separated_facts_4_score_eval glue_mrpc_replace_score_eval quail_description_context_question_answer_text_score_eval glue_qqp_meaning_score_eval quail_context_description_question_answer_text_score_eval glue_mrpc_same_thing_score_eval rotten_tomatoes_Reviewer_Opinion_bad_good_choices_score_eval cosmos_qa_context_question_description_answer_text_score_eval quail_no_prompt_id_score_eval cos_e_v1.11_description_question_option_id_score_eval cos_e_v1.11_question_description_option_id_score_eval qasc_qa_with_separated_facts_5_score_eval cosmos_qa_context_description_question_text_score_eval yelp_review_full_format_score_score_eval ag_news_classify_with_choices_score_eval cosmos_qa_context_description_question_answer_id_score_eval quartz_use_info_from_paragraph_question_score_eval quail_description_context_question_answer_id_score_eval cosmos_qa_description_context_question_text_score_eval glue_mrpc_equivalent_score_eval yelp_review_full_based_on_that_score_eval social_i_qa_Show_choices_and_generate_index_score_eval quail_no_prompt_text_score_eval ag_news_which_section_score_eval rotten_tomatoes_Sentiment_with_choices__score_eval quartz_answer_question_below_score_eval social_i_qa_Show_choices_and_generate_answer_score_eval glue_qqp_duplicate_score_eval ag_news_recommend_score_eval story_cloze_2016_Answer_Given_options_score_eval cosmos_qa_description_context_question_answer_text_score_eval ag_news_classify_with_choices_question_first_score_eval quartz_read_passage_below_choose_score_eval quail_context_question_description_answer_text_score_eval wiki_qa_automatic_system_score_eval sciq_Multiple_Choice_score_eval story_cloze_2016_Movie_What_Happens_Next_score_eval quartz_use_info_from_question_paragraph_score_eval yelp_review_full_format_star_score_eval sciq_Direct_Question_Closed_Book__score_eval social_i_qa_I_was_wondering_score_eval ag_news_classify_question_first_score_eval glue_qqp_quora_score_eval yelp_review_full_so_i_would_score_eval story_cloze_2016_Choose_Story_Ending_score_eval quartz_answer_question_based_on_score_eval cos_e_v1.11_question_option_description_id_score_eval social_i_qa_Generate_answer_score_eval rotten_tomatoes_Reviewer_Enjoyment_score_eval rotten_tomatoes_Movie_Expressed_Sentiment_score_eval glue_qqp_same_thing_score_eval quail_context_description_question_answer_id_score_eval rotten_tomatoes_Text_Expressed_Sentiment_score_eval cosmos_qa_description_context_question_answer_id_score_eval rotten_tomatoes_Reviewer_Enjoyment_Yes_No_score_eval rotten_tomatoes_Reviewer_Sentiment_Feeling_score_eval glue_mrpc_paraphrase_score_eval cosmos_qa_context_question_description_answer_id_score_eval social_i_qa_Check_if_a_random_answer_is_valid_or_not_score_eval Could you please shade some light? I'm trying to reproduce (close enough is good) the T0 Training in pytorch. Thank you.

Hope it is okay to add this as an issue - this is amazing work and would love to see the training implementation in pytorch. I see it is in the works, and wondered if this is still the case and whether an ETA is on the horizon?