Two goals:

• have quantitative measurements for the paper's Broader impact section
• reflect these in the model cards when we release checkpoints

4 held out evaluation sets identified by the Evaluation WG:

• jigsaw_toxicity_pred
• crows_pairs
• winogender (AXG in SuperGLUE)
• winobias

At the current state, I see crows_pairs and winobias prompted, jigsaw_toxicity_pred has an opened PR (#451) we need to check, winogender needs to be prompted.

Workflow:

• [x] prompt those that were not prompted yet
• [x] making sure these were actually cached (@VictorSanh can have this caching step done fairly quickly)
• [x] evaluation (normal and score rank evaluation) or maybe they have some special evaluation?
• [x] when we know which checkpoints exactly to eval, get the final numbers to report

• [x] Remove the if statement to allow English prompts to work across the Dataset.
• [x] Add Arabic prompts for primary task subset with if statement to include only Arabic set.
• [x] Add Arabic prompts for secondary task subset.

Main runs

• D4 only (finetune-t5-xxl-lm-d4-091621-512)
  • [x] Training
  • [x] Eval
    • [x] Last
      • [x] Normal
      • [x] Rank
    • [x] 1'112'200 (half fine-tuning) (cf #456)
      • [x] Normal
      • [x] Rank
    • [x] 1'124'700 (second to last checkpoint)
      • [x] Normal
      • [x] Rank
  • [ ] SuperGLUE test (1380485) - RUNNING
  • [x] BigBench - see #469
• D4 + GPT (finetune-t5-xxl-lm-d4-gpt-091621/)
  • [x] Training
  • [x] Eval
    • [x] Last
      • [x] Normal
      • [x] Rank
    • [x] 1'112'200 (half fine-tuning)
      • [x] Normal
      • [x] Rank
  • [x] BigBench - see #469 - RUNNING
• D4 + GPT + Sglue (finetune-t5-xxl-lm-d4-all-091621)
  • [X] Training
  • [X] Eval
    • [x] Last
      • [x] Normal
      • [x] Rank
    • [x] 1'112'000 (half fine-tuning)
      • [x] Normal
      • [x] Rank
  • [x] BigBench - see #469 - RUNNING

Ablations

• Nb Template - 1 OG Template (finetune-t5-xxl-lm-d4-og-091621)
  • [X] Training
  • [X] Eval
    • [x] Last
      • [x] Normal
      • [x] Rank
    • [x] 1'112'000 (half fine-tuning)
      • [x] Normal
      • [x] Rank
• all OG Templates (finetune-t5-xxl-lm-d4-og-all-091621) #465
  • [x] Training - RUNNING
  • [ ] Eval
    • [ ] Last
      • [ ] Normal
      • [ ] Rank
    • [x] 1'112'000 (half fine-tuning)
      • [ ] Normal ``
      • [x] Rank ``
• Size XL (finetune-t5-xl-lm-d4-091621)
  • [X] Training
  • [X] Eval
    • [x] Last
      • [x] Normal
      • [x] Rank
    • [x] 1'112'000 (half fine-tuning)
      • [x] Normal
      • [x] Rank
• D4 only duplicate (finetune-t5-xxl-lm-d4-091621/)
  • [x] Training
  • [ ] Eval
    • [ ] Last
      • [ ] Normal
      • [ ] Rank
    • [X] 1'112'000 (half fine-tuning)
      • [X] Normal
      • [X] Rank

Baseline

• T5 zero-shot baseline
  • [x] Eval - RUNNING under finetune-t5-xxl-lm-d4-091621
    • [x] Normal 1384362 - RUNNING
    • [x] Rank 1384360 - RUNNING

Since we have the string outputs of all tasks, in principal we should be able to run arbitrary metrics, especially for datasets require fancy metrics.@lintangsutawika has imported the official eval scripts for ReCoRD, SQuAD v2, Natural Questions, TriviaQA, and DROP.

Update: Even when using Lintang's eval scripts, all extractive QAs and closed-book (generative) QAs still have abnormally low numbers, namely:

• [ ] ReCoRD
• [x] SQuAD2 (contains unanswerables)
• [x] DROP
• [ ] CoQA (contains unanswerables, multiple questions per example)
• [ ] Quac (contains unanswerables, multiple questions per example)
• [ ] Natural Questions
• [x] TriviaQA
• [x] WebQuestions

Also, I think all eval of extractive QA from the training mixture also failed.

(Note that ARC is closed-book, but its performance is fine because it's multiple-choice. A great point in case that machine task categories care more about format way more than human skill/knowledge.)

Others with issues to keep an eye on:

• [x] HellaSwag
• [x] Lambada
• [x] Winogrande

The rendering is messed up in a bunch of places. A few issues that are symptomatic: #355 #326

Replace st.markdown by st.text and extensively test that the rendering is better.

This one compared to the others has been a real pain, but I think the templates are quite interesting. Not sure the Jinja style is top notch, but it seems functional on the example sets I checked.

For now, it looks like we can sort of uniquely identify each template using a combination of template name and dataset name, but I'm expecting potential collisions when a lot of people start contributing. Besides, naming each template might not be useful (like if we end up with names like template1 template2 etc...), and it would help contributors if they don't have to add a name/check conflicts on the naming part before merging their template.yaml.

I was thinking that we could add an ID to each entry by getting the hash of timestamp + dataset + string of prompt python function or jinja template? That should be more than enough to prevent collisions

Added a range of prompts for different tasks:

• sentence-level translation
• analogy-based translation
• contextual translation (with context in different languages, automatically vs. manually translated)
• detection of errors in translations
• identification of machine translated output compared to human-produced translations

Hi,

Is anyone facing this issue while running streamlit run promptsource/app.py ?

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\I355109\Anaconda3\envs\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\I355109\promptsource\promptsource\app.py", line 59, in <module>
    state = _get_state()
  File "c:\users\i355109\promptsource\promptsource\session.py", line 84, in _get_state
    session = _get_session()
  File "c:\users\i355109\promptsource\promptsource\session.py", line 74, in _get_session
    session_id = get_report_ctx().session_id
AttributeError: 'NoneType' object has no attribute 'session_id'

Prompt Description:

1. flat_question_with_label : Regular task. Label are normalized label in-case of POS tagging.
2. flat_question_with_random_label : It is not expected that user will always provide labels strictly from the dataset. They may provide a subset of labels. So here we provided subset of labels. If the gold labels in the sample are not available in the subset we replace the gold label with "O". In case of choosing random tags, We always include "O" tag for ner, pos and chunk labels.
3. flat_question_without_label : Regular task. No label is provided.

POS label Normalization

Both NER and Chunk task contains "O" tags. But POS doesn't contain "O" tag.

In case of parts-of-speech tags, there are few labels that are weird in natural sense. For example see a prompt with all pos labels,

Generate parts of speech from the following sentence. The parts of speech tags are ", '', #, $, (, ), ,, ., :, ``, CC, CD, DT, EX, FW, IN, JJ, JJR, JJS, LS, MD, NN, NNP, NNPS, NNS, NN|SYM, PDT, POS, PRP, PRP$, RB, RBR, RBS, RP, SYM, TO, UH, VB, VBD, VBG, VBN, VBP, VBZ, WDT, WP, WP$, WRB

Here first 9 labels are normalized to "O" tag.

Earlier Pull

Earlier zip was not available so I wrote the a brute force code in O(n^2) complexity. But now that zip is available, I wrote the code with simpler notation and loop (with O(n) complexity). While merging I messed up in previous pull https://github.com/bigscience-workshop/promptsource/pull/170 . So I closed that and created the new pull.

• Default sort by popularity and include number of prompts

• Global progress table

• Separated out new prompts and select, and the columns (I think this is okay, but might break)

• Added text wrapping so that you can view long prompts

• Added a template viewer section

I would like to get all xP3 prompts merged into eval-hackathon & then have eval-hackathon be merged into main once & for all, but I'm not sure if you want that? cc @VictorSanh @stephenbach

It would include adding:

• Normal english prompts for various new & existing datasets (many PRs already open - will clean them up & request reviews when ready if someone gives me rights to do so / merge if i can & tests pass)
• Long prompts https://github.com/bigscience-workshop/promptsource/pull/823/files ; Would request reviews once ready
• Human-translated & Machine-translated prompts (Always suffixed with ht or mt, roughly 20K lines; see https://github.com/Muennighoff/promptsource/pull/47/files)

Lmk if you're okay with that & I'll get started 🙂

@KhalidAlt made a great suggestion to change

yaml.dump(self.format_for_dump(), open(self.yaml_path, "w"))

to

yaml.dump(self.format_for_dump(), open(self.yaml_path, "w"), allow_unicode=True)

in templates.py so that the yaml files display as unicode (rather than the unicode code points). We should definitely do this, but we should do it when the corpus is relatively stable and we can regenerate (read and write) all the yaml as one commit.

changes were induced here https://huggingface.co/datasets/trec/commit/1f97567bdd2adedefe8abdaa9bd6ee0e6725b458 to the field names (coarse_label and fine_label)