Datasets from Instructions (DINO
🦕
)
This repository contains the code for Generating Datasets with Pretrained Language Models. The paper introduces a method called Datasets from Instructions (DINO
🔧
Setup
All requirements for DINO can be found in requirements.txt. You can install all required packages in a new environment with pip install -r requirements.txt.
💬
CLI Usage
Single Texts
To generate datasets for (single) text classification, you can use DINO as follows:
python3 dino.py \
--output_dir <OUTPUT_DIR> \
--task_file <TASK_FILE> \
--num_entries_per_label <N>
where <OUTPUT_DIR> is a directory to which the generated dataset is written, <TASK_FILE> is a JSON file containing a task specification (see Task Specs), and <N> is the number of examples to generate per label. To get an overview of additional parameters, run python3 dino.py --help.
Text Pairs
To generate datasets for text pair classification, you first need a dataset of raw input texts (which you can also generate using DINO). You can then run
python3 dino.py \
--output_dir <OUTPUT_DIR> \
--task_file <TASK_FILE> \
--input_file <INPUT_FILE> \
--input_file_type <INPUT_FILE_TYPE> \
--num_entries_per_input_and_label <N>
with <OUTPUT_DIR> and <TASK_FILE> as before. <INPUT_FILE> refers to the file containing raw input texts, <INPUT_FILE_TYPE> specifies its type, which should be one of
plain: for a plain text file with one input text per linejsonl: for a dataset file generated by DINO in a previous step
and <N> is the number of examples to generate per label and input text.
📋
Task Specs
To generate a dataset for a task, you need to provide a file containing a task specification, containing (among other things) the instructions given to the pretrained language model. A task specification is a single JSON object that looks like this:
{
"task_name": "<TASK_NAME>",
"labels": {
"<LABEL_1>": {
"instruction": "<INSTRUCTION_1>",
"counter_labels": [<COUNTER_LABELS_1>]
},
...,
"<LABEL_n>": {
"instruction": "<INSTRUCTION_n>",
"counter_labels": [<COUNTER_LABELS_n>]
}
}
}
Here, <TASK_NAME> is the name for the task and <LABEL_1>, ..., <LABEL_n> are the task's labels. For each label <LABEL_i>, <INSTRUCTION_i> is the instruction provided to the language model for generating examples with label <LABEL_i> (see Writing Instructions). You can additionally specify a list of counter labels <COUNTER_LABELS_n> for each label. This tells the model to generate outputs that are not only likely given the current label, but also unlikely given all counter labels (see the paper for details).
Examples
You can find two examples of task specifications in /task_specs:
sts.jsonis a task specification for generating a semantic textual similarity dataset if a set of raw input texts is already given.sts-x1.jsonis a task specification for generating a set of raw input texts. This set can then be used in a subsequent step to generate a full STS dataset usingsts.json.
Writing Instructions
When writing instructions for a new task, you should consider the following things:
- Always end your instructions with an (opening) quotation mark (
"). This is required because it allows us to interpret the next quotation mark generated by the language model as a signal that it is done generating an example. - For good results, keep the instructions as short and simple as possible as this makes it easier for a pretrained language model to understand them.
- If you are writing instructions for a text pair classification task, make sure that each instruction contains the placeholder
<X1>exactly once. At this position, the provided raw input sentences are inserted during generation.
An example for an instruction that prompts the model to generate a positive review for a restaurant would be:
Task: Write a review for a really great restaurant.
Review: "
An example for an instruction that prompts the model to generate a sentence that has the same meaning as another given sentence would be:
Task: Write two sentences that mean the same thing.
Sentence 1: "<X1>"
Sentence 2: "
🦕
Generated DINOs
In this section, we will soon make publicly available a list of datasets that we have generated using DINO.
📕
Citation
If you make use of the code in this repository or of any DINO-based dataset, please cite the following paper:
@article{schick2020generating,
title={Generating Datasets with Pretrained Language Models},
author={Timo Schick and Hinrich Schütze},
journal={Computing Research Repository},
volume={arXiv:2104.07540},
url={https://arxiv.org/abs/2104.07540},
year={2021}
}
189 Jan 2, 2023
79 Dec 27, 2022
37 Dec 4, 2022
1 Oct 5, 2021
10 Dec 19, 2022
74 Dec 13, 2022
1 Dec 22, 2021
357 Dec 24, 2022
12 Jan 20, 2022
19 Feb 17, 2022
1.1k Dec 27, 2022
1 Nov 19, 2021
52 Nov 10, 2022
0 Oct 25, 2022
42 Dec 13, 2022
49 Aug 24, 2022
684 Jan 9, 2023
22 Nov 13, 2022