UnifiedSKG
๐
: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Code for paper UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. Please refer to our project page for up-to-date related resources (e.g., papers, code, tools, tutorials) in Structured Knowledge Grounding.
Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they were historically studied in separate by different communities, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UnifiedSKG framework, which unifies 21 SKG tasks into the text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset. We show that large language models like T5, with simple modification when necessary, achieve state-of-the-art performance on all 21 tasks. UnifiedSKG facilitates the investigation of multi-task, zero-shot, and few-shot learning. We demonstrate that multi-task prefix-tuning with UNIFIEDSKG improves the performance on most tasks and show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. UnifiedSKG also enables a series of controlled experiments on structured knowledge encoding variants across SKG tasks. We find that T5โs sensitivity to structured knowledge encoding variations varies across tasks.
UnifiedSKG is easily extensible to more tasks. We encourage researchers to make a pull request to add their datasets, metrics, models to the UnifiedSKG framework!
Updates
- 2022-01-12: We released our code, colab demo, weights and project page. Check it out!
Content
Cloning this repo
In order to include third-party dependencies in this repository, make sure to clone recursively, e.g.:
git clone --recurse-submodules [email protected]:HKUNLP/UnifiedSKG.git
Dependencies
To establish the environment run this code in the shell (the third line is for CUDA11.1):
conda env create -f py3.7pytorch1.8.yaml
conda activate py3.7pytorch1.8new
pip install datasets==1.14.0
# The following line to be replaced depending on your cuda version.
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
That will create the environment py3.7pytorch1.8new we used.
Usage
Environment setup
Activate the environment by running
conda activate py3.7pytorch1.8new
WandB setup
Setup WandB for logging (registration needed):
export WANDB_ENTITY=YOUR_WANDB_USERNAME
export WANDB_API_KEY=YOUR_WANDB_API_KEY
export WANDB_PROJECT=YOUR_PROJECT_NAME
Training
T5-base finetuning on WikiTQ (4 GPUs, 128 effective batch size)
python -m torch.distributed.launch --nproc_per_node 4 --master_port 1234 train.py --seed 2 --cfg Salesforce/T5_base_finetune_wikitq.cfg --run_name T5_base_finetune_wikitq --logging_strategy steps --logging_first_step true --logging_steps 4 --evaluation_strategy steps --eval_steps 500 --metric_for_best_model avr --greater_is_better true --save_strategy steps --save_steps 500 --save_total_limit 1 --load_best_model_at_end --gradient_accumulation_steps 8 --num_train_epochs 400 --adafactor true --learning_rate 5e-5 --do_train --do_eval --do_predict --predict_with_generate --output_dir output/T5_base_finetune_wikitq --overwrite_output_dir --per_device_train_batch_size 4 --per_device_eval_batch_size 16 --generation_num_beams 4 --generation_max_length 128 --input_max_length 1024 --ddp_find_unused_parameters true
If you want to resume training, remove the --overwrite_output_dir flag from the above command:
python -m torch.distributed.launch --nproc_per_node 4 --master_port 1234 train.py --seed 2 --cfg Salesforce/T5_base_finetune_wikitq.cfg --run_name T5_base_finetune_wikitq --logging_strategy steps --logging_first_step true --logging_steps 4 --evaluation_strategy steps --eval_steps 500 --metric_for_best_model avr --greater_is_better true --save_strategy steps --save_steps 500 --save_total_limit 1 --load_best_model_at_end --gradient_accumulation_steps 8 --num_train_epochs 400 --adafactor true --learning_rate 5e-5 --do_train --do_eval --do_predict --predict_with_generate --output_dir output/T5_base_finetune_wikitq --per_device_train_batch_size 4 --per_device_eval_batch_size 16 --generation_num_beams 4 --generation_max_length 128 --input_max_length 1024 --ddp_find_unused_parameters true
T5-base prefix-tuning on WikiTQ (4 GPUs, 128 effective batch size)
python -m torch.distributed.launch --nproc_per_node 4 --master_port 1234 train.py --seed 2 --cfg Salesforce/T5_base_prefix_wikitq.cfg --run_name T5_base_prefix_wikitq --logging_strategy steps --logging_first_step true --logging_steps 4 --evaluation_strategy steps --eval_steps 500 --metric_for_best_model avr --greater_is_better true --save_strategy steps --save_steps 500 --save_total_limit 1 --load_best_model_at_end --gradient_accumulation_steps 8 --num_train_epochs 400 --adafactor true --learning_rate 5e-5 --do_train --do_eval --do_predict --predict_with_generate --output_dir output/T5_base_prefix_wikitq --overwrite_output_dir --per_device_train_batch_size 4 --per_device_eval_batch_size 16 --generation_num_beams 4 --generation_max_length 128 --input_max_length 1024 --ddp_find_unused_parameters true
T5-3b finetuning on WikiTQ (8 GPUs, 128 effective batch size)
deepspeed train.py --deepspeed deepspeed/ds_config_zero2.json --seed 2 --cfg Salesforce/T5_3b_finetune_wikitq.cfg --run_name T5_3b_finetune_wikitq --logging_strategy steps --logging_first_step true --logging_steps 4 --evaluation_strategy steps --eval_steps 500 --metric_for_best_model avr --greater_is_better true --save_strategy steps --save_steps 500 --save_total_limit 1 --load_best_model_at_end --gradient_accumulation_steps 16 --num_train_epochs 50 --adafactor false --learning_rate 5e-5 --do_train --do_eval --do_predict --predict_with_generate --output_dir output/T5_3b_finetune_wikitq --overwrite_output_dir --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --generation_num_beams 4 --generation_max_length 128 --input_max_length 1024 --ddp_find_unused_parameters true
Load weights
See
Code structure overview of UnifiedSKG
.
โโโ configure # Config files for experiments, tasks, and settings
โ โโโ META_TUNING # Config files for tasks and settings
โ โโโ Salesforce # Config files for experiments. We name this diretory as Salesforce to thank Salesforce Research for providing a large number of GPUs. We would like also to thank Amazon Research Awards, ServiceNow Research, and Yale NLP for providing computing resources generously.
โ
โโโ metrics # Code for evaluation
โ โโโ ... # Please check the README of the ./seq2seq_construction.
โโโ models # Code for models
โ โโโ adapter # Code for T5 and BART with adapters (based on HuggingFace Transformers)
โ โโโ prompt # Code for T5 and BART with prefix-tuning (based on HuggingFace Transformers)
โ โโโ unified
โ โโโ base.py # Code for the base model that enables an arbitrary model to be pushed to HuggingFace Model Hub (namely, PushToHubFriendlyModel)
โ โโโ finetune.py # Code for finetuning
โ โโโ adaptertuning.py # Code for adapter-tuning
โ โโโ prefixtuning.py # Code for prefix-tuning
โ
โโโ seq2seq_construction # Code for converting raw data into sequences
โ โโโ ... # Please check the README in this directory.
โ
โโโ tasks # Code for loading raw data
โ โโโ ... # Please check the README in this directory.
โ
โโโ third_party # Packages from third parties
โ โโโ ... # Please check the README in this directory.
โ
โโโ utils # Code for some (probably) useful stuff
โ โโโ processor # Adopted from Tapex: the processor that handles table truncation and linearization
โ โโโ ...
โ โโโ configure.py # Code for parsing config files in ./configure
โ โโโ dataset.py # Code for converting input and output sequences into Datasets for training
โ โโโ tool.py # Code for loading models, seq2seq constructors, and evaluators
โ โโโ trainer.py # Code for EvaluationFriendlyTrainer. If you want make training-specific modifications, you may want to change something here.
โ โโโ training_arguments.py # Code for seq2seq training arguments
โ
โโโ .gitignore
โโโ .gitmodules
โโโ py3.7pytorch1.8.yaml # Anaconda environment config file
โโโ README.md # The README file you are looking at :)
โโโ train.py # Entry code, which controls train, eval, test, storage, and logging
How to unify a new task into the framework?
(README in ./tasks, ./seq2seq_construction, ./metrics, ./configure can also be useful)
-
step 1, Add the "Loader" of raw data in ./tasks, (you can search in huggingface dataset website firstly to find whether there is already a usable script, if not, that's great because you can be the contributor of both this project and huggingface community.
-
step 2, Add the "Wrapper" which construct "seq_in"("user request input" & "structured knowledge input") and "seq_out" from and add to the raw_data for seq2seq unification.
-
step 3, Add the "Evaluator"(for task) in ./metrics. if any third_party repo are used, please add them into .gitmodules.
-
step 3.5(optional), You can always add new "Model" into the ./models/ if you like, change the path in config files to drive new model.
-
step 4, Add the "Config" file to drive your task or all the tasks we have by finetune/multi-task-finetune/pretrain/prefix-tuning/multi-task-prefix-tuning... or other ways.
And this is all for it ! =)
Contributors
71 Oct 25, 2022
6 Dec 25, 2022
118 Dec 5, 2022
51 Oct 8, 2022
45 Dec 12, 2022
52 Dec 19, 2022
112 Dec 31, 2022
22 Dec 11, 2022
108 Dec 27, 2022
76 Dec 24, 2022
67 Dec 20, 2022
21 Nov 22, 2022
84 Jan 4, 2023
47 Jan 9, 2023
4 May 20, 2022
159 Dec 30, 2022
118 Dec 29, 2022
2.2k Jan 8, 2023