Meta learning algorithms to train cross-lingual NLI (multi-task) models

M.Hassan Mojab

Last update: Nov 20, 2022

Related tags

Text Data & NLP MetaNLI

Overview

MetaNLI

Meta learning algorithms to train cross-lingual NLI (multi-task) models

Train (source task)

Reptile

To train the model using Reptile algorithm, run the command below:

python reptile.py \
    --meta_tasks sc_en,sc_de,sc_es,sc_fr \
    --queue_len 4 \
    --temp 5.0 \
    --epochs 1 \
    --meta_lr 1e-5 \
    --scheduler \
    --gamma 0.5 \
    --step_size 4000 \
    --shot 4 \
    --meta_iteration 8000 \
    --log_interval 300

Prototypical

To train the model using Prototypical Networks algorithm, run the command below:

python prototype.py \
    --meta_tasks sc_en,sc_de,sc_es,sc_fr \
    --target_task sc_fa \
    --epochs 1 \
    --meta_lr 1e-5 \
    --lambda_1 1 \
    --lambda_2 1 \
    --scheduler \
    --gamma 0.5 \
    --step_size 1000 \
    --shot 8 \
    --query_num 0 \
    --target_shot 8 \
    --meta_iteration 2500 \
    --log_interval 50

Zero-shot Test (on target task)

To perform a zero-shot test of the trained model on the target task, run the command below:

python zeroshot.py \
    --load saved/model_sc.pt \
    --task sc_fa

Fine-tune (target task)

To fine-tune the trained model on the target task, run the command below:

python finetune.py \
    --save saved \
    --model_filename fine.pt \
    --load saved/model_sc.pt \
    --task sc_fa \
    --epochs 5 \
    --lr 1e-5

You might also like...

PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

PhoNLP is a multi-task learning model for joint part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing. Experiments on Vietnamese benchmark datasets show that PhoNLP produces state-of-the-art results, outperforming a single-task learning approach that fine-tunes the pre-trained Vietnamese language model PhoBERT for each task independently.

109 Dec 2, 2022

A simple recipe for training and inferencing Transformer architecture for Multi-Task Learning on custom datasets. You can find two approaches for achieving this in this repo.

multitask-learning-transformers A simple recipe for training and inferencing Transformer architecture for Multi-Task Learning on custom datasets. You

48 Jan 2, 2023

simpleT5 is built on top of PyTorch-lightning⚡️ and Transformers🤗 that lets you quickly train your T5 models.

Quickly train T5 models in just 3 lines of code + ONNX support simpleT5 is built on top of PyTorch-lightning ⚡️ and Transformers 🤗 that lets you quic

220 Dec 30, 2022

IMS-Toucan is a toolkit to train state-of-the-art Speech Synthesis models

IMS-Toucan is a toolkit to train state-of-the-art Speech Synthesis models. Everything is pure Python and PyTorch based to keep it as simple and beginner-friendly, yet powerful as possible.

Digital Phonetics at the University of Stuttgart

247 Jan 5, 2023

Codes to pre-train Japanese T5 models

t5-japanese Codes to pre-train a T5 (Text-to-Text Transfer Transformer) model pre-trained on Japanese web texts. The model is available at https://hug

37 Dec 25, 2022

PyTorch implementation of "data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language" from Meta AI

data2vec-pytorch PyTorch implementation of "data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language" from Meta AI (F

105 Jan 4, 2023

Meta learning algorithms to train cross-lingual NLI (multi-task) models

Related tags

Overview

MetaNLI

Train (source task)

Reptile

Prototypical

Zero-shot Test (on target task)

Fine-tune (target task)

You might also like...

PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

A simple recipe for training and inferencing Transformer architecture for Multi-Task Learning on custom datasets. You can find two approaches for achieving this in this repo.

simpleT5 is built on top of PyTorch-lightning⚡️ and Transformers🤗 that lets you quickly train your T5 models.

IMS-Toucan is a toolkit to train state-of-the-art Speech Synthesis models

Codes to pre-train Japanese T5 models

PyTorch implementation of "data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language" from Meta AI

Task-based datasets, preprocessing, and evaluation for sequence models.

A design of MIDI language for music generation task, specifically for Natural Language Processing (NLP) models.

Official implementation of Meta-StyleSpeech and StyleSpeech

Owner

M.Hassan Mojab

Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources (NAACL-2021).

PyTorch original implementation of Cross-lingual Language Model Pretraining.

This repository contains examples of Task-Informed Meta-Learning

A multi-lingual approach to AllenNLP CoReference Resolution along with a wrapper for spaCy.

An implementation of model parallel GPT-3-like models on GPUs, based on the DeepSpeed library. Designed to be able to train models in the hundreds of billions of parameters or larger.

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

PyTorch Implementation of Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation

Using Bert as the backbone model for lime, designed for NLP task explanation (sentence pair text classification task)

PyTorch implementation and pretrained models for XCiT models. See XCiT: Cross-Covariance Image Transformer