Meta Representation Transformation for Low-resource Cross-lingual Learning

Related tags

Deep Learning MetaXL
Overview

MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning

This repo hosts the code for MetaXL, published at NAACL 2021.

[MetaXL: Meta Representation Transformation for Low- resource Cross-lingual Learning] (https://arxiv.org/pdf/2104.07908.pdf)

Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, Ahmed Hassan Awadallah

NAACL 2021

MetaXL is a meta-learning framework that learns a main model and a relatively small structure, called representation transformation network (RTN) through a bi-level optimization procedure with the goal to transform representations from auxiliary languages such that it benefits the target task the most.

Data

Please download [WikiAnn] (https://github.com/afshinrahimi/mmner), [MARC] (https://registry.opendata.aws/amazon-reviews-ml/), [SentiPers] (https://github.com/phosseini/sentipers) and [Sentiraama] (https://ltrc.iiit.ac.in/showfile.php?filename=downloads/sentiraama/) on its corresponding. Please refer to data/data_index.txt for data splits.

Scripts

The following script shows how to run metaxl on the named entity recognition task on Quechua.

python3 mtrain.py \
      --data_dir data_dir \
      --bert_model xlm-roberta-base \
      --tgt_lang qa \
      --task_name panx \
      --train_max_seq_length 200 \
      --max_seq_length 512 \
      --epochs 20 \
      --batch_size 10 \
      --method metaxl \
      --output_dir output_dir \
      --warmup_proportion 0.1 \
      --main_lr 3e-05 \
      --meta_lr 1e-06 \
      --train_size 1000\
      --target_train_size 100 \
      --source_languages en \
      --source_language_strategy specified \
      --layers 12 \
      --struct perceptron \
      --tied  \
      --transfer_component_add_weights \
      --tokenizer_dir None \
      --bert_model_type ori \
      --bottle_size 192 \
      --portion 2 \
      --data_seed 42  \
      --seed 11 \
      --do_train  \
      --do_eval

The following script shows how to run metaxl on the sentiment analysis task on fa.

python3 mtrain.py  \
		--data_dir data_dir \
		--task_name sent \
		--bert_model xlm-roberta-base \
		--tgt_lang fa \
		--train_max_seq_length 256 \
		--max_seq_length 256 \
		--epochs 20 \
		--batch_size 10 \
		--method metaxl \
		--output_dir ${output_dir} \
		--warmup_proportion 0.1 \
		--main_lr 3e-05 \
		--meta_lr 1e-6 \
		--train_size 1000 \
		--target_train_size 100 \
		--source_language_strategy specified  \
		--source_languages en \
		--layers 12 \
		--struct perceptron \
		--tied  \
		--transfer_component_add_weights \
		--tokenizer_dir None  \
		--bert_model_type ori  \
		--bottle_size 192  \
		--portion 2 	\
		--data_seed 42 \
		--seed 11  \
		--do_train  \
		--do_eval

Citation

If you find MetaXL useful, please cite the following paper

@inproceedings{xia2021metaxl,
  title={MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning},
  author={Mengzhou, Xia and Zheng, Guoqing and Mukherjee, Subhabrata and Shokouhi, Milad and Newbig, Graham and Awadallah, Ahmed Hassan},
  journal={NAACL},
  year={2021},
}

This repository is released under MIT License. (See LICENSE)

Comments
  • Sentiment Classification - Clarification

    Sentiment Classification - Clarification

    Thank you team for sharing the wonderful approach and repo. While experimenting the existing implementation for sentiment analysis meta learning for en and te languages, I came across the SequenceTagger object which inherits XLMRobertaForTokenClassification class. As per my understanding, it should be XLMRobertaForSequenceClassification for text classifications like Sentiment, Intent. Where as XLMRobertaForTokenClassification, would be apt for NER use cases.

    Please correct me if I am misunderstood.

    opened by ashissamal 2
  • Could you add more details on data preparation?

    Could you add more details on data preparation?

    I could see that the training code is looking for some file which is in json format. Could you provide details with respect to the structure in which the data has to be sent and if possible a example.

    opened by mahesh-kumar-nlp 2
  • Format of Data Directory

    Format of Data Directory

    Can you please provide the data directory structure which we provide as input in the command line argument?

    In one of the issues you have mentioned how the json files should be formatted but I am unable to find structure of the directory. I am using Sentiraama Product Review dataset and MARC dataset.

    PS: I am trying to train on Sentiment Analysis (Target Language: Telugu, Source Language: English). It would be great if you can provide the input directory structure for this task.

    Thanks in advance.

    opened by charansingh9 0
  • some code can not find definitions in utils.py

    some code can not find definitions in utils.py

    hi , Thanks for sharing , i have downloaded the code and tried . Unfortunately , i can not find definition of variable 'a' in utils.py. details in below: Can you help me explain my question ? Much appreciated.

    _for text in texts:
if text in a.text.values.tolist():
        index.append(a.text.values.tolist().index(text))_
    opened by joekerrXun 1
  • Question about detach()

    Question about detach()

    Hi, thank you for sharing all the codes.

    I have a question about bilevel optimization procedure.

    why do we need detach() operation here?

    As far as I understand the equation (3), we need to update the whole parameter of \theta. But if we detach the hidden state, then we consider it as constant, which means we cannot update the lower layer of BERT or Roberta.

    Thank you.

    opened by seanie12 1
Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
GitHub
CVPR 2021 Official Pytorch Code for UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

UC2 UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu,

Mingyang Zhou 28 Dec 30, 2022
Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Punctuation Restoration using Transformer Models This repository contins official implementation of the paper Punctuation Restoration using Transforme

Tanvirul Alam 142 Jan 1, 2023
Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.

One model to speak them all ?? Audio Language Text ▷ Chinese 人人生而自由，在尊严和权利上一律平等。 ▷ English All human beings are born free and equal in dignity and rig

Mutian He 60 Nov 14, 2022
Code and data for ACL2021 paper Cross-Lingual Abstractive Summarization with Limited Parallel Resources.

Multi-Task Framework for Cross-Lingual Abstractive Summarization (MCLAS) The code for ACL2021 paper Cross-Lingual Abstractive Summarization with Limit

Yu Bai 43 Nov 7, 2022
Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021)

L1-Refinement Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021) ?? A more detailed readme is co

Lincedo Lab 4 Jun 9, 2021
Code for ACL2021 paper Consistency Regularization for Cross-Lingual Fine-Tuning.

xTune Code for ACL2021 paper Consistency Regularization for Cross-Lingual Fine-Tuning. Environment DockerFile: dancingsoul/pytorch:xTune Install the f

Bo Zheng 42 Dec 9, 2022
This is the official implementation of "One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval".

CORA This is the official implementation of the following paper: Akari Asai, Xinyan Yu, Jungo Kasai and Hannaneh Hajishirzi. One Question Answering Mo

Akari Asai 59 Dec 28, 2022
This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

This reporistory contains the test-dev data of the paper "xGQA: Cross-lingual Visual Question Answering".

AdapterHub 18 Dec 9, 2022
PyTorch original implementation of Cross-lingual Language Model Pretraining.

XLM NEW: Added XLM-R model. PyTorch original implementation of Cross-lingual Language Model Pretraining. Includes: Monolingual language model pretrain

Facebook Research 2.7k Dec 27, 2022
Code and data for ACL2021 paper Cross-Lingual Abstractive Summarization with Limited Parallel Resources.

Multi-Task Framework for Cross-Lingual Abstractive Summarization (MCLAS) The code for ACL2021 paper Cross-Lingual Abstractive Summarization with Limit

Yu Bai 43 Nov 7, 2022
For AILAB: Cross Lingual Retrieval on Yelp Search Engine

Cross-lingual Information Retrieval Model for Document Search Train Phase CUDA_VISIBLE_DEVICES="0,1,2,3" \ python -m torch.distributed.launch --nproc_

Chilia Waterhouse 104 Nov 12, 2022
Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

Liyan 5 Dec 7, 2022
EMNLP 2021 paper Models and Datasets for Cross-Lingual Summarisation.

This repository contains data and code for our EMNLP 2021 paper Models and Datasets for Cross-Lingual Summarisation. Please contact me at [email protected]

null 9 Oct 28, 2022
Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

WECHSEL Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. arXiv: https://arx

Institute of Computational Perception 45 Dec 29, 2022
[CVPR'21] Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation

Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-view Transformation Weixiang Yang, Qi Li, Wenxi Liu, Yuanlong Yu, Y

null 118 Dec 26, 2022
Implementation of "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner"

Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner This repository is the official implementation of Meta-rPPG: Remote Heart Ra

Eugene Lee 137 Dec 13, 2022
Code for HLA-Face: Joint High-Low Adaptation for Low Light Face Detection (CVPR21)

HLA-Face: Joint High-Low Adaptation for Low Light Face Detection The official PyTorch implementation for HLA-Face: Joint High-Low Adaptation for Low L

Wenjing Wang 77 Dec 8, 2022
Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

R2RNet Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network." Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu

null 77 Dec 24, 2022
A resource for learning about deep learning techniques from regression to LSTM and Reinforcement Learning using financial data and the fitness functions of algorithmic trading

A tour through tensorflow with financial data I present several models ranging in complexity from simple regression to LSTM and policy networks. The s

null 195 Dec 7, 2022