Meta Language-Specific Layers in Multilingual Language Models

Zirui Wang

Last update: Feb 13, 2022

Related tags

Deep Learning MetaAdapter

Overview

Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

This is the code of NeurIPS'21 paper "Towards Enabling Meta-Learning from Target Models".

ST This is the code of NeurIPS 2021 paper "Towards Enabling Meta-Learning from Target Models". If you use any content of this repo for your work, plea

7 Dec 6, 2022

Datasets, Transforms and Models specific to Computer Vision

torchvision The torchvision package consists of popular datasets, model architectures, and common image transformations for computer vision. Installat

13.1k Jan 2, 2023

TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

1.3k Dec 30, 2022

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks. Bayesian-Torch is designed to be flexible and seamless in extending a deterministic deep neural network architecture to corresponding Bayesian form by simply replacing the deterministic layers with Bayesian layers.

210 Jan 4, 2023

Improving Deep Network Debuggability via Sparse Decision Layers

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

You might also like...

This is the code of NeurIPS'21 paper "Towards Enabling Meta-Learning from Target Models".

Datasets, Transforms and Models specific to Computer Vision

TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Improving Deep Network Debuggability via Sparse Decision Layers

Unofficial PyTorch implementation of Attention Free Transformer (AFT) layers by Apple Inc.

Spectral Tensor Train Parameterization of Deep Learning Layers

TensorFlow, PyTorch and Numpy layers for generating Orthogonal Polynomials

RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

Owner

Zirui Wang

Deep Text Search is an AI-powered multilingual text search and recommendation engine with state-of-the-art transformer-based multilingual text embedding (50+ languages).

Implementation of "Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-Learner"

XtremeDistil framework for distilling/compressing massive multilingual neural network models to tiny and efficient models for AI at scale

Repository for XLM-T, a framework for evaluating multilingual language models on Twitter data

A library to inspect itermediate layers of PyTorch models.

EMNLP 2021 Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

DEMix Layers for Modular Language Modeling

Code and models used in "MUSS Multilingual Unsupervised Sentence Simplification by Mining Paraphrases".

This repository contains the code, data, and models of the paper titled "XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages" published in Findings of the Association for Computational Linguistics: ACL 2021.

Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.