Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Last update: Jan 1, 2023

Overview

Pretrained Language Model

This repository provides the latest pretrained language models and its related optimization techniques developed by Huawei Noah's Ark Lab.

Directory structure

PanGu-α is a Large-scale autoregressive pretrained Chinese language model with up to 200B parameter. The models are developed under the MindSpore and trained on a cluster of Ascend 910 AI processors.
NEZHA-TensorFlow is a pretrained Chinese language model which achieves the state-of-the-art performances on several Chinese NLP tasks developed under TensorFlow.
NEZHA-PyTorch is the PyTorch version of NEZHA.
NEZHA-Gen-TensorFlow provides two GPT models. One is Yuefu (乐府), a Chinese Classical Poetry generation model, the other is a common Chinese GPT model.
TinyBERT is a compressed BERT model which achieves 7.5x smaller and 9.4x faster on inference.
TinyBERT-MindSpore is a MindSpore version of TinyBERT.
DynaBERT is a dynamic BERT model with adaptive width and depth.
BBPE provides a byte-level vocabulary building tool and its correspoinding tokenizer.
PMLM is a probabilistically masked language model. Trained without the complex two-stream self-attention, PMLM can be treated as a simple approximation of XLNet.
TernaryBERT is a weights ternarization method for BERT model developed under PyTorch.
TernaryBERT-MindSpore is the MindSpore version of TernaryBERT.
HyperText is an efficient text classification model based on hyperbolic geometry theories.
BinaryBERT is a weights binarization method using ternary weight splitting for BERT model, developed under PyTorch.
AutoTinyBERT provides a model zoo that can meet different latency requirements.

Comments

$Question towards TinyBERT Data Augmentation ${GLOVE_EMB}$$

Question towards TinyBERT Data Augmentation ${GLOVE_EMB}$

Hi, all

In the part of Data Augmentation I have seen “--glove_embs ${GLOVE_EMB}$”, I am wondering what should I use to replace this part: "${GLOVE_EMB}$"

I have noticed from the code in the data_augmentation.py, it mentioned it is the glove embedding file. If we should replace "${GLOVE_EMB}$" with the location of the glove embedding file.

May I know where can we get the glove embedding file? Could you provide me with a link?

opened by MichaelCaohn 6
Does Ternary BERT only use KD Loss(Teacher, Student Loss) while training?

Hi, Thanks for this great source code. It really helps me a lot!

While I'm studying the TernaryBERT with Paper and this source code, I have a question about KD Training Loss. In Paper Algorithm1, It says that when compute the gradient, It only uses Distillation Loss, not the Distillation Loss + GT Label Cross Entropy Loss.

And also in source code, there is only KD loss which is used for backward. https://github.com/huawei-noah/Pretrained-Language-Model/blob/54ca698e4f907f32a108de371a42b76f92e7686d/TernaryBERT/quant_task_glue.py#L363-L392

Does TernaryBERT only use KD loss and not using ground truth label as training objective?

In Paper's Ablation Study, bottom row performance (-Trm-Logits) means It uses GT label Loss. Then would it be possible to say that TernaryBERT top row means It uses all three losses(Trm, Logits and GT Label)?

I'm little confused which loss should I use while reproducing TernaryBERT performance. It would be very helpful if you could answer my question.

Thanks in advance!

opened by MarsJacobs 4
how to set the config file of student_model?

when I intend to train the general chinese tinybert model, I meet some problems that the project doesn't offer the example of the config file, so could anyone offer me a reference of the config file of student_model. thanks for your kindness!

opened by jinsongpan 4
Data Augmentation

In the phase of Data Augmentation, pretrained_bert_model is General_TinyBERT in data_augmentation.py but is "pre-trained language model BERT" in the description.

opened by gongel 4
TinyBERT的疑问

看过TinyBERT的论文后，想请教如下几个问题： (1)预训练的蒸馏阶段，是指在预训练teacher BERT的同时蒸馏 student TinyBERT吗？比如每个epoch蒸馏一次或者其他？因为看到如下示意图，一开始觉得是预训练的同时进行蒸馏。另一种是可能是预训练完BERT之后，固定teacher BERT，再用相同的预训练语料同时输入到teacher BERT和要蒸馏出的TinyBERT？再逐个目标函数蒸馏？ (2)论文中似乎没有透露预训练和微调阶段的资源消耗，比如两阶段一共用了多少时间？多谢！

opened by MrRace 4
teacher和student的hidden_size不同时，fit_size作用

假设teacher和student的hidden_size分别为d和d' 当d不等于d'时，利用student模型的fit_dense层，将d‘映射到和d一样的维度，使得student和teacher之间可以计算hidden_state loss。但是当d和d'像当时，就可以不经过fit_dense映射直接计算hidden_state loss吧。但是代码里用了 if is_student判断，实际应该是判断d是否等于d'吧？

opened by littttttlebird 3
tinyBert general model with `cased`

Hello,

Have you done general distillation using the bert-base-cased model? and would you have the General_TinyBERT_v2(4layer-312dim) cased model available?

When trying python3 task_distill.py --teacher_model $FT_BERT_BASE_DIR --student_model $GENERAL_TINYBERT_DIR ... on a Fine-Tuned model that is 'bert-cased', a CUDA error is thrown

opened by sv-v5 3
词典大小对不上
Task-specific Distillation阶段，teacher是fine-tuned bert base，student是general_tinybert，两者都是由bert base而来，bert base词典大小是21128，但是为啥下载的general_tinybert词典是30522？两者怎么对齐？在task-specific distill阶段，student词典较大，输入到teacher会造成index越界。

student_logits, student_atts, student_reps = student_model(input_ids, segment_ids, input_mask, is_student=True) teacher_logits, teacher_atts, teacher_reps = teacher_model(input_ids, segment_ids, input_mask)
opened by littttttlebird 3
AutoTinyBERT models not accessible

Thanks for the awesome work on AutoTinyBERT!

We would like to use your final model checkpoints. However, the links provided in the AutoTinyBERT Model Zoo are not accessible. It would be of great help to our work if you could share the model checkpoints.

Looking forward to your response.

opened by AdityaKane2001 2
#TinyBert Training Pipeline Problems

Hi Huawei team:

Sorry to disturb you, can you guys answer my following question?

Why did the training pipeline of TinyBert "general_distill.py" not use DDP to initialize the student model, instead of only initializing the teacher model? And why there is no synchronization of the normalization layer?

And when opening the mixed-precision, where can I find the function "backward" from "optimizer"？

thx

opened by mexiQQ 2
Bump horovod from 0.22.1 to 0.24.0 in /JABER-PyTorch
Bumps horovod from 0.22.1 to 0.24.0.

Release notes

Sourced from horovod's releases.

Elastic mode improvements, MXNet async dependency engine, fixes for latest PyTorch and TensorFlow versions

Added

Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. (#3190)

TensorFlow: Added in-place broadcasting of variables. (#3128)

Elastic: Added support for resurrecting blacklisted hosts. (#3319)

MXNet: Added support for MXNet async dependency engine. (#3242, #2963)

Spark/Lightning: Added history to lightning estimator. (#3214)

Changed

Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. (#3261, #3371)

Moved released Docker image horovod and horovod-cpu to Ubuntu 20.04 and Python 3.8. (#3393)

Spark Estimator: Don't shuffle row groups if training data requires non-shuffle (#3369)

Spark/Lightning: Reduced memory footprint of async dataloader. (#3239)

Elastic: Improved handling NCCL errors under elastic scenario. (#3112)

Spark/Lightning: Do not overwrite model with checkpoint by default. (#3201)

Make checkpoint name optional so that user can save to h5 format. (#3411)

Deprecated

Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. (#3190)

Removed

Spark: Removed h5py<3 constraint as this is not needed anymore for Tensorflow >2.5.0. (#3301)

Fixed

Elastic Spark: Fixed indices in initial task-to-task registration. (#3410)

PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. (#3352)

PyTorch: Fixed finalization of ProcessSetTable. (#3351)

Fixed remote trainers to point to the correct shared lib path. (#3258)

Fixed imports from tensorflow.python.keras with tensorflow 2.6.0+. (#3403)

Fixed Adasum communicator init logic. (#3379)

Lightning: Fixed resume logger. (#3375)

Fixed the checkpoint directory structure for pytorch and pytorch lightning. (#3362)

Fixed possible integer overflow in multiplication. (#3368)

Fixed the pytorch_lightning_mnist.py example. (#3245, #3290)

Fixed barrier segmentation fault. (#3313)

Fixed hvd.barrier() tensor queue management. (#3300)

Fixed PyArrow "list index out of range" IndexError. (#3274)

Elastic: Fixed all workers sometimes failing on elastic Horovod failure. (#3264)

Spark/Lightning: Fixed setting limit_train_batches and limit_val_batches. (#3237)

Elastic: Fixed ElasticSampler and hvd.elastic.state losing some indices of processed samples when nodes dropped. (#3143)

Spark/Lightning: Fixed history metrics for estimator serialization. (#3216)

Ray: Fixed RayExecutor to fail when num_workers=0 and num_hosts=None. (#3210)

Spark/Lightning: Fixed checkpoint callback dirpath typo. (#3204)

Process sets, XLA support, improved GPU backend

... (truncated)

Changelog

Sourced from horovod's changelog.

[v0.24.0] - 2022-03-01

Added

Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. (#3190)

TensorFlow: Added in-place broadcasting of variables. (#3128)

Elastic: Added support for resurrecting blacklisted hosts. (#3319)

MXNet: Added support for MXNet async dependency engine. (#3242, #2963)

Spark/Lightning: Added history to lightning estimator. (#3214)

Changed

Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. (#3261, #3371)

Moved released Docker image horovod and horovod-cpu to Ubuntu 20.04 and Python 3.8. (#3393)

Spark Estimator: Don't shuffle row groups if training data requires non-shuffle (#3369)

Spark/Lightning: Reduced memory footprint of async dataloader. (#3239)

Elastic: Improved handling NCCL errors under elastic scenario. (#3112)

Spark/Lightning: Do not overwrite model with checkpoint by default. (#3201)

Make checkpoint name optional so that user can save to h5 format. (#3411)

Deprecated

Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. (#3190)

Removed

Spark: Removed h5py<3 constraint as this is not needed anymore for Tensorflow >2.5.0. (#3301)

Fixed

Elastic Spark: Fixed indices in initial task-to-task registration. (#3410)

PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. (#3352)

PyTorch: Fixed finalization of ProcessSetTable. (#3351)

Fixed remote trainers to point to the correct shared lib path. (#3258)

Fixed imports from tensorflow.python.keras with tensorflow 2.6.0+. (#3403)

Fixed Adasum communicator init logic. (#3379)

Lightning: Fixed resume logger. (#3375)

Fixed the checkpoint directory structure for pytorch and pytorch lightning. (#3362)

Fixed possible integer overflow in multiplication. (#3368)

Fixed the pytorch_lightning_mnist.py example. (#3245, #3290)

Fixed barrier segmentation fault. (#3313)

Fixed hvd.barrier() tensor queue management. (#3300)

Fixed PyArrow "list index out of range" IndexError. (#3274)

Elastic: Fixed all workers sometimes failing on elastic Horovod failure. (#3264)

Spark/Lightning: Fixed setting limit_train_batches and limit_val_batches. (#3237)

Elastic: Fixed ElasticSampler and hvd.elastic.state losing some indices of processed samples when nodes dropped. (#3143)

Spark/Lightning: Fixed history metrics for estimator serialization. (#3216)

Ray: Fixed RayExecutor to fail when num_workers=0 and num_hosts=None. (#3210)

Spark/Lightning: Fixed checkpoint callback dirpath typo. (#3204)

... (truncated)

Commits

b089df6 Bump version to 0.24.0 (#3433)

db19aa4 Move apt-get into non-interactive mode (#3441)

2632c05 Build Horovod with temporarily installed CMake if necessary (#3371)

7bf9b04 Make checkpoint name optional so that user can save to h5 format. (#3411)

b553974 Fix flaky ray tests (#3430)

7b5346e Fix indices in initial task-to-task registration (#3410)

71e10b4 Fixing GPU and CPU TF head CI failures (#3431)

79ded4b Fix FindNVTX.cmake (#3421)

642a6b3 [TF - Fix] Fix imports from tensorflow.python.keras with tf.version >= 2....

046c071 Allow stderr of executed cmake python code appear in logs (#3398)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
TinyBERT实验到底用哪个enwiki-latest-pages-articles数据集？

原文第6页提到： For the general distillation, we set the maximum sequence length to 128 and use English Wikipedia (2,500M words) 我从 https://github.com/google-research/bert 指定的链接下载 the latest dump 此压缩包解压后形成了一个86G的xml文件，经本工程的预处理代码总是报超磁盘空间，且每跑十几个小时就断掉，查代码以后，将pregenerate_training_date.py文件第52行self.document_shelf_filepath的路径从/cache/目录改到外部磁盘的500G文件目录，这次终于不再报超磁盘空间，但处理速度很慢，84个小时才从第367行跑到第390行。然后最崩溃的来了！由于后面还要跑3个epoch，又跑了2天才跑完第一个epoch的5%，合着40天才能跑完一个epoch，总共3个epoch就要120天！仅仅数据预处理就要跑这么久吗？即使跑完，后面还要上GPU训练，会不会更久？？？请问原文用的是哪个数据集？是不是要用华为云平台跑才能快一些？

opened by ra225 0
Bump certifi from 2021.5.30 to 2022.12.7 in /JABER-PyTorch
Bumps certifi from 2021.5.30 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
使用nezha_base_www模型，得到的嵌入向量为nan
#引用nezha模型 from transformers import NezhaModel, NezhaConfig

self.config = BertConfig.from_pretrained(config_path) self.bert_module = NezhaModel.from_pretrained(bert_dir, config=self.config) bert_outputs = self.bert_module(input_ids=x, attention_mask=mask, token_type_ids=segs, output_hidden_states =True)

bert_outputs结果中，多层结果是nan，不知道是什么原因。 BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state=tensor([[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., [nan, nan, nan, ..., nan, nan, nan]], device='cuda:0'), hidden_states=(tensor([[[ 0.5742, -0.2564, 0.4186, ..., 0.8307, -1.6965, 0.6848], [-0.6152, 0.1826, -1.1161, ..., 0.6985, -3.4405, 1.4675], [-0.2423, 0.8284, 0.5155, ..., 1.0843, -1.4233, 0.5122], ..., [-0.2828, -0.2603, -0.6676, ..., 0.5609, -2.0621, 0.5314],

[ 0.5203, 0.3228, -0.4273, ..., -0.2345, -0.1468, -0.2845], [ 0.5203, 0.3228, -0.4273, ..., -0.2345, -0.1468, -0.2845], [ 0.5203, 0.3228, -0.4273, ..., -0.2345, -0.1468, -0.2845]]], device='cuda:0'), tensor([[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan]]], device='cuda:0'),), past_key_values=None, attentions=None, cross_attentions=None)
opened by yixiu00001 0

Owner

HUAWEI Noah's Ark Lab

Working with and contributing to the open source community in data mining, artificial intelligence, and related fields.

GitHub

RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

RITA: a Study on Scaling Up Generative Protein Sequence Models RITA is a family of autoregressive protein models, developed by a collaboration of Ligh

69 Dec 22, 2022

All the code and files related to the MI-Lab of UE19CS305 course in sem 5

Machine-Intelligence-Lab-CS305 The compilation of all the code an drelated files from MI-Lab UE19CS305 (of batch 2019-2023) offered by PES University

3 Nov 10, 2022

A trusty face recognition research platform developed by Tencent Youtu Lab

Introduction TFace: A trusty face recognition research platform developed by Tencent Youtu Lab. It provides a high-performance distributed training fr

956 Jan 1, 2023

Ultra-lightweight human body posture key point CNN model. ModelSize:2.3MB HUAWEI P40 NCNN benchmark: 6ms/img,

Ultralight-SimplePose Support NCNN mobile terminal deployment Based on MXNET(>=1.5.1) GLUON(>=0.7.0) framework Top-down strategy: The input image is t

223 Dec 27, 2022

Developed an optimized algorithm which finds the most optimal path between 2 points in a 3D Maze using various AI search techniques like BFS, DFS, UCS, Greedy BFS and A*

Developed an optimized algorithm which finds the most optimal path between 2 points in a 3D Maze using various AI search techniques like BFS, DFS, UCS, Greedy BFS and A*. The algorithm was extremely optimal running in ~15s to ~30s for search spaces as big as 10000000 nodes where a set of 18 actions could be performed at each node in the 3D Maze.

1 Mar 28, 2022

🐦 Opytimizer is a Python library consisting of meta-heuristic optimization techniques.

Opytimizer: A Nature-Inspired Python Optimizer Welcome to Opytimizer. Did you ever reach a bottleneck in your computational experiments? Are you tired

546 Dec 31, 2022

An implementation of quantum convolutional neural network with MindQuantum. Huawei, classifying MNIST dataset

关于实现的一点说明山东大学 2020级苏博南 www.subonan.com 文件说明 tools.py 这里面主要有两个函数： resize(a, lenb) 这其实是我找同学写的一个小算法hhh。给出一个$28\times 28$的方阵a，返回一个$lenb\times lenb$的方阵。因

2 Aug 29, 2022

Yolo ros - YOLO-ROS for HUAWEI ATLAS200

YOLO-ROS YOLO-ROS for NVIDIA YOLO-ROS for HUAWEI ATLAS200, please checkout for b

5 Oct 18, 2022

[ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links

LinkBERT: A Knowledgeable Language Model Pretrained with Document Links This repo provides the model, code & data of our paper: LinkBERT: Pretraining

264 Jan 1, 2023

SAS output to EXCEL converter for Cornell/MIT Language and acquisition lab

CORNELLSASLAB SAS output to EXCEL converter for Cornell/MIT Language and acquisition lab Instructions: This python code can be used to convert SAS out

2 Jan 26, 2022

Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

3.7k Jan 3, 2023

library for nonlinear optimization, wrapping many algorithms for global and local, constrained or unconstrained, optimization

NLopt is a library for nonlinear local and global optimization, for functions with and without gradient information. It is designed as a simple, unifi

1.4k Dec 25, 2022

Implemented fully documented Particle Swarm Optimization algorithm (basic model with few advanced features) using Python programming language

Implemented fully documented Particle Swarm Optimization (PSO) algorithm in Python which includes a basic model along with few advanced features such as updating inertia weight, cognitive, social learning coefficients and maximum velocity of the particle.

9 Nov 29, 2022

Racing line optimization algorithm in python that uses Particle Swarm Optimization.

Racing Line Optimization with PSO This repository contains a racing line optimization algorithm in python that uses Particle Swarm Optimization. Requi

6 Dec 14, 2022

This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

UCPhrase: Unsupervised Context-aware Quality Phrase Tagging To appear on KDD'21...[pdf] This project provides an unsupervised framework for mining and

146 Dec 22, 2022

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Related tags

Overview

Pretrained Language Model

Directory structure

Comments

Elastic mode improvements, MXNet async dependency engine, fixes for latest PyTorch and TensorFlow versions

Added

Changed

Deprecated

Removed

Fixed

Process sets, XLA support, improved GPU backend

[v0.24.0] - 2022-03-01