Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Related tags

Text Data & NLP quantization knowledge-distillation model-compression pretrained-language-model
Overview

Pretrained Language Model

This repository provides the latest pretrained language models and its related optimization techniques developed by Huawei Noah's Ark Lab.

Directory structure

  • PanGu-α is a Large-scale autoregressive pretrained Chinese language model with up to 200B parameter. The models are developed under the MindSpore and trained on a cluster of Ascend 910 AI processors.
  • NEZHA-TensorFlow is a pretrained Chinese language model which achieves the state-of-the-art performances on several Chinese NLP tasks developed under TensorFlow.
  • NEZHA-PyTorch is the PyTorch version of NEZHA.
  • NEZHA-Gen-TensorFlow provides two GPT models. One is Yuefu (乐府), a Chinese Classical Poetry generation model, the other is a common Chinese GPT model.
  • TinyBERT is a compressed BERT model which achieves 7.5x smaller and 9.4x faster on inference.
  • TinyBERT-MindSpore is a MindSpore version of TinyBERT.
  • DynaBERT is a dynamic BERT model with adaptive width and depth.
  • BBPE provides a byte-level vocabulary building tool and its correspoinding tokenizer.
  • PMLM is a probabilistically masked language model. Trained without the complex two-stream self-attention, PMLM can be treated as a simple approximation of XLNet.
  • TernaryBERT is a weights ternarization method for BERT model developed under PyTorch.
  • TernaryBERT-MindSpore is the MindSpore version of TernaryBERT.
  • HyperText is an efficient text classification model based on hyperbolic geometry theories.
  • BinaryBERT is a weights binarization method using ternary weight splitting for BERT model, developed under PyTorch.
  • AutoTinyBERT provides a model zoo that can meet different latency requirements.
Comments
  • Question towards TinyBERT Data Augmentation ${GLOVE_EMB}$

    Question towards TinyBERT Data Augmentation ${GLOVE_EMB}$

    Hi, all

    In the part of Data Augmentation I have seen “--glove_embs ${GLOVE_EMB}$”, I am wondering what should I use to replace this part: "${GLOVE_EMB}$"

    I have noticed from the code in the data_augmentation.py, it mentioned it is the glove embedding file. If we should replace "${GLOVE_EMB}$" with the location of the glove embedding file.

    May I know where can we get the glove embedding file? Could you provide me with a link?

    opened by MichaelCaohn 6
  • Does Ternary BERT only use KD Loss(Teacher, Student Loss) while training?

    Does Ternary BERT only use KD Loss(Teacher, Student Loss) while training?

    Hi, Thanks for this great source code. It really helps me a lot!

    While I'm studying the TernaryBERT with Paper and this source code, I have a question about KD Training Loss. In Paper Algorithm1, It says that when compute the gradient, It only uses Distillation Loss, not the Distillation Loss + GT Label Cross Entropy Loss.

    스크린샷 2021-09-25 오후 3 15 15

    And also in source code, there is only KD loss which is used for backward. https://github.com/huawei-noah/Pretrained-Language-Model/blob/54ca698e4f907f32a108de371a42b76f92e7686d/TernaryBERT/quant_task_glue.py#L363-L392

    Does TernaryBERT only use KD loss and not using ground truth label as training objective?

    스크린샷 2021-09-25 오후 3 08 25

    In Paper's Ablation Study, bottom row performance (-Trm-Logits) means It uses GT label Loss. Then would it be possible to say that TernaryBERT top row means It uses all three losses(Trm, Logits and GT Label)?

    I'm little confused which loss should I use while reproducing TernaryBERT performance. It would be very helpful if you could answer my question.

    Thanks in advance!

    opened by MarsJacobs 4
  • how to set the config file of student_model?

    how to set the config file of student_model?

    when I intend to train the general chinese tinybert model, I meet some problems that the project doesn't offer the example of the config file, so could anyone offer me a reference of the config file of student_model. image thanks for your kindness!

    opened by jinsongpan 4
  • Data Augmentation

    Data Augmentation

    In the phase of Data Augmentation, pretrained_bert_model is General_TinyBERT in data_augmentation.py but is "pre-trained language model BERT" in the description.

    opened by gongel 4
  • TinyBERT的疑问

    TinyBERT的疑问

    看过TinyBERT的论文后，想请教如下几个问题： (1)预训练的蒸馏阶段，是指在预训练teacher BERT的同时蒸馏 student TinyBERT吗？比如每个epoch蒸馏一次或者其他？因为看到如下示意图，一开始觉得是预训练的同时进行蒸馏。 image 另一种是可能是预训练完BERT之后，固定teacher BERT，再用相同的预训练语料同时输入到teacher BERT和 要蒸馏出的TinyBERT？再逐个目标函数蒸馏？ (2)论文中似乎没有透露预训练和微调阶段的资源消耗，比如两阶段一共用了多少时间？ 多谢！

    opened by MrRace 4
  • teacher和student的hidden_size不同时，fit_size作用

    teacher和student的hidden_size不同时，fit_size作用

    假设teacher和student的hidden_size分别为d和d' 当d不等于d'时，利用student模型的fit_dense层，将d‘映射到和d一样的维度，使得student和teacher之间可以计算hidden_state loss。 但是当d和d'像当时，就可以不经过fit_dense映射直接计算hidden_state loss吧。但是代码里用了 if is_student判断，实际应该是判断d是否等于d'吧？

    opened by littttttlebird 3
  • tinyBert general model with `cased`

    tinyBert general model with `cased`

    Hello,

    Have you done general distillation using the bert-base-cased model? and would you have the General_TinyBERT_v2(4layer-312dim) cased model available?

    When trying python3 task_distill.py --teacher_model $FT_BERT_BASE_DIR --student_model $GENERAL_TINYBERT_DIR ... on a Fine-Tuned model that is 'bert-cased', a CUDA error is thrown

    opened by sv-v5 3
  • 词典大小对不上

    词典大小对不上

    Task-specific Distillation阶段，teacher是fine-tuned bert base，student是general_tinybert，两者都是由bert base而来，bert base词典大小是21128，但是为啥下载的general_tinybert词典是30522？两者怎么对齐？ 在task-specific distill阶段，student词典较大，输入到teacher会造成index越界。

    student_logits, student_atts, student_reps = student_model(input_ids, segment_ids, input_mask, is_student=True)
teacher_logits, teacher_atts, teacher_reps = teacher_model(input_ids, segment_ids, input_mask)
    opened by littttttlebird 3
  • AutoTinyBERT models not accessible

    AutoTinyBERT models not accessible

    Thanks for the awesome work on AutoTinyBERT!

    We would like to use your final model checkpoints. However, the links provided in the AutoTinyBERT Model Zoo are not accessible. It would be of great help to our work if you could share the model checkpoints.

    Looking forward to your response.

    opened by AdityaKane2001 2
  • #TinyBert Training Pipeline Problems

    #TinyBert Training Pipeline Problems

    Hi Huawei team:

    Sorry to disturb you, can you guys answer my following question?

    Why did the training pipeline of TinyBert "general_distill.py" not use DDP to initialize the student model, instead of only initializing the teacher model? And why there is no synchronization of the normalization layer?

    image

    And when opening the mixed-precision, where can I find the function "backward" from "optimizer"？

    image

    thx

    opened by mexiQQ 2
  • Bump horovod from 0.22.1 to 0.24.0 in /JABER-PyTorch

    Bump horovod from 0.22.1 to 0.24.0 in /JABER-PyTorch

    Bumps horovod from 0.22.1 to 0.24.0.

    Release notes

    Sourced from horovod's releases.

    Elastic mode improvements, MXNet async dependency engine, fixes for latest PyTorch and TensorFlow versions

    Added

    • Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. (#3190)
    • TensorFlow: Added in-place broadcasting of variables. (#3128)
    • Elastic: Added support for resurrecting blacklisted hosts. (#3319)
    • MXNet: Added support for MXNet async dependency engine. (#3242, #2963)
    • Spark/Lightning: Added history to lightning estimator. (#3214)

    Changed

    • Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. (#3261, #3371)
    • Moved released Docker image horovod and horovod-cpu to Ubuntu 20.04 and Python 3.8. (#3393)
    • Spark Estimator: Don't shuffle row groups if training data requires non-shuffle (#3369)
    • Spark/Lightning: Reduced memory footprint of async dataloader. (#3239)
    • Elastic: Improved handling NCCL errors under elastic scenario. (#3112)
    • Spark/Lightning: Do not overwrite model with checkpoint by default. (#3201)
    • Make checkpoint name optional so that user can save to h5 format. (#3411)

    Deprecated

    • Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. (#3190)

    Removed

    • Spark: Removed h5py<3 constraint as this is not needed anymore for Tensorflow >2.5.0. (#3301)

    Fixed

    • Elastic Spark: Fixed indices in initial task-to-task registration. (#3410)
    • PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. (#3352)
    • PyTorch: Fixed finalization of ProcessSetTable. (#3351)
    • Fixed remote trainers to point to the correct shared lib path. (#3258)
    • Fixed imports from tensorflow.python.keras with tensorflow 2.6.0+. (#3403)
    • Fixed Adasum communicator init logic. (#3379)
    • Lightning: Fixed resume logger. (#3375)
    • Fixed the checkpoint directory structure for pytorch and pytorch lightning. (#3362)
    • Fixed possible integer overflow in multiplication. (#3368)
    • Fixed the pytorch_lightning_mnist.py example. (#3245, #3290)
    • Fixed barrier segmentation fault. (#3313)
    • Fixed hvd.barrier() tensor queue management. (#3300)
    • Fixed PyArrow "list index out of range" IndexError. (#3274)
    • Elastic: Fixed all workers sometimes failing on elastic Horovod failure. (#3264)
    • Spark/Lightning: Fixed setting limit_train_batches and limit_val_batches. (#3237)
    • Elastic: Fixed ElasticSampler and hvd.elastic.state losing some indices of processed samples when nodes dropped. (#3143)
    • Spark/Lightning: Fixed history metrics for estimator serialization. (#3216)
    • Ray: Fixed RayExecutor to fail when num_workers=0 and num_hosts=None. (#3210)
    • Spark/Lightning: Fixed checkpoint callback dirpath typo. (#3204)

    Process sets, XLA support, improved GPU backend

    ... (truncated)

    Changelog

    Sourced from horovod's changelog.

    [v0.24.0] - 2022-03-01

    Added

    • Ray: Added elastic keyword parameters to RayExecutor API: This API supports both static (non-elastic) and elastic Horovod jobs. (#3190)
    • TensorFlow: Added in-place broadcasting of variables. (#3128)
    • Elastic: Added support for resurrecting blacklisted hosts. (#3319)
    • MXNet: Added support for MXNet async dependency engine. (#3242, #2963)
    • Spark/Lightning: Added history to lightning estimator. (#3214)

    Changed

    • Moved to CMake version 3.13 with first-class CUDA language support and re-enabled parallelized builds. Uses a temporary installation of CMake if CMake 3.13 is not found. (#3261, #3371)
    • Moved released Docker image horovod and horovod-cpu to Ubuntu 20.04 and Python 3.8. (#3393)
    • Spark Estimator: Don't shuffle row groups if training data requires non-shuffle (#3369)
    • Spark/Lightning: Reduced memory footprint of async dataloader. (#3239)
    • Elastic: Improved handling NCCL errors under elastic scenario. (#3112)
    • Spark/Lightning: Do not overwrite model with checkpoint by default. (#3201)
    • Make checkpoint name optional so that user can save to h5 format. (#3411)

    Deprecated

    • Deprecated ElasticRayExecutor APIs in favor of the new RayExecutor API. (#3190)

    Removed

    • Spark: Removed h5py<3 constraint as this is not needed anymore for Tensorflow >2.5.0. (#3301)

    Fixed

    • Elastic Spark: Fixed indices in initial task-to-task registration. (#3410)
    • PyTorch: Fixed GIL-related deadlock with PyTorch 1.10.1. (#3352)
    • PyTorch: Fixed finalization of ProcessSetTable. (#3351)
    • Fixed remote trainers to point to the correct shared lib path. (#3258)
    • Fixed imports from tensorflow.python.keras with tensorflow 2.6.0+. (#3403)
    • Fixed Adasum communicator init logic. (#3379)
    • Lightning: Fixed resume logger. (#3375)
    • Fixed the checkpoint directory structure for pytorch and pytorch lightning. (#3362)
    • Fixed possible integer overflow in multiplication. (#3368)
    • Fixed the pytorch_lightning_mnist.py example. (#3245, #3290)
    • Fixed barrier segmentation fault. (#3313)
    • Fixed hvd.barrier() tensor queue management. (#3300)
    • Fixed PyArrow "list index out of range" IndexError. (#3274)
    • Elastic: Fixed all workers sometimes failing on elastic Horovod failure. (#3264)
    • Spark/Lightning: Fixed setting limit_train_batches and limit_val_batches. (#3237)
    • Elastic: Fixed ElasticSampler and hvd.elastic.state losing some indices of processed samples when nodes dropped. (#3143)
    • Spark/Lightning: Fixed history metrics for estimator serialization. (#3216)
    • Ray: Fixed RayExecutor to fail when num_workers=0 and num_hosts=None. (#3210)
    • Spark/Lightning: Fixed checkpoint callback dirpath typo. (#3204)

    ... (truncated)

    Commits
    • b089df6 Bump version to 0.24.0 (#3433)
    • db19aa4 Move apt-get into non-interactive mode (#3441)
    • 2632c05 Build Horovod with temporarily installed CMake if necessary (#3371)
    • 7bf9b04 Make checkpoint name optional so that user can save to h5 format. (#3411)
    • b553974 Fix flaky ray tests (#3430)
    • 7b5346e Fix indices in initial task-to-task registration (#3410)
    • 71e10b4 Fixing GPU and CPU TF head CI failures (#3431)
    • 79ded4b Fix FindNVTX.cmake (#3421)
    • 642a6b3 [TF - Fix] Fix imports from tensorflow.python.keras with tf.version >= 2....
    • 046c071 Allow stderr of executed cmake python code appear in logs (#3398)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • TinyBERT实验到底用哪个enwiki-latest-pages-articles数据集？

    TinyBERT实验到底用哪个enwiki-latest-pages-articles数据集？

    原文第6页提到： For the general distillation, we set the maximum sequence length to 128 and use English Wikipedia (2,500M words) 我从 https://github.com/google-research/bert 指定的链接下载 the latest dump 此压缩包解压后形成了一个86G的xml文件，经本工程的预处理代码总是报超磁盘空间，且每跑十几个小时就断掉，查代码以后，将pregenerate_training_date.py文件第52行self.document_shelf_filepath的路径从/cache/目录改到外部磁盘的500G文件目录，这次终于不再报超磁盘空间，但处理速度很慢，84个小时才从第367行跑到第390行。 然后最崩溃的来了！由于后面还要跑3个epoch，又跑了2天才跑完第一个epoch的5%，合着40天才能跑完一个epoch，总共3个epoch就要120天！ 仅仅数据预处理就要跑这么久吗？即使跑完，后面还要上GPU训练，会不会更久？？？ 请问原文用的是哪个数据集？是不是要用华为云平台跑才能快一些？

    opened by ra225 0
  • Bump certifi from 2021.5.30 to 2022.12.7 in /JABER-PyTorch

    Bump certifi from 2021.5.30 to 2022.12.7 in /JABER-PyTorch

    Bumps certifi from 2021.5.30 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • 使用nezha_base_www模型，得到的嵌入向量为nan

    使用nezha_base_www模型，得到的嵌入向量为nan

    #引用nezha模型 from transformers import NezhaModel, NezhaConfig

    self.config = BertConfig.from_pretrained(config_path) self.bert_module = NezhaModel.from_pretrained(bert_dir, config=self.config) bert_outputs = self.bert_module(input_ids=x, attention_mask=mask, token_type_ids=segs, output_hidden_states =True)

    bert_outputs结果中，多层结果是nan，不知道是什么原因。 BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state=tensor([[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., [nan, nan, nan, ..., nan, nan, nan]], device='cuda:0'), hidden_states=(tensor([[[ 0.5742, -0.2564, 0.4186, ..., 0.8307, -1.6965, 0.6848], [-0.6152, 0.1826, -1.1161, ..., 0.6985, -3.4405, 1.4675], [-0.2423, 0.8284, 0.5155, ..., 1.0843, -1.4233, 0.5122], ..., [-0.2828, -0.2603, -0.6676, ..., 0.5609, -2.0621, 0.5314],

         [ 0.5203,  0.3228, -0.4273,  ..., -0.2345, -0.1468, -0.2845],
     [ 0.5203,  0.3228, -0.4273,  ..., -0.2345, -0.1468, -0.2845],
     [ 0.5203,  0.3228, -0.4273,  ..., -0.2345, -0.1468, -0.2845]]],
   device='cuda:0'), tensor([[[nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan],
     ...,
   
     [nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan]]], device='cuda:0'),), past_key_values=None, attentions=None, cross_attentions=None)
    opened by yixiu00001 0
Owner
HUAWEI Noah's Ark Lab
Working with and contributing to the open source community in data mining, artificial intelligence, and related fields.
HUAWEI Noah's Ark Lab
GitHub
In this repository, I have developed an end to end Automatic speech recognition project. I have developed the neural network model for automatic speech recognition with PyTorch and used MLflow to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry.

End to End Automatic Speech Recognition In this repository, I have developed an end to end Automatic speech recognition project. I have developed the

Victor Basu 22 Nov 13, 2022
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

A Deep Learning NLP/NLU library by Intel® AI Lab Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing NLP Architect

Intel Labs 2.9k Jan 2, 2023
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

A Deep Learning NLP/NLU library by Intel® AI Lab Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing NLP Architect

Intel Labs 2.6k Feb 18, 2021
A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

A Deep Learning NLP/NLU library by Intel® AI Lab Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing NLP Architect

Intel Labs 2.9k Dec 31, 2022
华为商城抢购手机的Python脚本 Python script of Huawei Store snapping up mobile phones

HUAWEI STORE GO 2021 说明 基于Python3+Selenium的华为商城抢购爬虫脚本，修改自近两年没更新的项目BUY-HW，为女神抢Nova 8（什么时候华为开始学小米玩饥饿营销了？） 原项目的登陆以及抢购部分已经不可用，本项目对原项目进行了改正以适应新华为商城，并增加一些功能

ZhangLiang 111 Dec 22, 2022
ProteinBERT is a universal protein language model pretrained on ~106M proteins from the UniRef90 dataset.

ProteinBERT is a universal protein language model pretrained on ~106M proteins from the UniRef90 dataset. Through its Python API, the pretrained model can be fine-tuned on any protein-related task in a matter of minutes. Based on our experiments with a wide range of benchmarks, ProteinBERT usually achieves state-of-the-art performance. ProteinBERT is built on TenforFlow/Keras.

null 241 Jan 4, 2023
BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs).

BMInf (Big Model Inference) is a low-resource inference package for large-scale pretrained language models (PLMs).

OpenBMB 377 Jan 2, 2023
Transformer related optimization, including BERT, GPT

This repository provides a script and recipe to run the highly optimized transformer-based encoder and decoder component, and it is tested and maintained by NVIDIA.

NVIDIA Corporation 1.7k Jan 4, 2023
This repository contains the code for "Generating Datasets with Pretrained Language Models".

Datasets from Instructions (DINO ?? ) This repository contains the code for Generating Datasets with Pretrained Language Models. The paper introduces

Timo Schick 154 Jan 1, 2023
Composed Image Retrieval using Pretrained LANguage Transformers (CIRPLANT)

CIRPLANT This repository contains the code and pre-trained models for Composed Image Retrieval using Pretrained LANguage Transformers (CIRPLANT) For d

Zheyuan (David) Liu 29 Nov 17, 2022
IndoBERTweet is the first large-scale pretrained model for Indonesian Twitter. Published at EMNLP 2021 (main conference)

IndoBERTweet ?? ???? 1. Paper Fajri Koto, Jey Han Lau, and Timothy Baldwin. IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effe

IndoLEM 40 Nov 30, 2022
🦅 Pretrained BigBird Model for Korean (up to 4096 tokens)

Pretrained BigBird Model for Korean What is BigBird • How to Use • Pretraining • Evaluation Result • Docs • Citation 한국어 | English What is BigBird? Bi

Jangwon Park 183 Dec 14, 2022
A machine learning model for analyzing text for user sentiment and determine whether its a positive, neutral, or negative review.

Sentiment Analysis on Yelp's Dataset Author: Roberto Sanchez, Talent Path: D1 Group Docker Deployment: Deployment of this application can be found her

Roberto Sanchez 0 Aug 4, 2021
A collection of Classical Chinese natural language processing models, including Classical Chinese related models and resources on the Internet.

GuwenModels: 古文自然语言处理模型合集, 收录互联网上的古文相关模型及资源. A collection of Classical Chinese natural language processing models, including Classical Chinese related models and resources on the Internet.

Ethan 66 Dec 26, 2022
ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab

AliceMind AliceMind: ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab This repository provides pre-trained encode

Alibaba 922 Dec 10, 2021
A Neural Language Style Transfer framework to transfer natural language text smoothly between fine-grained language styles like formal/casual, active/passive, and many more. Created by Prithiviraj Damodaran. Open to pull requests and other forms of collaboration.

Styleformer A Neural Language Style Transfer framework to transfer natural language text smoothly between fine-grained language styles like formal/cas

Prithivida 431 Dec 19, 2022
Sentello is python script that simulates the anti-evasion and anti-analysis techniques used by malware.

sentello Sentello is a python script that simulates the anti-evasion and anti-analysis techniques used by malware. For techniques that are difficult t

Malwation 62 Oct 2, 2022
The projects lets you extract glossary words and their definitions from a given piece of text automatically using NLP techniques

Unsupervised technique to Glossary and Definition Extraction Code Files GPT2-DefinitionModel.ipynb - GPT-2 model for definition generation. Data_Gener

Prakhar Mishra 28 May 25, 2021
The proliferation of disinformation across social media has led the application of deep learning techniques to detect fake news.

Fake News Detection Overview The proliferation of disinformation across social media has led the application of deep learning techniques to detect fak

Kushal Shingote 1 Feb 8, 2022