PatrickStar enables Larger, Faster, Greener Pretrained Models for NLP. Democratize AI for everyone.

Overview

PatrickStar: Parallel Training of Large Language Models via a Chunk-based Memory Management

logo

Meeting PatrickStar

Pre-Trained Models (PTM) are becoming the hotspot of both NLP research and industry application. They are models that are trained with massive data and have learned generic features of the language. In pactice, they are fine-tuned for downstream tasks with task-specific datasets. In this way, PTMs have achieved great performances in almost every tasks. However, the training of PTMs requires enormous hardware resources, which makes it only accessible to small portion of people in the AI community. Now, PatrickStar will make PTM training available to everyone!

Out of memory error (OOM) is the nightmare of every engineer training PTMs. To prevent such error, we often have to introduce more GPUs to store the model params. PatrickStar brings a better solution for such problem. With the heterogeneous training (DeepSpeed Zero Stage 3 also uses it), PatrickStar could make full use of both the CPU and GPU memory, so that you could use fewer GPUs to train larger models.

We noticed that the GPU memory usage varies during training, but the current heterogenous training solutions are all statically spliting the model and optimizer states to CPU and GPU. To make better use of the GPU, PatrickStar proposes a dynamic memory scheduling with the help of a chunk-based memory management module. The memory management of PatrickStar supports offloading everything but the current computing part of the model to CPU. This results in training a much larger model within the same hardware environment. In terms of performance, the chunk-based memory management takes advantage of the linear structure of the transformer-based PTMs, so that it will inherently prefetch the upcoming layers to GPUs, resulting in a great performance.

In experiment, Patrickstar is able to train a 12B param model with 8 Tesla V100 GPU and 240GB GPU memory, which is twice as large as the state of art. And the performance of PatrickStar is better for models of the same size as well. The deeps indicates performance of DeepSpeed v0.4.3 using the official example DeepSpeed example zero3 stage with activation optimzations openinig by default.

alt perf

We've also trained the CLUE-GPT2 model with PatrickStar, the loss and accuracy curve is shown below:

CLUE-GPT2

Installation

pip install .

Note that PatrickStar requires gcc of version 7 or higher. You could also use NVIDIA NGC images, the following image is tested:

docker pull nvcr.io/nvidia/pytorch:21.06-py3

Usage

PatrickStar is based on PyTorch, which makes it easy to migrate a pytorch project. Here is a example of PatrickStar:

from patrickstar.runtime import initialize_engine

config = {
    "optimizer": {
        "type": "Adam",
        "params": {
            "lr": 0.001,
            "betas": (0.9, 0.999),
            "eps": 1e-6,
            "weight_decay": 0,
            "use_hybrid_adam": True,
        },
    },
    "fp16": {  # loss scaler params
        "enabled": True,
        "loss_scale": 0,
        "initial_scale_power": 2 ** 3,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1,
    },
    "default_chunk_size": 64 * 1024 * 1024,
    "release_after_init": True,
    "use_cpu_embedding": False,
}

def model_func():
    # MyModel is a derived class for torch.nn.Module
    return MyModel(...)

model, optimizer = initialize_engine(model_func=model_func, local_rank=0, config=config)

...

for data in dataloader:
    optimizer.zero_grad()

    loss = model(data)
    model.backward(loss)
    optimizer.step()

We use the same config format as DeepSpeed configuration JSON, which mainly includes params of optimizer, loss scaler and some PatrickStar specific configuration.

For some detail explanation of the above example, please check the guide here

For more examples, please check here.

Inside PatrickStar

See this doc for the idea behind PatrickStar.

License

BSD 3-Clause License

Cite Us

@article{fang2021patrickstar,
  title={PatrickStar: Parallel Training of Pre-trained Models via a Chunk-based Memory Management},
  author={Fang, Jiarui and Yu, Yang and Zhu, Zilin and Li, Shenggui and You, Yang and Zhou, Jie},
  journal={arXiv preprint arXiv:2108.05818},
  year={2021}
}

Contact Us

{jiaruifang, zilinzhu, josephyu}@tencent.com

Powered by WeChat AI Team, Tencent NLP Oteam.

Comments
  • CPU Embedding

    CPU Embedding

    因为embedding的参数比其他layer参数大很多,我们不把ebd参数交给chunk管理,并将其计算固定在CPU中。 这样每次计算前用hook将input从gpu拷贝到cpu,CPU ebd layers计算之后,再将输出的activations拷贝回GPU,以参与后面计算。 但是,有些PyTorch版本不支持torch.half类型的CPU embedding计算(比如torch, 1.4.0+cu100不支持,1.7.1+cu110则支持)。 现在cpu ebd也有param fp16和param fp32两份参数,但是param fp16也存成torch.float类型,用于FWD和BWD的计算,param fp32用于ADAM计算。而且每个进程都存储全部参数。 这样存在巨大内存浪费,首先其实只需要存一份torch.float类型的param,并可以用模型并行方式,分布在多个进程。

    help wanted 
    opened by feifeibear 6
  • C++ adam速度

    C++ adam速度

    Aug 10的性能结果 log.GPT2small_gpu_1_cs_64_bs_128_cpueb_1_margin_0.8_warmup_0.2_gpu_0.8_adamcvt_1

    2021-08-10:14:34:53,509 INFO [memory_monitor.py:65] CPU Virtual Memory: used = 15.08 GB, percent = 96.6% 605 2021-08-10:14:34:53,509 INFO [test_bert.py:223] ckp True fp16 True ps True: step elapse 5.177955627441406 sec/iter, 18.463766371092152 Tflops 606 2021-08-10:14:34:53,509 INFO [test_bert.py:225] model 0.72940493 607 2021-08-10:14:34:53,509 INFO [global_timer.py:45] *********** PROFILE RESULTS ************* 608 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CHUNK_LIST_prepare_device, 0, 0.0 % 609 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CHUNK_allocate_payload, 0, 0.0 % 610 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CLIENT_access, 0.019408226013183594, 0.338427821424322 % 611 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CLIENT_release, 0.014924049377441406, 0.2602357121256555 % 612 2021-08-10:14:34:53,509 INFO [global_timer.py:50] chunk_cpu_gpu_move, 0, 0.0 % 613 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CLIENT_access_dist, 0.03873419761657715, 0.6754213447995139 % 614 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CLIENT_release_dist, 0.3606679439544678, 6.289089298897653 % 615 2021-08-10:14:34:53,509 INFO [global_timer.py:50] chunk_gpu_cpu_move, 0, 0.0 % 616 2021-08-10:14:34:53,509 INFO [global_timer.py:50] CHUNK_LIST_chunk_move, 0, 0.0 % 617 2021-08-10:14:34:53,509 INFO [global_timer.py:50] FWD, 0.28232502937316895, 4.9229973187357 % 618 2021-08-10:14:34:53,509 INFO [global_timer.py:50] BWD, 2.9886157512664795, 52.1135067722565 % 619 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM_prepare_data_fp16_grad_to_fp32_grad_copy, 0.2039637565612793, 3.5565852198787224 % 620 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM_prepare_data, 0.22702884674072266, 3.958779022397416 % 621 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM_compute, 0.013135433197021484, 0.2290470049819615 % 622 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM_param_fp32_to_fp16, 0.5844182968139648, 10.190700111226695 % 623 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM_release_data, 0.016661882400512695, 0.29053889612597344 % 624 2021-08-10:14:34:53,509 INFO [global_timer.py:50] ADAM, 0.9849364757537842, 17.174671477149886 % 625 2021-08-10:14:34:53,509 INFO [global_timer.py:76] *********** DATA MOVE RESULTS ************* 626 2021-08-10:14:34:53,509 INFO [global_timer.py:86] chunk_cpu_gpu_move: 0.0 MB 627 2021-08-10:14:34:53,509 INFO [global_timer.py:86] chunk_gpu_cpu_move: 0.0 MB 628 2021-08-10:14:34:53,509 INFO [global_timer.py:83] ADAM_prepare_data_fp16_grad_to_fp32_grad_copy: 2782.4589920043945 MB, 393 times, 13641.92854120348 MB/s 629 2021-08-10:14:34:53,509 INFO [global_timer.py:83] ADAM_param_fp32_to_fp16: 2782.4589920043945 MB, 393 times, 4761.0744002597885 MB/s

    opened by feifeibear 5
  • [idea]进一步减少内存消耗,通过融合FWD+BWD+ADAM

    [idea]进一步减少内存消耗,通过融合FWD+BWD+ADAM

    有一个想法,我们其实可以进一步缩减memory footprint。 我们可以只保留param fp32,FWD时, submodule(etc. :Linear)需要的param fp16临时分配,并从param fp32拷贝数据。计算完毕就释放。 BWD时候, submodule需要的话再从param fp32转化,产生grad fp16,后立刻开始adam计算,更新param fp32。这样grad fp16也可以扔掉。 总的内存消耗从14M降低到12M,也就是等于OS的大小(M参数个数)。 也就是fusion FWD,BWD and ADAM 有一个paper支持了我这个想法: OPTIMIZER FUSION: EFFICIENT TRAINING WITH BETTER LOCALITY AND PARALLELISM https://arxiv.org/pdf/2104.00237.pdf

    opened by feifeibear 5
  • Support partial chunk management

    Support partial chunk management

    As we hope to support MoE in #187, and MoE is mainly of model parallel structure instead of data parallel, we need to support managing only part of the model with chunk.

    There are several design choices need to make, including but not limited to:

    • Shall we use mixed precision training in the unmanaged part of the model
    • How could we connect the backward of the unmanaged parts with the managed parts, i.e. if there are 3 parts in the model:
      class Net(nn.Module):
        def __init__(self, ...):
          self.A = SubNetA(...)  # managed by chunk
          self.B = SubNetB(...)  # not managed by chunk
          self.C = SubNetC(...)  # managed by chunk
      

      Then self.A and self.C need model.backward(loss) while self.B only need loss.backward().

    cc @feifeibear

    enhancement 
    opened by zhuzilin 4
  • RuntimeError: chunk move failed.

    RuntimeError: chunk move failed.

    While training a GPT3_6B model on 4x v100, the program stop because of runtime error at step 47. The exception show like this:

    RuntimeError: chunk move failed. cpu has not 385.875968 MB memory space. Free space is 320.948224 MB. The reason may be that the overall memory of CPU and GPU is not enough for the model.

    But the training progress only cost like 60% of the cpu memory, and the overall_cpu_mem_ratio is 0.9. 76127907-bcb1-4c69-91a0-10425b19874f

    opened by ouyangliqi 3
  • support using PatrickStar on MegatronDeepSpeed?

    support using PatrickStar on MegatronDeepSpeed?

    PatrickStar is awesome, it helps reduce memory used by model state!

    currently trends show using MegatronDeepSpeed as framework to train transformer based NLP models, both pretrain and finetuing. so will you guys support PatrickStar on MegatronDeepSpeed?

    opened by Jack47 3
  • Improve Memory Saving Communication.

    Improve Memory Saving Communication.

    Memory saving communication (MSC): using one-to-all communication to replace the original collective communication. More specifically, reduce scatter is replaced with Nx reduce. all gather is replaced with Nx bcast. In this way, we do not need to keep a Nx chunk buffer for distributed training, therefore save the GPU memory. I have implemented MSC but may not be optimal. I triggered communication in the granularity of the chunk group (Nx chunk). This may lead to chunks frequently move between CPU and GPU.

    This MR further optimizes the MSC. Communication is triggered in the granularity of chunks.

    opened by feifeibear 3
  • Add cuda event to enable async move

    Add cuda event to enable async move

    This MR introduce the compute_finished_event for each chunk to both enable async move and prevent the error solved by #243 .

    This MR would reduce the running time of GPT_DS_20B model from 28s to 25s.

    before:

    LOSS of step 4: 36.15625
    After step 4. using patrickstar, gradient checkpoint: True, fp16 True
    MA 24654.34 MB         Max_MA 26702.34 MB         CA 35922.0 MB         Max_CA 35922 MB 
    CPU Virtual Memory: used = 383.02 GB, percent = 38.0%
    Step 4 elaspe 28.63487219810486 s, 93.65687834254476 Tflops
    CLIENT_access_dist ........... 7.611969232559204, 26.592323953642254 %
    CLIENT_release ............... 0.05824542045593262, 0.20347968341163614 %
    CHUNK_LIST_prepare_device .... 2.1636338233947754, 7.558629021077558 %
    chunk_cpu_gpu_move ........... 5.4131646156311035, 18.910826183786217 %
    chunk_gpu_cpu_move ........... 4.49863076210022, 15.715913046770075 %
    CHUNK_LIST_chunk_move ........ 4.499696254730225, 15.719635332596733 %
    FWD .......................... 5.740366220474243, 20.05390109755791 %
    CLIENT_release_dist .......... 0.013424396514892578, 0.04689796951348792 %
    CHUNK_LIST_make_room ......... 2.359687566757202, 8.243540441044644 %
    BWD .......................... 12.1508309841156, 42.44878348344568 %
    ADAM_prepare_data_grad_copy .. 2.031972646713257, 7.098672266725892 %
    ADAM_prepare_data ............ 2.0699024200439453, 7.231179478602624 %
    ADAM_compute ................. 5.176488876342773, 18.084002304335637 %
    ADAM_param_fp32_to_fp16 ...... 3.381661891937256, 11.813795587537914 %
    ADAM_release_data ............ 0.041625261306762695, 0.14541735515558452 %
    CLIENT_access ................ 0.02957916259765625, 0.10333445262887372 %
    ADAM ......................... 10.73348879814148, 37.49731541899641 %
    TOTAL ........................ 28.624686002731323
    ------------- DATA MOVE RESULTS --------------
    chunk_cpu_gpu_move: 456704.0 MB, 446 times, 84369.13200112506 MB/s
    chunk_gpu_cpu_move: 434176.0 MB, 424 times, 96512.9220334815 MB/s
    ADAM_prepare_data_grad_copy: 195130.4687690735 MB, 2045 times, 96030.06668652732 MB/s
    ADAM_param_fp32_to_fp16: 390260.937538147 MB, 2045 times, 115405.07301118085 MB/s
    ******************** LOSS ********************
    [1.125, 51.21875, 107.125, 74.75, 36.15625]
    

    After:

    CPU Virtual Memory: used = 383.02 GB, percent = 38.0%
    Step 4 elaspe 25.567237615585327 s, 104.89411418374799 Tflops
    CLIENT_access_dist ........... 4.2600257396698, 16.6691598791813 %
    CLIENT_release ............... 0.057218313217163086, 0.22389095027108755 %
    CHUNK_LIST_prepare_device .... 1.330047607421875, 5.204376116458931 %
    chunk_cpu_gpu_move ........... 2.8953936100006104, 11.329457131871948 %
    chunk_gpu_cpu_move ........... 2.739708662033081, 10.720273655751937 %
    CHUNK_LIST_chunk_move ........ 2.74080228805542, 10.724552932011854 %
    FWD .......................... 3.507220506668091, 13.723489699319316 %
    CLIENT_release_dist .......... 0.013592243194580078, 0.05318542393237539 %
    CHUNK_LIST_make_room ......... 1.4351551532745361, 5.61565402729667 %
    BWD .......................... 11.206431150436401, 43.84992108900761 %
    ADAM_prepare_data_grad_copy .. 2.0489084720611572, 8.017224539409424 %
    ADAM_prepare_data ............ 2.0858242511749268, 8.161673202801694 %
    ADAM_compute ................. 5.232354164123535, 20.47380777399615 %
    ADAM_param_fp32_to_fp16 ...... 3.4178943634033203, 13.373963228245337 %
    ADAM_release_data ............ 0.0403749942779541, 0.1579843118019314 %
    CLIENT_access ................ 0.02886509895324707, 0.1129469582541441 %
    ADAM ......................... 10.842679738998413, 42.42658921167307 %
    TOTAL ........................ 25.556331396102905
    ------------- DATA MOVE RESULTS --------------
    chunk_cpu_gpu_move: 456704.0 MB, 446 times, 157734.68533692858 MB/s
    chunk_gpu_cpu_move: 434176.0 MB, 424 times, 158475.2444728218 MB/s
    ADAM_prepare_data_grad_copy: 195130.4687690735 MB, 2045 times, 95236.30334388558 MB/s
    ADAM_param_fp32_to_fp16: 390260.937538147 MB, 2045 times, 114181.68499202828 MB/s
    ******************** LOSS ********************
    [1.125, 51.21875, 107.125, 74.75, 36.15625]
    
    opened by zhuzilin 3
  • Mem cache to avoid too much allocation and free.

    Mem cache to avoid too much allocation and free.

    针对payload allocate提升还是比较大的

    40B 8GPU两个方案对比,差别只有是否使用mem_cache log.GPT_DS_40B_gpu_8_cs_384_bs_8_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_1 CHUNK_allocate_payload_cuda ........... 4.351799488067627, 4.512294773814017 % log.GPT_DS_40B_gpu_8_cs_384_bs_8_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_0 CHUNK_allocate_payload_cuda ........... 6.351553440093994, 6.1092681342726864 %

    40B 4GPU两个方案对比,差别只有是否使用mem_cache log.GPT_DS_40B_gpu_4_cs_384_bs_8_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_1 CHUNK_allocate_payload_cuda ........... 1.5783910751342773, 2.8674014523329285 % log.GPT_DS_40B_gpu_4_cs_384_bs_8_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_0 CHUNK_allocate_payload_cuda ........... 6.586869716644287, 7.966604944847982 %

    log.GPT_DS_20B_gpu_4_cs_384_bs_4_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_1 CHUNK_allocate_payload_cuda ........... 1.8855564594268799, 8.558600729826443 % log.GPT_DS_20B_gpu_4_cs_384_bs_4_cpueb_0_lightseq_0_offload_0_SP_0_AMM_1_MSC_1_CACHE_0 CHUNK_allocate_payload_cuda ........... 7.258604526519775, 52.21233450104307 %

    opened by feifeibear 3
  • Move the gradients of torch based params to CPU before Adam

    Move the gradients of torch based params to CPU before Adam

    The CPUAdam requires the torch based params to be on CPU before updation, otherwise we need to move the momentum and variance of the tensor to GPU. There are 2 design I could think of:

    1. Load the torch based params to CPU the entire forward and backward computation and offload them in the loop of adam.
    2. Load them to GPU before modules and offload after modules for both forward and backward.

    This PR implements the latter design, which may introduce more offloading overhead, but more consistent with other parts of the current design.

    opened by zhuzilin 3
  • 目前Chunk Reuse的局限

    目前Chunk Reuse的局限

    现在chunk reuse方式可以将overall memory footprint从DeepSpeed的18M降低到14M(M是参数量)。但是派大星目前实现有局限。派大星采用静态方式去设计重用方案。在训练开始前,它规定param fp16所在的chunk内存,被grad fp16复用。这种方式前提是参数不会被BWD更新两边。对于BERT和GPT不会有任何问题,但是对于LSTM和seq2seq transformer不能work。

    针对后者,我们可以改成一种动态重用方式。也就是BWD时候实时监测chunk的空隙(param fp16不用时候可以释放出chunk的空隙),把grad fp16内存分配在空隙处。 不过LSTM和seq2seq目前很少有超大模型需求。我们可以暂时保留这个需求,等到必要时再去做。

    enhancement 
    opened by feifeibear 3
  • Support communication config before training

    Support communication config before training

    Currently, the training will start whether the config of 2 nodes are the same or not. This may cause some weird result during benchmarking. We should consider communicate the config among nodes to make sure they are running the same program...

    opened by zhuzilin 0
  • PatrickStar's Performance in Models Like GANs

    PatrickStar's Performance in Models Like GANs

    Hi! I am a newbie in this field. DeepSpeed provides a tutorial on GAN (https://www.deepspeed.ai/tutorials/gan/). I am curious about PatrickStar's performance in models like GANs or other CV models. I really hope that PatrickStar can make my poor GPU accommodate a large-scale GAN.

    opened by openRiemann 2
  • 支持TencentPretrain

    支持TencentPretrain

    TencentPretrain是TEG数据安全中心的repo,我们可以利用它们的模型结构和数据 https://git.woa.com/TencentNLP/TencentPretrain/merge_requests/61 TencentPretrain还有一个野生开源项目 https://github.com/dbiir/UER-py

    documentation 
    opened by feifeibear 5
  • Add CI

    Add CI

    We would like to have a CI to run unitests each time an MR proposed to branch develop and master. However, we currently have no idea how to find a GPU to run the unitests. Does anyone have ideas?

    help wanted 
    opened by feifeibear 1
  • 我们真的需要模型并行(MP)么?

    我们真的需要模型并行(MP)么?

    MP的风潮是Megatron-LM引入到PTM训练中的,通过对transformer的实现插入定制的集合通信操作,实现了模型切分。 模型并行有很多诟病,

    1. 在FWD和BWD都有大量的activations全局通信,通信量和batch size成正比。不仅通信量大于DP,还限制了batch size从而限制MP训练的计算负载规模,影响了计算性能(越大batch计算效率越高)。
    2. MP需要对model定义代码进行定制修改。因此DeepSpeed的Example中也是在Megatron-LM基础上改的。有一些工作尝试简化这个修改工作,比如Mesh-TensorFlow和阿里巴巴的Whale,PyTorch似乎没有相关工作。如果从刷性能角度,这样并无大碍。如果从使用角度,算法同学不会接受的,因为推理端的代码还需要把自定义并行算子转化成PyTorch串行的。
    3. 在HP(异构并行),MP,PP,DP等组合下,MP的用法已经非常局限,并有被替代之势。DeepSpeed吧MP被安排在节点内并行,PP和DP用在节点间。HP+DP的引入,让GPU内存墙被进一步打破,模型并行的主要优势正在被HP和ZeroDP代替,以后节点内是否继续用MP都不一定。

    MP and PatrickStar

    在PatrickStar中,显存的最大消耗量和chunk size有关,即使不使用异构存储空间,把所有chunk都放在gpu中,model data的尺寸也是原来的1/N,和MP消耗类似。PatrickStar和PP兼容即可,不需要兼容MP。 之前Zero-Offload会去兼容MP,这是很奇怪的。阅读代码,我觉得是因为Zero3的通信用了非常差的设计,需要临时在gpu分配world_size*tensor_numel大小的临时buffer,加上预取的存在,可能同时分配了多个这样的buffer,尤其对于embedding layer这种大参数层,可能会爆炸内存,因此需要用MP减少单个进程的tensor_numel。

    documentation 
    opened by feifeibear 5
Releases(v0.4.6)
Owner
Tencent
Tencent
SparseML is a libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models

SparseML is a toolkit that includes APIs, CLIs, scripts and libraries that apply state-of-the-art sparsification algorithms such as pruning and quantization to any neural network. General, recipe-driven approaches built around these algorithms enable the simplification of creating faster and smaller models for the ML performance community at large.

Neural Magic 1.5k Dec 30, 2022
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 170.1k Jan 4, 2023
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 170.1k Jan 5, 2023
An Open Source Machine Learning Framework for Everyone

Documentation TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries, a

null 153.2k Feb 13, 2021
Repository providing a wide range of self-supervised pretrained models for computer vision tasks.

Hierarchical Pretraining: Research Repository This is a research repository for reproducing the results from the project "Self-supervised pretraining

Colorado Reed 53 Nov 9, 2022
(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

Res2Net Applications 928 Dec 29, 2022
Pretrained Pytorch face detection (MTCNN) and recognition (InceptionResnet) models

Face Recognition Using Pytorch Python 3.7 3.6 3.5 Status This is a repository for Inception Resnet (V1) models in pytorch, pretrained on VGGFace2 and

Tim Esler 3.3k Jan 4, 2023
Pretrained models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet.

Pretrained models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet.

Matthias Wright 169 Dec 26, 2022
This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

UCPhrase: Unsupervised Context-aware Quality Phrase Tagging To appear on KDD'21...[pdf] This project provides an unsupervised framework for mining and

Xiaotao Gu 146 Dec 22, 2022
Official PyTorch implementation and pretrained models of the paper Self-Supervised Classification Network

Self-Classifier: Self-Supervised Classification Network Official PyTorch implementation and pretrained models of the paper Self-Supervised Classificat

Elad Amrani 24 Dec 21, 2022
Using pretrained language models for biomedical knowledge graph completion.

LMs for biomedical KG completion This repository contains code to run the experiments described in: Scientific Language Models for Biomedical Knowledg

Rahul Nadkarni 41 Nov 30, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Measuring and Improving Consistency in Pretrained Language Models

ParaRel ?? This repository contains the code and data for the paper: Measuring and Improving Consistency in Pretrained Language Models as well as the

Yanai Elazar 26 Dec 2, 2022
Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

Facebook Research 408 Jan 1, 2023
A library for finding knowledge neurons in pretrained transformer models.

knowledge-neurons An open source repository replicating the 2021 paper Knowledge Neurons in Pretrained Transformers by Dai et al., and extending the t

EleutherAI 96 Dec 21, 2022
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

VisualGPT Our Paper VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning Main Architecture of Our VisualGPT Downloa

Vision CAIR Research Group, KAUST 140 Dec 28, 2022
YOLOv5 🚀 is a family of object detection architectures and models pretrained on the COCO dataset

YOLOv5 ?? is a family of object detection architectures and models pretrained on the COCO dataset, and represents Ultralytics open-source research int

阿才 73 Dec 16, 2022
Music Source Separation; Train & Eval & Inference piplines and pretrained models we used for 2021 ISMIR MDX Challenge.

Music Source Separation with Channel-wise Subband Phase Aware ResUnet (CWS-PResUNet) Introduction This repo contains the pretrained Music Source Separ

Lau 100 Dec 25, 2022
Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)

This is a playground for pytorch beginners, which contains predefined models on popular dataset. Currently we support mnist, svhn cifar10, cifar100 st

Aaron Chen 2.4k Dec 28, 2022