DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

Overview

DeeBERT

This is the code base for the paper DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference.

Code in this repository is also available in the Huggingface Transformer repo (with minor modification for version compatibility). Check this page for models that we have trained in advance (the latest version of Huggingface Transformers Library is needed).

Installation

This repo is tested on Python 3.7.5, PyTorch 1.3.1, and Cuda 10.1. Using a virtulaenv or conda environemnt is recommended, for example:

conda install pytorch==1.3.1 torchvision cudatoolkit=10.1 -c pytorch

After installing the required environment, clone this repo, and install the following requirements:

git clone https://github.com/castorini/deebert
cd deebert
pip install -r ./requirements.txt
pip install -r ./examples/requirements.txt

Usage

There are four scripts in the scripts folder, which can be run from the repo root, e.g., scripts/train.sh.

In each script, there are several things to modify before running:

  • path to the GLUE dataset. Check this for more details.
  • path for saving fine-tuned models. Default: ./saved_models.
  • path for saving evaluation results. Default: ./plotting. Results are printed to stdout and also saved to npy files in this directory to facilitate plotting figures and further analyses.
  • model_type (bert or roberta)
  • model_size (base or large)
  • dataset (SST-2, MRPC, RTE, QNLI, QQP, or MNLI)

train.sh

This is for fine-tuning and evaluating models as in the original BERT paper.

train_highway.sh

This is for fine-tuning DeeBERT models.

eval_highway.sh

This is for evaluating each exit layer for fine-tuned DeeBERT models.

eval_entropy.sh

This is for evaluating fine-tuned DeeBERT models, given a number of different early exit entropy thresholds.

Citation

Please cite our paper if you find the repository useful:

@inproceedings{xin-etal-2020-deebert,
    title = "{D}ee{BERT}: Dynamic Early Exiting for Accelerating {BERT} Inference",
    author = "Xin, Ji  and
      Tang, Raphael  and
      Lee, Jaejun  and
      Yu, Yaoliang  and
      Lin, Jimmy",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.204",
    pages = "2246--2251",
}
Comments
  • Replication results

    Replication results

    Ubuntu 18.10 Python 3.7.7 CUDA 10.1

    train.sh (time 0.09) acc = 0.8676470588235294 f1 = 0.906896551724138

    train_highway.sh (time 0.13) acc = 0.8676470588235294 f1 = 0.9072164948453608

    eval_entropy.sh 0.0 Result: 0.9072164948453608 Eval time: 14.078018426895142 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 408} Expected saving 1.0

    0.001 Result: 0.9072164948453608 Eval time: 13.798500061035156 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 408} Expected saving 1.0

    0.005 Result: 0.9072164948453608 Eval time: 14.127072811126709 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 408} Expected saving 1.0

    0.01 Result: 0.9072164948453608 Eval time: 13.872061491012573 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 408} Expected saving 1.0

    0.05 Result: 0.9072164948453608 Eval time: 13.464979648590088 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 49, 11: 62, 12: 297} Expected saving 0.9673202614379085

    0.1 Result: 0.9072164948453608 Eval time: 12.848443746566772 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 4, 7: 0, 8: 1, 9: 10, 10: 110, 11: 58, 12: 225} Expected saving 0.9313725490196079

    0.15 Result: 0.9090909090909091 Eval time: 12.455684423446655 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 21, 7: 13, 8: 5, 9: 19, 10: 112, 11: 54, 12: 184} Expected saving 0.8884803921568627

    0.2 Result: 0.9094017094017094 Eval time: 11.937445402145386 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 58, 7: 19, 8: 3, 9: 23, 10: 107, 11: 44, 12: 154} Expected saving 0.8402777777777778

    0.3 Result: 0.8989898989898989 Eval time: 10.921518564224243 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 0, 5: 9, 6: 120, 7: 21, 8: 1, 9: 22, 10: 88, 11: 46, 12: 101} Expected saving 0.7589869281045751

    0.4 Result: 0.8870151770657673 Eval time: 9.648621082305908 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 1, 5: 69, 6: 117, 7: 18, 8: 3, 9: 31, 10: 72, 11: 37, 12: 60} Expected saving 0.6795343137254902

    0.5 Result: 0.8795986622073578 Eval time: 8.683350801467896 Exit layer counter {1: 0, 2: 0, 3: 0, 4: 61, 5: 105, 6: 86, 7: 19, 8: 5, 9: 27, 10: 43, 11: 31, 12: 31} Expected saving 0.5808823529411765

    0.6 Result: 0.8516129032258065 Eval time: 6.725062131881714 Exit layer counter {1: 1, 2: 56, 3: 20, 4: 158, 5: 35, 6: 66, 7: 12, 8: 4, 9: 13, 10: 16, 11: 13, 12: 14} Expected saving 0.42483660130718953

    0.7 Result: 0.8122270742358079 Eval time: 2.7605879306793213 Exit layer counter {1: 408, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0, 11: 0, 12: 0} Expected saving 0.08333333333333333

    opened by wongalvis14 5
  • RuntimeError: Boolean value of Tensor with more than one value is ambiguous

    RuntimeError: Boolean value of Tensor with more than one value is ambiguous

    Hi there I was using your model for my new classification task on Japanese text so I added the configuration needed for Japanese Bert I'm doing the training on the BertForSequenceClassifcation model and when I do validation I encountered a bug that said the if condition was done on a tensor with more than one value (that tensor is actually the highway entropy of a batch) so that threw a runtime error when compared with the early exit threshold Can you help me with this ? I'd appreciate your help on this problem Here is the snippet of the error code image

    opened by quanhoang288 3
  • Entropy calculation can be modified

    Entropy calculation can be modified

    When calculating entropy, dim=1 is better to be replaced with dim=-1, since num_labels is the last dimension of logits but not always the 2nd dimension (e.g., in Token Classification, num_labels is the 3rd dimension).

    And what about using softmax directly?

    def entropy(x):  # my attempt
        x = torch.softmax(x, dim=-1)               # softmax normalized prob distribution
        return -torch.sum(x*torch.log(x), dim=-1)  # entropy calculation on probs: -\sum(p \ln(p))
    
    def entropy(x):  # original code
        exp_x = torch.exp(x)
        A = torch.sum(exp_x, dim=1)    # sum of exp(x_i)
        B = torch.sum(x*exp_x, dim=1)  # sum of x_i * exp(x_i)
        return torch.log(A) - B/A
    

    It seems that the outputs are no different, and using softmax is generally more efficient than manual calculation.

    Since DeeBERT is proposed for efficiency, it's better to choose the efficient way.

    Mind if I open a PR for this? @ji-xin

    opened by sbwww 1
  • Confusing code in class BertHighway in modeling_highway_bert.py

    Confusing code in class BertHighway in modeling_highway_bert.py

    Greetings, DeeBERT is really a crucial and easy-to-understand achievement in BERT inference acceleration.

    However, in transformers/modeling_highway_bert.py, the forward function of class BertHighway is a bit confusing. Your original code is as follows

    def forward(self, encoder_outputs):
        # Pooler
        pooler_input = encoder_outputs[0]
        pooler_output = self.pooler(pooler_input)
        # "return" pooler_output
    
        # BertModel
        bmodel_output = (pooler_input, pooler_output) +encoder_outputs[1:]
        # "return" bodel_output
    
        # Dropout and classification
        pooled_output = bmodel_output[1]
    
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)
    
        return logits, pooled_output
    

    I am not quite sure about introducing bmodel_output for it's not actually used.

    It seems that pooler_input is sequence_output, and pooled_output is equivalent to pooler_output.

    Is there any trick that should be noticed? Maybe the comments starting with "return" can be updated for more details.

    opened by sbwww 1
  • Bump tensorflow from 2.0.0rc2 to 2.5.0rc0 in /docs

    Bump tensorflow from 2.0.0rc2 to 2.5.0rc0 in /docs

    Bumps tensorflow from 2.0.0rc2 to 2.5.0rc0.

    Release notes

    Sourced from tensorflow's releases.

    TensorFlow 2.5.0-rc0

    Release 2.5.0

    Major Features and Improvements

    • TPU embedding support
      • Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.
    • tf.keras.metrics.AUC now support logit predictions.
    • Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.
    • tf.data:
      • tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.
      • tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).
      • tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.
      • Options returned by tf.data.Dataset.options() are no longer mutable.
      • tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().
    • tf.lite
      • Enabled the new MLIR-based quantization backend by default
        • The new backend is used for 8 bits full integer post-training quantization
        • The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)
        • Set experimental_new_quantizer in tf.lite.TFLiteConverter to False to disable this change
    • tf.keras
      • Enabled a new supported input type in Model.fit, tf.keras.utils.experimental.DatasetCreator, which takes a callable, dataset_fn. DatasetCreator is intended to work across all tf.distribute strategies, and is the only input type supported for Parameter Server strategy.
    • tf.distribute
      • tf.distribute.experimental.ParameterServerStrategy now supports training with Keras Model.fit when used with DatasetCreator.
    • PluggableDevice

    ... (truncated)

    Changelog

    Sourced from tensorflow's changelog.

    Release 2.6.0

    Breaking Changes

    • tf.train.experimental.enable_mixed_precision_graph_rewrite is removed, as the API only works in graph mode and is not customizable. The function is still accessible under tf.compat.v1.mixed_precision.enable_mixed_precision_graph_rewrite, but it is recommended to use the Keras mixed precision API instead.

    • tf.lite:

      • Remove experimental.nn.dynamic_rnn, experimental.nn.TfLiteRNNCell and experimental.nn.TfLiteLSTMCell since they're no longer supported. It's recommended to just use keras lstm instead.

    * *

    Known Caveats

    * * *

    • TF Core:
      • A longstanding bug in tf.while_loop, which caused it to execute sequentially, even when parallel_iterations>1, has now been fixed. However, the increased parallelism may result in increased memory use. Users who experience unwanted regressions should reset their while_loop's parallel_iterations value to 1, which is consistent with prior behavior.

    Major Features and Improvements

    * *

    • tf.keras:
      • tf.keras.utils.experimental.DatasetCreator now takes an optional tf.distribute.InputOptions for specific options when used with distribution.
      • Updates to Preprocessing layers API for consistency and clarity:
        • StringLookup and IntegerLookup default for mask_token changed to None. This matches the default masking behavior of Hashing and Embedding layers. To keep existing behavior, pass mask_token="" during layer creation.

    ... (truncated)

    Commits
    • a8b6d5f Merge pull request #48222 from tensorflow/mm-fix-fileystem-on-r2.5
    • b9e31e6 Fix typo/logic bug in modular plugins.
    • 158505e Switch TF filesystems to keep backwards compatibility.
    • 96dfa5c Merge pull request #48107 from tensorflow/mihaimaruseac-patch-1
    • 5f7fd89 Fix typo in setup.py
    • f8b5b9b Merge pull request #48093 from tensorflow/mihaimaruseac-patch-1
    • b84dac5 Update setup.py
    • b42047d Merge pull request #48091 from tensorflow-jenkins/version-numbers-2.5.0rc0-30114
    • 1d4885b Update version numbers to 2.5.0-rc0
    • 6af4297 Merge pull request #48082 from njeffrie:f1_depthwise
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump tensorflow from 2.0.0 to 2.5.0

    Bump tensorflow from 2.0.0 to 2.5.0

    Bumps tensorflow from 2.0.0 to 2.5.0.

    Release notes

    Sourced from tensorflow's releases.

    TensorFlow 2.5.0

    Release 2.5.0

    Major Features and Improvements

    • Support for Python3.9 has been added.
    • tf.data:
      • tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.
      • tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).
      • tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.
      • Options returned by tf.data.Dataset.options() are no longer mutable.
      • tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().
    • tf.lite
      • Enabled the new MLIR-based quantization backend by default
        • The new backend is used for 8 bits full integer post-training quantization
        • The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)
        • Set experimental_new_quantizer in tf.lite.TFLiteConverter to False to disable this change
    • tf.keras
      • tf.keras.metrics.AUC now support logit predictions.
      • Enabled a new supported input type in Model.fit, tf.keras.utils.experimental.DatasetCreator, which takes a callable, dataset_fn. DatasetCreator is intended to work across all tf.distribute strategies, and is the only input type supported for Parameter Server strategy.
    • tf.distribute
      • tf.distribute.experimental.ParameterServerStrategy now supports training with Keras Model.fit when used with DatasetCreator.
      • Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.
    • TPU embedding support
      • Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.
    • PluggableDevice
    • oneAPI Deep Neural Network Library (oneDNN) CPU performance optimizations from Intel-optimized TensorFlow are now available in the official x86-64 Linux and Windows builds.
      • They are off by default. Enable them by setting the environment variable TF_ENABLE_ONEDNN_OPTS=1.
      • We do not recommend using them in GPU systems, as they have not been sufficiently tested with GPUs yet.
    • TensorFlow pip packages are now built with CUDA11.2 and cuDNN 8.1.0

    Breaking Changes

    • The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

    Bug Fixes and Other Changes

    • tf.keras:
      • Preprocessing layers API consistency changes:
        • StringLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization.
        • IntegerLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization. Renamed max_values, oov_value and mask_value to max_tokens, oov_token and mask_token to align with StringLookup and TextVectorization.
        • TextVectorization default for pad_to_max_tokens switched to False.
        • CategoryEncoding no longer supports adapt, IntegerLookup now supports equivalent functionality. max_tokens argument renamed to num_tokens.
        • Discretization added num_bins argument for learning bins boundaries through calling adapt on a dataset. Renamed bins argument to bin_boundaries for specifying bins without adapt.
      • Improvements to model saving/loading:
        • model.load_weights now accepts paths to saved models.

    ... (truncated)

    Changelog

    Sourced from tensorflow's changelog.

    Release 2.5.0

    Breaking Changes

    • The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

    Known Caveats

    Major Features and Improvements

    • TPU embedding support

      • Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.
    • tf.keras.metrics.AUC now support logit predictions.

    • Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.

    • tf.data:

      • tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.
      • tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).
      • tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.
      • Options returned by tf.data.Dataset.options() are no longer mutable.
      • tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().
    • tf.lite

      • Enabled the new MLIR-based quantization backend by default
        • The new backend is used for 8 bits full integer post-training quantization
        • The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)

    ... (truncated)

    Commits
    • a4dfb8d Merge pull request #49124 from tensorflow/mm-cherrypick-tf-data-segfault-fix-...
    • 2107b1d Merge pull request #49116 from tensorflow-jenkins/version-numbers-2.5.0-17609
    • 16b8139 Update snapshot_dataset_op.cc
    • 86a0d86 Merge pull request #49126 from geetachavan1/cherrypicks_X9ZNY
    • 9436ae6 Merge pull request #49128 from geetachavan1/cherrypicks_D73J5
    • 6b2bf99 Validate that a and b are proper sparse tensors
    • c03ad1a Ensure validation sticks in banded_triangular_solve_op
    • 12a6ead Merge pull request #49120 from geetachavan1/cherrypicks_KJ5M9
    • b67f5b8 Merge pull request #49118 from geetachavan1/cherrypicks_BIDTR
    • a13c0ad [tf.data][cherrypick] Fix snapshot segfault when using repeat and prefecth
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump urllib3 from 1.25.6 to 1.25.8

    Bump urllib3 from 1.25.6 to 1.25.8

    Bumps urllib3 from 1.25.6 to 1.25.8.

    Release notes

    Sourced from urllib3's releases.

    1.25.8

    Release: 1.25.8

    1.25.7

    No release notes provided.

    Changelog

    Sourced from urllib3's changelog.

    1.25.8 (2020-01-20)

    • Drop support for EOL Python 3.4 (Pull #1774)

    • Optimize _encode_invalid_chars (Pull #1787)

    1.25.7 (2019-11-11)

    • Preserve chunked parameter on retries (Pull #1715, Pull #1734)

    • Allow unset SERVER_SOFTWARE in App Engine (Pull #1704, Issue #1470)

    • Fix issue where URL fragment was sent within the request target. (Pull #1732)

    • Fix issue where an empty query section in a URL would fail to parse. (Pull #1732)

    • Remove TLS 1.3 support in SecureTransport due to Apple removing support (Pull #1703)

    Commits
    • 2a57bc5 Release 1.25.8 (#1788)
    • a2697e7 Optimize _encode_invalid_chars (#1787)
    • d2a5a59 Move IPv6 test skips in server fixtures
    • d44f0e5 Factorize test certificates serialization
    • 84abc7f Generate IPV6 certificates using trustme
    • 6a15b18 Run IPv6 Tornado server from fixture
    • 4903840 Use trustme to generate IP_SAN cert
    • 9971e27 Empty responses should have no lines.
    • 62ef68e Use trustme to generate NO_SAN certs
    • fd2666e Use fixture to configure NO_SAN test certs
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump urllib3 from 1.25.3 to 1.25.8 in /docs

    Bump urllib3 from 1.25.3 to 1.25.8 in /docs

    Bumps urllib3 from 1.25.3 to 1.25.8.

    Release notes

    Sourced from urllib3's releases.

    1.25.8

    Release: 1.25.8

    1.25.7

    No release notes provided.

    1.25.6

    Release: 1.25.6

    1.25.5

    Release: 1.25.5

    1.25.4

    Release: 1.25.4

    Changelog

    Sourced from urllib3's changelog.

    1.25.8 (2020-01-20)

    • Drop support for EOL Python 3.4 (Pull #1774)

    • Optimize _encode_invalid_chars (Pull #1787)

    1.25.7 (2019-11-11)

    • Preserve chunked parameter on retries (Pull #1715, Pull #1734)

    • Allow unset SERVER_SOFTWARE in App Engine (Pull #1704, Issue #1470)

    • Fix issue where URL fragment was sent within the request target. (Pull #1732)

    • Fix issue where an empty query section in a URL would fail to parse. (Pull #1732)

    • Remove TLS 1.3 support in SecureTransport due to Apple removing support (Pull #1703)

    1.25.6 (2019-09-24)

    • Fix issue where tilde (~) characters were incorrectly percent-encoded in the path. (Pull #1692)

    1.25.5 (2019-09-19)

    • Add mitigation for BPO-37428 affecting Python <3.7.4 and OpenSSL 1.1.1+ which caused certificate verification to be enabled when using cert_reqs=CERT_NONE. (Issue #1682)

    1.25.4 (2019-09-19)

    • Propagate Retry-After header settings to subsequent retries. (Pull #1607)

    • Fix edge case where Retry-After header was still respected even when explicitly opted out of. (Pull #1607)

    • Remove dependency on rfc3986 for URL parsing.

    • Fix issue where URLs containing invalid characters within Url.auth would raise an exception instead of percent-encoding those characters.

    ... (truncated)

    Commits
    • 2a57bc5 Release 1.25.8 (#1788)
    • a2697e7 Optimize _encode_invalid_chars (#1787)
    • d2a5a59 Move IPv6 test skips in server fixtures
    • d44f0e5 Factorize test certificates serialization
    • 84abc7f Generate IPV6 certificates using trustme
    • 6a15b18 Run IPv6 Tornado server from fixture
    • 4903840 Use trustme to generate IP_SAN cert
    • 9971e27 Empty responses should have no lines.
    • 62ef68e Use trustme to generate NO_SAN certs
    • fd2666e Use fixture to configure NO_SAN test certs
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump py from 1.8.0 to 1.10.0

    Bump py from 1.8.0 to 1.10.0

    Bumps py from 1.8.0 to 1.10.0.

    Changelog

    Sourced from py's changelog.

    1.10.0 (2020-12-12)

    • Fix a regular expression DoS vulnerability in the py.path.svnwc SVN blame functionality (CVE-2020-29651)
    • Update vendored apipkg: 1.4 => 1.5
    • Update vendored iniconfig: 1.0.0 => 1.1.1

    1.9.0 (2020-06-24)

    • Add type annotation stubs for the following modules:

      • py.error
      • py.iniconfig
      • py.path (not including SVN paths)
      • py.io
      • py.xml

      There are no plans to type other modules at this time.

      The type annotations are provided in external .pyi files, not inline in the code, and may therefore contain small errors or omissions. If you use py in conjunction with a type checker, and encounter any type errors you believe should be accepted, please report it in an issue.

    1.8.2 (2020-06-15)

    • On Windows, py.path.locals which differ only in case now have the same Python hash value. Previously, such paths were considered equal but had different hashes, which is not allowed and breaks the assumptions made by dicts, sets and other users of hashes.

    1.8.1 (2019-12-27)

    • Handle FileNotFoundError when trying to import pathlib in path.common on Python 3.4 (#207).

    • py.path.local.samefile now works correctly in Python 3 on Windows when dealing with symlinks.

    Commits
    • e5ff378 Update CHANGELOG for 1.10.0
    • 94cf44f Update vendored libs
    • 5e8ded5 testing: comment out an assert which fails on Python 3.9 for now
    • afdffcc Rename HOWTORELEASE.rst to RELEASING.rst
    • 2de53a6 Merge pull request #266 from nicoddemus/gh-actions
    • fa1b32e Merge pull request #264 from hugovk/patch-2
    • 887d6b8 Skip test_samefile_symlink on pypy3 on Windows
    • e94e670 Fix test_comments() in test_source
    • fef9a32 Adapt test
    • 4a694b0 Add GitHub Actions badge to README
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump pygments from 2.4.2 to 2.7.4 in /docs

    Bump pygments from 2.4.2 to 2.7.4 in /docs

    Bumps pygments from 2.4.2 to 2.7.4.

    Release notes

    Sourced from pygments's releases.

    2.7.4

    • Updated lexers:

      • Apache configurations: Improve handling of malformed tags (#1656)

      • CSS: Add support for variables (#1633, #1666)

      • Crystal (#1650, #1670)

      • Coq (#1648)

      • Fortran: Add missing keywords (#1635, #1665)

      • Ini (#1624)

      • JavaScript and variants (#1647 -- missing regex flags, #1651)

      • Markdown (#1623, #1617)

      • Shell

        • Lex trailing whitespace as part of the prompt (#1645)
        • Add missing in keyword (#1652)
      • SQL - Fix keywords (#1668)

      • Typescript: Fix incorrect punctuation handling (#1510, #1511)

    • Fix infinite loop in SML lexer (#1625)

    • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

    • Limit recursion with nesting Ruby heredocs (#1638)

    • Fix a few inefficient regexes for guessing lexers

    • Fix the raw token lexer handling of Unicode (#1616)

    • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

    • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

    • Fix incorrect MATLAB example (#1582)

    Thanks to Google's OSS-Fuzz project for finding many of these bugs.

    2.7.3

    ... (truncated)

    Changelog

    Sourced from pygments's changelog.

    Version 2.7.4

    (released January 12, 2021)

    • Updated lexers:

      • Apache configurations: Improve handling of malformed tags (#1656)

      • CSS: Add support for variables (#1633, #1666)

      • Crystal (#1650, #1670)

      • Coq (#1648)

      • Fortran: Add missing keywords (#1635, #1665)

      • Ini (#1624)

      • JavaScript and variants (#1647 -- missing regex flags, #1651)

      • Markdown (#1623, #1617)

      • Shell

        • Lex trailing whitespace as part of the prompt (#1645)
        • Add missing in keyword (#1652)
      • SQL - Fix keywords (#1668)

      • Typescript: Fix incorrect punctuation handling (#1510, #1511)

    • Fix infinite loop in SML lexer (#1625)

    • Fix backtracking string regexes in JavaScript/TypeScript, Modula2 and many other lexers (#1637)

    • Limit recursion with nesting Ruby heredocs (#1638)

    • Fix a few inefficient regexes for guessing lexers

    • Fix the raw token lexer handling of Unicode (#1616)

    • Revert a private API change in the HTML formatter (#1655) -- please note that private APIs remain subject to change!

    • Fix several exponential/cubic-complexity regexes found by Ben Caller/Doyensec (#1675)

    • Fix incorrect MATLAB example (#1582)

    Thanks to Google's OSS-Fuzz project for finding many of these bugs.

    Version 2.7.3

    (released December 6, 2020)

    ... (truncated)

    Commits
    • 4d555d0 Bump version to 2.7.4.
    • fc3b05d Update CHANGES.
    • ad21935 Revert "Added dracula theme style (#1636)"
    • e411506 Prepare for 2.7.4 release.
    • 275e34d doc: remove Perl 6 ref
    • 2e7e8c4 Fix several exponential/cubic complexity regexes found by Ben Caller/Doyensec
    • eb39c43 xquery: fix pop from empty stack
    • 2738778 fix coding style in test_analyzer_lexer
    • 02e0f09 Added 'ERROR STOP' to fortran.py keywords. (#1665)
    • c83fe48 support added for css variables (#1633)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Bump jinja2 from 2.10.1 to 2.11.3 in /docs

    Bump jinja2 from 2.10.1 to 2.11.3 in /docs

    Bumps jinja2 from 2.10.1 to 2.11.3.

    Release notes

    Sourced from jinja2's releases.

    2.11.3

    This contains a fix for a speed issue with the urlize filter. urlize is likely to be called on untrusted user input. For certain inputs some of the regular expressions used to parse the text could take a very long time due to backtracking. As part of the fix, the email matching became slightly stricter. The various speedups apply to urlize in general, not just the specific input cases.

    2.11.2

    2.11.1

    This fixes an issue in async environment when indexing the result of an attribute lookup, like {{ data.items[1:] }}.

    2.11.0

    This is the last version to support Python 2.7 and 3.5. The next version will be Jinja 3.0 and will support Python 3.6 and newer.

    2.10.3

    2.10.2

    Changelog

    Sourced from jinja2's changelog.

    Version 2.11.3

    Released 2021-01-31

    • Improve the speed of the urlize filter by reducing regex backtracking. Email matching requires a word character at the start of the domain part, and only word characters in the TLD. :pr:1343

    Version 2.11.2

    Released 2020-04-13

    • Fix a bug that caused callable objects with __getattr__, like :class:~unittest.mock.Mock to be treated as a :func:contextfunction. :issue:1145
    • Update wordcount filter to trigger :class:Undefined methods by wrapping the input in :func:soft_str. :pr:1160
    • Fix a hang when displaying tracebacks on Python 32-bit. :issue:1162
    • Showing an undefined error for an object that raises AttributeError on access doesn't cause a recursion error. :issue:1177
    • Revert changes to :class:~loaders.PackageLoader from 2.10 which removed the dependency on setuptools and pkg_resources, and added limited support for namespace packages. The changes caused issues when using Pytest. Due to the difficulty in supporting Python 2 and :pep:451 simultaneously, the changes are reverted until 3.0. :pr:1182
    • Fix line numbers in error messages when newlines are stripped. :pr:1178
    • The special namespace() assignment object in templates works in async environments. :issue:1180
    • Fix whitespace being removed before tags in the middle of lines when lstrip_blocks is enabled. :issue:1138
    • :class:~nativetypes.NativeEnvironment doesn't evaluate intermediate strings during rendering. This prevents early evaluation which could change the value of an expression. :issue:1186

    Version 2.11.1

    Released 2020-01-30

    • Fix a bug that prevented looking up a key after an attribute ({{ data.items[1:] }}) in an async template. :issue:1141

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • The real inference acceleration is not GPU friendly---batch operator not supported

    The real inference acceleration is not GPU friendly---batch operator not supported

    When inference with the highway early-exit given a batch B; when |B| = 1, the code is ok to run; when |B| > 1, the code can corrupt in the code of modeling_highway_bert.py due to if highway_entropy < self.early_exit_entropy[i]: conditional statement ; Only one sample can be inferred. Therefore, the inference is not GPU friendly.

    opened by zhangzhenyu13 0
  • Adding Additional Transformer Models

    Adding Additional Transformer Models

    Hello, I was running some experiments with DeeBERT and was wondering if there is any set of steps for adding new models beyond the ones currently included in the repository. I see that the RoBERTa and BERT models are preloaded in a custom setup so I was wondering if there was any process for adding our own models. Thanks!

    opened by jonsaadfalcon 0
Owner
Castorini
Deep learning for natural language processing and information retrieval at the University of Waterloo
Castorini
Official implementation of "Accelerating Reinforcement Learning with Learned Skill Priors", Pertsch et al., CoRL 2020

Accelerating Reinforcement Learning with Learned Skill Priors [Project Website] [Paper] Karl Pertsch1, Youngwoon Lee1, Joseph Lim1 1CLVR Lab, Universi

Cognitive Learning for Vision and Robotics (CLVR) lab @ USC 134 Dec 6, 2022
source code for https://arxiv.org/abs/2005.11248 "Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics"

Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics This work will be published in Nature Biomedical

International Business Machines 71 Nov 15, 2022
Channel Pruning for Accelerating Very Deep Neural Networks (ICCV'17)

Channel Pruning for Accelerating Very Deep Neural Networks (ICCV'17)

Yihui He 1k Jan 3, 2023
The code for the NSDI'21 paper "BMC: Accelerating Memcached using Safe In-kernel Caching and Pre-stack Processing".

BMC The code for the NSDI'21 paper "BMC: Accelerating Memcached using Safe In-kernel Caching and Pre-stack Processing". BibTex entry available here. B

Orange 383 Dec 16, 2022
A PyTorch Library for Accelerating 3D Deep Learning Research

Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research Overview NVIDIA Kaolin library provides a PyTorch API for working with a variety

NVIDIA GameWorks 3.5k Jan 7, 2023
NPBG++: Accelerating Neural Point-Based Graphics

[CVPR 2022] NPBG++: Accelerating Neural Point-Based Graphics Project Page | Paper This repository contains the official Python implementation of the p

Ruslan Rakhimov 57 Dec 3, 2022
Sky Computing: Accelerating Geo-distributed Computing in Federated Learning

Sky Computing Introduction Sky Computing is a load-balanced framework for federated learning model parallelism. It adaptively allocate model layers to

HPC-AI Tech 72 Dec 27, 2022
I-BERT: Integer-only BERT Quantization

I-BERT: Integer-only BERT Quantization HuggingFace Implementation I-BERT is also available in the master branch of HuggingFace! Visit the following li

Sehoon Kim 139 Dec 27, 2022
LV-BERT: Exploiting Layer Variety for BERT (Findings of ACL 2021)

LV-BERT Introduction In this repo, we introduce LV-BERT by exploiting layer variety for BERT. For detailed description and experimental results, pleas

Weihao Yu 14 Aug 24, 2022
The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

null 124 Dec 27, 2022
VD-BERT: A Unified Vision and Dialog Transformer with BERT

VD-BERT: A Unified Vision and Dialog Transformer with BERT PyTorch Code for the following paper at EMNLP2020: Title: VD-BERT: A Unified Vision and Dia

Salesforce 44 Nov 1, 2022
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor

Pranaydeep Singh 22 Dec 8, 2022
[CVPR22] Official codebase of Semantic Segmentation by Early Region Proxy.

RegionProxy Figure 2. Performance vs. GFLOPs on ADE20K val split. Semantic Segmentation by Early Region Proxy Yifan Zhang, Bo Pang, Cewu Lu CVPR 2022

Yifan 54 Nov 29, 2022
Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready inference.

Yolov5 running on TorchServe (GPU compatible) ! This is a dockerfile to run TorchServe for Yolo v5 object detection model. (TorchServe (PyTorch librar

null 82 Nov 29, 2022
Monocular 3D pose estimation. OpenVINO. CPU inference or iGPU (OpenCL) inference.

human-pose-estimation-3d-python-cpp RealSenseD435 (RGB) 480x640 + CPU Corei9 45 FPS (Depth is not used) 1. Run 1-1. RealSenseD435 (RGB) 480x640 + CPU

Katsuya Hyodo 8 Oct 3, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

Marco 3 Feb 8, 2022
Simple Dynamic Batching Inference

Simple Dynamic Batching Inference 解决了什么问题? 众所周知,Batch对于GPU上深度学习模型的运行效率影响很大。。。 是在Inference时。搜索、推荐等场景自带比较大的batch,问题不大。但更多场景面临的往往是稀碎的请求(比如图片服务里一次一张图)。 如果

null 116 Jan 1, 2023
Code accompanying "Dynamic Neural Relational Inference" from CVPR 2020

Code accompanying "Dynamic Neural Relational Inference" This codebase accompanies the paper "Dynamic Neural Relational Inference" from CVPR 2020. This

Colin Graber 48 Dec 23, 2022