This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust.

Related tags

Deep Learning bert-onnx-rs-pipeline
Overview

Demo BERT ONNX pipeline written in rust

This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust.

Requirement

  • Linux x86_64
  • NVIDIA GPU with CUDA 11 (Not sure if CUDA 10 works)
  • Rust (obviously)

training and converting to ONNX (Python)

Installation

Download the data https://www.kaggle.com/START-UMD/gtd -> train_October_9_2012.csv and put it in the folder training.

cd training
pip install -r requirements.txt

Run

python pytorch-model.py

I have not done any parameter search to find an optimal model as it is not the point of this repo.

I have limited the training to 500 rows to iterate faster.

Inference using Python

cd src
python python_alternative.py

Inference using Rust

Installation

export ORT_USE_CUDA=1
cargo build --release

Run

cargo run --release

or after build:

export LD_LIBRARY_PATH=path/to/onnxruntime-linux-x64-gpu-1.8.0/lib:${LD_LIBRARY_PATH}
./target/release/machine-learning
You might also like...
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled -
Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Ancient Greek BERT The first and only available Ancient Greek sub-word BERT model! State-of-the-art post fine-tuning on Part-of-Speech Tagging and Mor

Pranaydeep Singh 22 Dec 8, 2022
Rust bindings for the C++ api of PyTorch.

tch-rs Rust bindings for the C++ api of PyTorch. The goal of the tch crate is to provide some thin wrappers around the C++ PyTorch api (a.k.a. libtorc

Laurent Mazare 2.3k Dec 30, 2022
Spatial color quantization in Rust
Spatial color quantization in Rust

rscolorq Rust port of Derrick Coetzee's scolorq, based on the 1998 paper "On spatial quantization of color images" by Jan Puzicha, Markus Held, Jens K

Collyn O'Kane 37 Dec 22, 2022
RLBot Python bindings for the Rust crate rl_ball_sym
RLBot Python bindings for the Rust crate rl_ball_sym

RLBot Python bindings for rl_ball_sym 0.6 Prerequisites: Rust & Cargo Build Tools for Visual Studio RLBot - Verify that the file %localappdata%\RLBotG

Eric Veilleux 2 Nov 25, 2022
Python/Rust implementations and notes from Proofs Arguments and Zero Knowledge

What is this? This is where I'll be collecting resources related to the Study Group on Dr. Justin Thaler's Proofs Arguments And Zero Knowledge Book. T

Thor 66 Jan 4, 2023
DWIPrep is a robust and easy-to-use pipeline for preprocessing of diverse dMRI data.

DWIPrep: A Robust Preprocessing Pipeline for dMRI Data DWIPrep is a robust and easy-to-use pipeline for preprocessing of diverse dMRI data. The transp

Gal Ben-Zvi 1 Jan 9, 2023
Extending JAX with custom C++ and CUDA code

Extending JAX with custom C++ and CUDA code This repository is meant as a tutorial demonstrating the infrastructure required to provide custom ops in

Dan Foreman-Mackey 237 Dec 23, 2022
Several simple examples for popular neural network toolkits calling custom CUDA operators.
Several simple examples for popular neural network toolkits calling custom CUDA operators.

Neural Network CUDA Example Several simple examples for neural network toolkits (PyTorch, TensorFlow, etc.) calling custom CUDA operators. We provide

WeiYang 798 Jan 1, 2023
Picasso: A CUDA-based Library for Deep Learning over 3D Meshes

The Picasso Library is intended for complex real-world applications with large-scale surfaces, while it also performs impressively on the small-scale applications over synthetic shape manifolds. We have upgraded the point cloud modules of SPH3D-GCN from homogeneous to heterogeneous representations, and included the upgraded modules into this latest work as well. We are happy to announce that the work is accepted to IEEE CVPR2021.

null 97 Dec 1, 2022
Comments
  • Install error

    Install error

    I tried to install your solution locally, and came up with 2 errors:

    1. Download the data https://www.kaggle.com/START-UMD/gtd -> train_October_9_2012.csv and put it in the folder training. By that link I have found file globalterrorismdb_0718dist.csv. If this file works, You should update filename, otherwise link to the file
    2. pip install -r requirements.txt this command produce error
    ERROR: Invalid requirement: 'version https://git-lfs.github.com/spec/v1' (from line 1 of requirement.txt)
Hint: It looks like a path. File 'version https://git-lfs.github.com/spec/v1' does not exist.
    opened by zanaviska 4
  • inference error

    inference error

    Running host: linux

    Since I cannot download data from train_October_9_2012.csv(because unknown error), I recreated small version of it:

    | X   | PostId   | PostCreationDate    | OwnerUserId | OwnerCreationDate   | ReputationAtPostCreation | OwnerUndeletedAnswerCountAtPostTime | Title                                             | BodyMarkdown                                      | Tag1       | Tag2            | Tag3  | Tag4 | Tag5 | PostClosedDate      | OpenStatus    |     |
| --- | -------- | ------------------- | ----------- | ------------------- | ------------------------ | ----------------------------------- | ------------------------------------------------- | ------------------------------------------------- | ---------- | --------------- | ----- | ---- | ---- | ------------------- | ------------- | --- |
| 1   | 10035653 | 2012-04-05 20:37:35 | 1159226     | 2012-01-19 18:46:16 | 1                        | 0                                   | what is the best way to connect my                | I know this question can be answered by           | c++        | NA              | NA    | NA   | NA   | 2012-04-05 23:31:34 | too localized |     |
| 2   | 6046168  | 05/18/2011 14:14:05 | 543315      | 09/17/2010 10:15:06 | 1                        | 2                                   | For Mongodb is it better to reference an objec... | I am building a corpus of indexed sentences in... | mongodb    | NA              | NA    | NA   | NA   | NA                  | open          |     |
| 3   | 4873911  | 02/02/2011 11:30:10 | 465076      | 10/03/2010 09:30:58 | 192                      | 24                                  | How to insert schemalocation in a xml document... | i create a xml document with JAXP and search a... | dom        | xsd             | jaxp  | NA   | NA   | NA                  | open          |     |
| 4   | 3311559  | 07/22/2010 17:21:54 | 406143      | 07/22/2010 16:58:20 | 1                        | 0                                   | Too many lookup tables                            | What are the adverse effects of having too man... | sql-server | database-design | enums | NA   | NA   | NA                  | open          |     |
| 5   | 9990413  | 04/03/2012 09:18:39 | 851755      | 07/19/2011 10:22:40 | 4                        | 1                                   | What is this PHP code in VB.net                   | I am looking for the vb.net equivalent of this... | php        | vb.net          | NA    | NA   | NA   | 04/15/2012 21:12:48 | too localized |     |
| 6   | 10421966 | 05/02/2012 21:25:01 | 603588      | 02/04/2011 18:05:34 | 334                      | 14                                  | Spring-Data mongodb querying multiple classes ... | With Spring-Data, you can use the @Document an... | mongodb    | spring-data     | NA    | NA   | NA   | NA                  | open          |     |
|     |          |                     |             |                     |                          |                                     |                                                   |                                                   |            |                 |       |      |      |                     |               |     |

    After training with this data I cannot inference using python nor rust If I use python, following error occurs:

    Traceback (most recent call last):
  File "src/python_alternative.py", line 39, in <module>
    df_tiny = df_tiny[
  File "/home/user/.local/lib/python3.8/site-packages/pandas/core/frame.py", line 3464, in getitem
    indexer = self.loc._get_listlike_indexer(key, axis=1)[1]
  File "/home/user/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1314, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis)
  File "/home/user/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1374, in _validate_read_indexer
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['Title', 'BodyMarkdown', 'OpenStatus'], dtype='object')] are in the [columns]"

    If I use rust:

    thread 'main' panicked at 'called Option::unwrap() on a None value', src/main.rs:97:55
    opened by zanaviska 2
Owner
Xavier Tao
Xavier Tao
GitHub
RobustVideoMatting and background composing in one model by using onnxruntime.

RVM_onnx_compose RobustVideoMatting and background composing in one model by using onnxruntime. Usage pip install -r requirements.txt python infer_cam

Quantum Liu 4 Apr 7, 2022
ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS. It currently supports four examples for you to quickly experience the power of ONNX Runtime Web.

Microsoft 58 Dec 18, 2022
Multiple types of NN model optimization environments. It is possible to directly access the host PC GUI and the camera to verify the operation. Intel iHD GPU (iGPU) support. NVIDIA GPU (dGPU) support.

mtomo Multiple types of NN model optimization environments. It is possible to directly access the host PC GUI and the camera to verify the operation.

Katsuya Hyodo 24 Mar 2, 2022
GrabGpu_py: a scripts for grab gpu when gpu is free

GrabGpu_py a scripts for grab gpu when gpu is free. WaitCondition: gpu_memory >

tianyuluan 3 Jun 18, 2022
The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

null 124 Dec 27, 2022
Run Effective Large Batch Contrastive Learning on Limited Memory GPU

Gradient Cache Gradient Cache is a simple technique for unlimitedly scaling contrastive learning batch far beyond GPU memory constraint. This means tr

Luyu Gao 198 Dec 29, 2022
I-BERT: Integer-only BERT Quantization

I-BERT: Integer-only BERT Quantization HuggingFace Implementation I-BERT is also available in the master branch of HuggingFace! Visit the following li

Sehoon Kim 139 Dec 27, 2022
Source code for NAACL 2021 paper "TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference"

TR-BERT Source code and dataset for "TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference". The code is based on huggaface's transformers.

THUNLP 37 Oct 30, 2022
LV-BERT: Exploiting Layer Variety for BERT (Findings of ACL 2021)

LV-BERT Introduction In this repo, we introduce LV-BERT by exploiting layer variety for BERT. For detailed description and experimental results, pleas

Weihao Yu 14 Aug 24, 2022
VD-BERT: A Unified Vision and Dialog Transformer with BERT

VD-BERT: A Unified Vision and Dialog Transformer with BERT PyTorch Code for the following paper at EMNLP2020: Title: VD-BERT: A Unified Vision and Dia

Salesforce 44 Nov 1, 2022