AI pipelines for Nvidia Jetson Platform

Last update: Dec 23, 2022

Overview

Jetson Multicamera Pipelines

Easy-to-use realtime CV/AI pipelines for Nvidia Jetson Platform. This project:

Builds a typical multi-camera pipeline, i.e. N×(capture)->preprocess->batch->DNN-> <<your application logic here>> ->encode->file I/O + display. Uses gstreamer and deepstream under-the-hood.
Gives programatic acces to configure the pipeline in python via jetmulticam package.
Utilizes Nvidia HW accleration for minimal CPU usage. For example, you can perform object detection in real-time on 6 camera streams using as little as 16.5% CPU. See benchmarks below for details.

Demos

You can easily build your custom logic in python by accessing image data (via np.array), as well object detection results. See examples of person following below:

DashCamNet (DLA0) + PeopleNet (DLA1) on 3 camera streams.

We have 3 intependent cameras with ~270° field of view. Red Boxes correspond to DashCamNet detections, green ones to PeopleNet. The PeopleNet detections are used to perform person following logic.

demo_8_follow_me.mp4

PeopleNet (GPU) on 3 cameras streams.

Robot is operated in manual mode.

demo_9_security_nvidia.mp4

DashCamNet (GPU) on 3 camera streams.

Robot is operated in manual mode.

demo_1_fedex_driver.mp4

(All demos are performed in real-time onboard Nvidia Jetson Xavier NX)

Quickstart

Install:

git clone https://github.com/NVIDIA-AI-IOT/jetson-multicamera-pipelines.git
cd jetson-multicamera-pipelines
bash scripts/install-dependencies.sh
pip3 install .

Run example with your cameras:

source scripts/env_vars.sh 
cd examples
python3 example.py

Usage example

import time
from jetmulticam import CameraPipelineDNN
from jetmulticam.models import PeopleNet, DashCamNet

if __name__ == "__main__":

    pipeline = CameraPipelineDNN(
        cameras=[2, 5, 8],
        models=[
            PeopleNet.DLA1,
            DashCamNet.DLA0,
            # PeopleNet.GPU
        ],
        save_video=True,
        save_video_folder="/home/nx/logs/videos",
        display=True,
    )

    while pipeline.running():
        arr = pipeline.images[0] # np.array with shape (1080, 1920, 3), i.e. (1080p RGB image)
        dets = pipeline.detections[0] # Detections from the DNNs
        time.sleep(1/30)

Benchmarks

#	Scenario	# cams	CPU util. (jetmulticam)	CPU util. (nvargus-deamon)	CPU total	GPU %	EMC util %	Power draw	Inference Hardware
1.	1xGMSL -> 2xDNNs + disp + encode	1	5.3%	4%	9.3%	<3%	57%	8.5W	DLA0: PeopleNet DLA1: DashCamNet
2.	2xGMSL -> 2xDNNs + disp + encode	2	7.2%	7.7%	14.9%	<3%	62%	9.4W	DLA0: PeopleNet DLA1: DashCamNet
3.	3xGMSL -> 2xDNNs + disp + encode	3	9.2%	11.3%	20.5%	<3%	68%	10.1W	DLA0: PeopleNet DLA1: DashCamNet
4.	Same as #3 with CPU @ 1.9GHz	3	7.5%	9.0%		<3%	68%	10.4w	DLA0: PeopleNet DLA1: DashCamNet
5.	3xGMSL+2xV4L -> 2xDNNs + disp + encode	5	9.5%	11.3%	20.8%	<3%	45%	9.1W	DLA0: PeopleNet (interval=1) DLA1: DashCamNet (interval=1)
6.	3xGMSL+2xV4L -> 2xDNNs + disp + encode	5	8.3%	11.3%	19.6%	<3%	25%	7.5W	DLA0: PeopleNet (interval=6) DLA1: DashCamNet (interval=6)
7.	3xGMSL -> DNN + disp + encode	5	10.3%	12.8%	23.1%	99%	25%	15W	GPU: PeopleNet

Notes:

All figures are in 15W 6 core mode. To reproduce do: sudo nvpmodel -m 2; sudo jetson_clocks;
Test platform: Jetson Xavier NX and XNX Box running JetPack v4.5.1
The residual GPU usage in DLA-accelerated nets is caused by Sigmoid activations being computed with CUDA backend. Remaining layers are computed on DLA.
CPU usage will vary depending on factors such as camera resolution, framerate, available video formats and driver implementation.

Supported models / acceleratorss

pipeline = CameraPipelineDNN(
    cam_ids = [0, 1, 2]
    models=[
        models.PeopleNet.DLA0,
        models.PeopleNet.DLA1,
        models.PeopleNet.GPU,
        models.DashCamNet.DLA0,
        models.DashCamNet.DLA1,
        models.DashCamNet.GPU
        ]
    # ...
)

You might also like...

Nvidia Semantic Segmentation monorepo

Paper | YouTube | Cityscapes Score Pytorch implementation of our paper Hierarchical Multi-Scale Attention for Semantic Segmentation. Please refer to t

1.6k Jan 4, 2023

PyTorch implementation of the Quasi-Recurrent Neural Network - up to 16 times faster than NVIDIA's cuDNN LSTM

Quasi-Recurrent Neural Network (QRNN) for PyTorch Updated to support multi-GPU environments via DataParallel - see the the multigpu_dataparallel.py ex

1.3k Dec 28, 2022

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

175 Dec 29, 2022

NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.

NVIDIA Merlin NVIDIA Merlin is an open source library designed to accelerate recommender systems on NVIDIA’s GPUs. It enables data scientists, machine

419 Jan 3, 2023

General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends)

General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases. Backed by the Linux Foundation.

1k Jan 6, 2023

Simply enable or disable your Nvidia dGPU

EnvyControl (WIP) Simply enable or disable your Nvidia dGPU Usage First clone this repo and install envycontrol with sudo pip install . CLI Turn off y

292 Jan 3, 2023

Deploy optimized transformer based models on Nvidia Triton server

1.2k Jan 5, 2023

A Pythonic library for Nvidia Codec.

A Pythonic library for Nvidia Codec. The project is still in active development; expect breaking changes. Why another Python library for Nvidia Codec?

12 Dec 27, 2022

Deploy optimized transformer based models on Nvidia Triton server

🤗 Hugging Face Transformer submillisecond inference 🤯 and deployment on Nvidia Triton server Yes, you can perfom inference with transformer based mo

1.2k Jan 5, 2023

Comments

Error while running example/python3 example.py

After following the steps in readme, and trying to run the python3 example.py, error .Please support.

System: Jetson Xavier AGX, with Jetpack 4.5.1

self._p.send_event(Gst.Event.new_eos()) AttributeError: 'CameraPipelineDNN' object has no attribute '_p'

opened by rrmte 0

Got Error in example.py: Pipeline element is None!

@dusty-nv @tomek-l HI I am working on jetson Nano and I want to run detection with 4 usb camera streams simulteaneously. I downloaded deepstream version 5.1 for jetson from Nvidia deepstream website. After going from all steps as defined in readme. I run the script example.py but I get the following error. I have attached 4 usb webcams with my nano. Any help please..... thanks

Downloading pretrained weights for DashCamNet model. This may take a while...
Dowloaded pre-trained DashCamNet model to: /home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/models/dashcamnet/dashcamnet_pruned_v1.0
Pipeline element is None!
Traceback (most recent call last):
  File "example.py", line 16, in <module>
    display=False,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/multicamDNN.py", line 47, in __init__
    super().__init__(**kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/basepipeline.py", line 25, in __init__
    self._p = self._create_pipeline(**kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/multicamDNN.py", line 52, in _create_pipeline
    p = self._create_pipeline_fully_connected(self._c, self._m, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/multicamDNN.py", line 86, in _create_pipeline_fully_connected
    nvinfers = [_make_element_safe("nvinfer") for _ in model_list]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/multicamDNN.py", line 86, in <listcomp>
    nvinfers = [_make_element_safe("nvinfer") for _ in model_list]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/utils/gst.py", line 45, in _make_element_safe
    raise NameError(f"Could not create element {el_type}")
NameError: Could not create element nvinfer
Exception ignored in: <bound method BasePipeline.__del__ of <jetmulticam.pipelines.multicamDNN.CameraPipelineDNN object at 0x7f7cf107f0>>
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/basepipeline.py", line 38, in __del__
    self.stop()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/jetmulticam/pipelines/basepipeline.py", line 41, in stop
    self._p.send_event(Gst.Event.new_eos())
AttributeError: 'CameraPipelineDNN' object has no attribute '_p'

Here is the code examples.py

import time
from jetmulticam import CameraPipelineDNN
from jetmulticam.models import PeopleNet, DashCamNet

if __name__ == "__main__":

    pipeline = CameraPipelineDNN(
        cameras=[0, 1, 2],
        models=[
            #PeopleNet.DLA1,
            #DashCamNet.DLA0,
            PeopleNet.GPU
        ],
        save_video=False,
        save_video_folder="/home/nx/logs/videos",
        display=False,
    )

    while pipeline.running():

        # We can access the captured images here.
        # For example `pipeline.images` is a list of numpy arrays for each camera
        # In my case (RGB 1080p image), `arr` will be np.ndarray with shape: (1080, 1920, 3)
        arr = pipeline.images[0]
        print(type(arr))

        # Detections in each image are available here as a list of dicts:
        dets = pipeline.detections[0]
        print(dets)

        # Assuming there's one detection in `image[0]`, `dets` can look like so:
        # [{
        #     'class': 'person',
        #     'position': (361.31, 195.60, 891.96, 186.05), # bbox (left, width, top, height)
        #     'confidence': 0.92
        # }]

        # Main thread is not tied in any computation.
        # We can perform any operations on our images.
        avg = pipeline.images[0].mean()
        print(avg)
        time.sleep(1 / 30)

I also run examples/example-no-ai.py with editing my camera 4 ids like here: import time

from jetmulticam import CameraPipeline

if __name__ == "__main__":

    p = CameraPipeline([1, 3, 0, 2])

    print(p.running())

    for _ in range(100):
        arr = p.read(0)
        if arr is not None:
            print(arr.shape)
        time.sleep(0.1)

and the output I get is as follows

with Sudo

sudo python3 example-no-ai.py


[sudo] password for ubuntu: 
<__gi__.GstNvArgusCameraSrc object at 0x7f73d01990 (GstNvArgusCameraSrc at 0xe28d1a0)> <__gi__.Gstnvvconv object at 0x7f73d01af8 (Gstnvvconv at 0xdfed5d0)>
<__gi__.GstNvArgusCameraSrc object at 0x7f73d019d8 (GstNvArgusCameraSrc at 0xe28d860)> <__gi__.Gstnvvconv object at 0x7f73d01b88 (Gstnvvconv at 0xdfb6770)>
<__gi__.GstNvArgusCameraSrc object at 0x7f73d01a20 (GstNvArgusCameraSrc at 0xe294df0)> <__gi__.Gstnvvconv object at 0x7f73d01bd0 (Gstnvvconv at 0xdf02df0)>
<__gi__.GstNvArgusCameraSrc object at 0x7f73d01a68 (GstNvArgusCameraSrc at 0xe295420)> <__gi__.Gstnvvconv object at 0x7f73d01cf0 (Gstnvvconv at 0xdefd510)>
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:740 No cameras available
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:740 No cameras available
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:740 No cameras available
NvMMLiteOpen : Block : BlockType = 4 
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:740 No cameras available
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
True

I also verify my camera ids that either they are correct or not with v4l2-ctl --list-devices But they are fine. The output is :

Webcam C170 (usb-70090000.xusb-2.1.1.1):
	/dev/video1

Webcam C170 (usb-70090000.xusb-2.1.1.4):
	/dev/video3

Webcam C170 (usb-70090000.xusb-2.1.3):
	/dev/video0

Webcam C170 (usb-70090000.xusb-2.1.4):
	/dev/video2

opened by sharoseali 1

Adding a segmentation model to jetmulticam.models

Hello @tomek-l

This looks great, thanks for making it available!

I just managed to deploy a deeplabv3plus from mmsegmentation with TensorRT engine to a Jetson Xavier, which reaches up to 50FPS.

Next, I would like to run my segmentation pipeline on 6 e-CAM25_CUXVR CSI-2 cameras. What would be the steps to add my segmentation engine to jetmulticam.models? Could you help out with that? With some guidance, I could try to do it myself and make a PR :))

Thanks!

opened by eliabruni 0
can i get multi rtsp use this repo

nice work！ I have deploy my detection model in tensorrt and it could infer vector,i have 8 rtsp camera,i use opencv to get the 8 rtsp camera and make the 8 images to a batch (vector),then send to the model get the inference resluts.However i find the image have delay when use opencv method. could i use this repo to get the 8 stream rtsp sync. Thank you ,very much!
enhancement

opened by jzx-gooner 4