{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "     \n", "     \n", "     \n", "     \n", "     \n", "  \n", "[Home Page](Start_Here.ipynb)\n", " \n", " \n", "     \n", "     \n", "     \n", "     \n", "     \n", "   \n", "[1]\n", "[2](Getting_started_with_Deepstream_Pipeline.ipynb)\n", "[3](Introduction_to_Multi-DNN_pipeline.ipynb)\n", "[4](Multi-stream_pipeline.ipynb)\n", "[5](Multi-stream_Multi_DNN.ipynb)\n", "     \n", "     \n", "     \n", "     \n", "[Next Notebook](Getting_started_with_Deepstream_Pipeline.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to DeepStream\n", "\n", "In this notebook, you will be introduced to DeepStream ,it's workflow and the underlying principle upon which it works on. \n", "\n", "\n", "**Contents of this Notebook :**\n", "\n", "- [DeepStream](#DeepStream) \n", "- [Overview of the DeepStream SDK](#Overview-of-the-DeepStream-SDK)\n", "- [GStreamer Foundations](#GStreamer-Foundations-:)\n", " - [Elements](#Elements)\n", " - [Pipeline](#Pipeline)\n", " - [Pads](#Pads)\n", " - [Caps](#Caps)\n", " - [Buffers](#Buffers)\n", " - [Plugin based Architecture](#Plugin-based-Architecture)\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## DeepStream \n", "\n", "![Worlflow](images/ds-workflow.jpg)\n", "\n", "DeepStream simplifies building IVA applications by separating the application into components managed and built by the user and components managed by DeepStream.\n", "\n", "As developers, we build components to manage important business tasks like :\n", "- Selecting the kind and number of video streams we want to analyze\n", "- Choosing the type of analysis, we want to do on the video\n", "- Handling and interacting with the results of our analysis\n", " \n", "We don't need to build components to manage difficult tasks like :\n", "- Efficiently leverage the GPU for accelerated processing and inference\n", "- Efficiently process data from multiple video streams at once\n", "- Keeping track of metadata associated with each frame of video from multiple sources\n", "- Optimizing our pipeline for maximum data throughput\n", "- Optimizing our neural networks for high-speed inference\n", "\n", "These are mundane tasks that the **DeepStream SDK** manages for us. That lets us focus on the more important tasks related to the project's goal and impact. DeepStream lets us focus on the intelligence portion of the application.\n", "\n", "Here is an illustration on DeepStream Workflow that describes the tasks by DeepStream vs developer \n", "\n", "![Workflow_split](images/ds-workflow-split.jpeg)\n", "\n", "\n", "### Performance and Scalability \n", "\n", "\n", "### Performance \n", "\n", "To give an overview of performance improvement when using DeepStream SDK, we will scrutinize the DeepStream-app reference application included within the release package. The below illustration shows that the performance is doubled using DeepStream 3.0 when tested on T4 compared to the previous version P4, while it consumes the same amount of power and equal number of streams.The reference application includes a primary detector, three classifiers and a tracker.\n", "\n", "![Performance](images/ds-perf.png)\n", "\n", "\n", "### Scalability \n", "\n", "DeepStream provides scalability at different levels of the system hierarchy. \n", "\n", "For example, \n", "- The DeepStream SDK 3.0 supports processing a higher number of concurrent streams, in addition to utilizing multiple GPUs upon availability. \n", "- The DeepStream SDK 4.0 delivers a unified code base for all NVIDIA GPUs and quick integration with IoT services.\n", "\n", "Furthermore DeepStream-in-containers provide flexibility to the deployment phase as shown below:\n", "\n", "![Scalability](images/ds-scalability.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Overview of the DeepStream SDK\n", "\n", "The DeepStream SDK consists of a set of building blocks which bridge the gap between low level APIs (such as TensorRT, Video Codec SDK) and the user application. By utilizing the DeepStream SDK, you can accelerate development of the IVA applications by focusing on building core deep learning models instead of designing end-to-end applications from scratch.\n", "\n", "Below, you can see a schematic presentation of the DeepStream SDK in a series of potential\n", "applications.\n", "\n", "![SDK](images/ds-sdk.png)\n", "\n", "\n", "In addition, the DeepStream SDK extends these capabilities by providing several other hardware accelerated building blocks. This includes support for TensorRT 7 and CUDA 11. In addition, DeepStream applications can be deployed as a part of a larger multi-GPU cluster or a microservice in containers. This allows highly flexible system architectures and opens new application capabilities.\n", "\n", "Below, you can see a shortened list of new capabilities provided by DeepStream:\n", "- Allowing addition and removal of video stream dynamically and during the pipeline execution,in addition to frame rate and resolution adjustments\n", "- Extending the video processing capabilities by supporting custom layers, and user-defined parsing of detector outputs\n", "- Providing Support for 360-degree camera using GPU-accelerated dewarping libraries\n", "- Augmenting the meta-data with application-specific, user-defined insights\n", "- Providing pruned and efficient inference models\n", "- Getting detailed performance analysis with the NVIDIA Nsight system profiler tool.\n", "\n", "\n", "The DeepStream SDK is based on the **GStreamer multimedia framework** and provides a pipeline of GPU accelerated plugins as shown below. The SDK facilitates application implementation procedure by providing plugins for video inputs, video decoding, image preprocessing, TensorRT-based inference, object tracking and display. You can utilize these capabilities to assemble flexible, multi-stream video analytics applications.\n", "\n", "![Sample_pipeline](images/ds-sample-pipeline.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# GStreamer Foundations :\n", "\n", "The DeepStream SDK is based on the open source [GStreamer multimedia framework](https://gstreamer.freedesktop.org/). There are a few key concepts in GStreamer that we need to\n", "touch on before getting started. These include Pipelines,Elements, Pads, Buffers, and Caps. We will be\n", "describing them at a high level, but encourage those who are interested in the details to read the\n", "[GStreamer Basics](https://gstreamer.freedesktop.org/documentation/?gi-language=c) documentation to learn more.\n", "\n", "\n", "### Elements \n", "\n", "Elements are the core building block with which we make pipelines. Every process in-between the source (i.e. input of the pipeline, e.g. camera and video files) and sink elements (e.g. screen display) is passed through elements. Video decoding and encoding, neural network inference, and displaying text on top of video streams are examples of \"element\". DeepStream allows us to instantiate elements and weave them into pipelines.\n", "\n", "\n", "### Pipeline \n", "\n", "All elements in GStreamer must typically be contained inside a pipeline before they can be used, because it takes care of some clocking and messaging functions.\n", "\n", "A pipeline is a particular type of bin, which is the element used to contain other elements. Therefore all methods which apply to bins also apply to pipelines. We need to add the elements to the pipeline and are then linked ,this linking must be established following the data flow (this is, from source elements to sink elements). \n", "\n", "![pipeline](images/pipeline.png)\n", "\n", "### Pads\n", "\n", "Pads are the interfaces between elements. When data flows from element to another element in a pipeline, it flows from the sink pad of one element to the source pad of another. Note that each element might have zero, one or many source/sink elements.\n", "\n", "\n", "![pads](images/pads.png)\n", "\n", "### Caps\n", "\n", "Caps (or Capabilities), are the data types that a pad is permitted to utilize or emit. Because pads\n", "can allow multiple data types, sometimes the data flow is ambiguous. Pads are \"negotiated\" in\n", "order to explicitly define the type of data that can flow through the pad. Caps streamline this\n", "process and allow elements of our pipeline with ambiguous pads to negotiate the correct data flow\n", "process. Later in this course, we will use caps to pass certain video data types (NV12, RGB) to the\n", "downstream elements in the pipeline.\n", "\n", "### Buffers\n", "\n", "Buffers carry the data that will passed on through the pipeline. Buffers are timestamped, contain\n", "metadata such as how many elements are using it, flags, and pointers to objects in memory. When\n", "we write application code, we rely on accessing data attached to the buffer.\n", "\n", "### Plugin based Architecture\n", "\n", "DeepStream applications can be thought of as pipelines consisting of individual components\n", "(plugins). Each plugin represents a functional block like inference using TensorRT or multi-stream\n", "decode. Where applicable, plugins are accelerated using the underlying hardware to deliver\n", "maximum performance. DeepStream’s key value is in making deep learning for video easily\n", "accessible, to allow you to concentrate on quickly building and customizing efficient and scalable\n", "video analytics applications.\n", "\n", "The plugin architecture provides functionality such as video encode/decode, scaling, inferencing, and more. By connecting plugins into a pipeline, we can build complex applications. Because DeepStream is built on top of GStreamer, we can inspect plugins using `gst-inspect-1.0`.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#To make sure that right paths to the NVidia Libraries are added run this cell\n", "!rm ~/.cache/gstreamer-1.0/registry.x86_64.bin\n", "!export LD_LIBRARY_PATH=/opt/tensorrtserver/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs:$LD_LIBRARY_PATH" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Inspect the nvinfer plugin\n", "!gst-inspect-1.0 nvinfer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## Licensing\n", " \n", "This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0).).\n", " \n", "     \n", "     \n", "     \n", "     \n", "     \n", "   \n", "[1]\n", "[2](Getting_started_with_Deepstream_Pipeline.ipynb)\n", "[3](Introduction_to_Multi-DNN_pipeline.ipynb)\n", "[4](Multi-stream_pipeline.ipynb)\n", "[5](Multi-stream_Multi_DNN.ipynb)\n", "     \n", "     \n", "     \n", "     \n", "[Next Notebook](Getting_started_with_Deepstream_Pipeline.ipynb)\n", "\n", "     \n", "     \n", "     \n", "     \n", "     \n", "  \n", "[Home Page](Start_Here.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.2" } }, "nbformat": 4, "nbformat_minor": 4 }