{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&ensp;\n",
    "[Home Page](Start_Here.ipynb)\n",
    "    \n",
    "    \n",
    "[Previous Notebook](Introduction_to_Performance_analysis.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&ensp;\n",
    "[1](Introduction_to_Performance_analysis.ipynb)\n",
    "[2]\n",
    "[3](Performance_Analysis_using_NSight_systems_Continued.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[Next Notebook](Performance_Analysis_using_NSight_systems_Continued.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance analysis using Nsight systems\n",
    "\n",
    "\n",
    "In this notebook, we will use Nsight systems to profile and improve the DeepStream application's throughput performance.\n",
    "\n",
    "\n",
    "- [Using NSight Systems to generate a report and finding bottlenecks to solve](#Using-NSight-Systems-to-generate-a-report-and-finding-bottlenecks-to-solve) \n",
    "    - [Streammux parameters](#Streammux-parameters)\n",
    "    - [Batch size across cascaded networks](#Batch-size-across-cascaded-networks)\n",
    "    - [NVInfer](#NVInfer)\n",
    "    - [NVTracker](#NVTracker)\n",
    "- [Summary](#Summary)\n",
    "\n",
    "\n",
    "## NVIDIA Profiler\n",
    "\n",
    "### What is profiling\n",
    "Profiling is the first step in optimizing and tuning your application. Profiling an application would help us understand where most of the execution time is spent. You will gain an understanding of its performance characteristics and can easily identify parts of the code that present opportunities for improvement. Finding hotspots and bottlenecks in your application, can help you decide where to focus our optimization efforts.\n",
    "\n",
    "### NVIDIA Nsight Tools\n",
    "NVIDIA offers Nsight tools (Nsight Systems, Nsight Compute, Nsight Graphics), a collection of applications which enable developers to debug, profile the performance of CUDA, OpenACC, or OpenMP applications. \n",
    "\n",
    "Your profiling workflow will change to reflect the individual Nsight tools. Start with Nsight Systems to get a system-level overview of the workload and eliminate any system level bottlenecks, such as unnecessary thread synchronization or data movement, and improve the system level parallelism of your algorithms. Once you have done that, then proceed to Nsight Compute or Nsight Graphics to optimize the most significant CUDA kernels or graphics workloads, respectively. Periodically return to Nsight Systems to ensure that you remain focused on the largest bottleneck. Otherwise the bottleneck may have shifted and your kernel level optimizations may not achieve as high of an improvement as expected.\n",
    "\n",
    "- **Nsight Systems** analyze application algorithm system-wide\n",
    "- **Nsight Compute** debug and optimize CUDA kernels \n",
    "- **Nsight Graphics** debug and optimize graphic workloads\n",
    "\n",
    "<img src=\"images/Nsight Diagram.png\" width=\"80%\" height=\"80%\">\n",
    "*The data flows between the NVIDIA Nsight tools.*\n",
    "\n",
    "In this lab, we only focus on Nsight Systems to get the system-wide actionable insights to eliminate bottlenecks.\n",
    "\n",
    "### Introduction to Nsight Systems \n",
    "Nsight Systems tool offers system-wide performance analysis in order to visualize application’s algorithms, help identify optimization opportunities, and improve the performance of applications running on a system consisting of multiple CPUs and GPUs.\n",
    "\n",
    "#### Nsight Systems Timeline\n",
    "- CPU rows help locating CPU core's idle times. Each row shows how the process' threads utilize the CPU cores.\n",
    "<img src=\"images/cpu.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "- Thread rows shows a detailed view of each thread's activity including OS runtime libraries usage, CUDA API calls, NVTX time ranges and events (if integrated in the application).\n",
    "<img src=\"images/thread.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "- CUDA Workloads rows display Kernel and memory transfer activites. \n",
    "<img src=\"images/cuda.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "\n",
    "### Profiling using command line interface \n",
    "To profile your application, you can either use the Graphical User Interface(GUI) or Command Line Interface (CLI). During this lab, we will profile the mini application using CLI.\n",
    "\n",
    "The Nsight Systems command line interface is named `nsys`. Below is a typical command line invocation:\n",
    "\n",
    "`nsys profile -t cuda,nvtx --stats=true --force-overwrite true -o deepstream ./exe`\n",
    "\n",
    "where command switch options used for this lab are:\n",
    "- `profile` – start a profiling session\n",
    "- `-t`: Selects the APIs to be traced (nvtx and openacc in this example)\n",
    "- `--stats`: if true, it generates summary of statistics after the collection\n",
    "- `--force-overwrite`e: if true, it overwrites the existing generated report\n",
    "- `-o` – name for the intermediate result file, created at the end of the collection (.qdrep filename)\n",
    "\n",
    "**Note**: You do not need to memorize the profiler options. You can always run `nsys --help` or `nsys [specific command] --help` from the command line and use the necessary options or profiler arguments.\n",
    "For more info on Nsight profiler and NVTX, please see the __[Profiler documentation](https://docs.nvidia.com/nsight-systems/)__.\n",
    "\n",
    "### How to view the report\n",
    "When using CLI to profile the application, there are two ways to view the profiler's report. \n",
    "\n",
    "1) On the Terminal using `--stats` option: By using `--stats` switch option, profiling results are displayed on the console terminal after the profiling data is collected.\n",
    "\n",
    "<img src=\"images/laplas3.png\" width=\"100%\" height=\"100%\">\n",
    "\n",
    "2) NVIDIA Nsight System GUI: After the profiling session ends, a `*.qdrep` file will be created. This file can be loaded into Nsight Systems GUI using *File -> Open*. If you would like to view this on your local machine, this requires that the local system has CUDA toolkit installed of same version and the Nsight System GUI version should match the CLI version. More details on where to download CUDA toolkit can be found in the “Links and Resources” at the end of this page.\n",
    "\n",
    "To view the profiler report, simply open the file from the GUI (File > Open).\n",
    "\n",
    "<img src=\"images/nsight_open.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "### Using NVIDIA Tools Extension (NVTX) \n",
    "NVIDIA Tools Extension (NVTX) is a C-based Application Programming Interface (API) for annotating events, time ranges and resources in applications. NVTX brings the profiled application’s logic into the Profiler, making the Profiler’s displayed data easier to analyse and enables correlating the displayed data to profiled application’s actions.  \n",
    "\n",
    "DeepStream framework integrates the most stages of the pipeline with NVTX. During this lab, profile the application using Nsight Systems command line interface and collect the timeline. We will also be tracing NVTX APIs (already integrated into the DeepStream Pipeline). The NVTX tool is a powerful mechanism that allows users to manually instrument their application. NVIDIA Nsight Systems can then collect the information and present it on the timeline. It is particularly useful for tracing of CPU events and time ranges and greatly improves the timeline's readability. \n",
    "\n",
    "- An example of NVTX domain being shown by default for GSTNVInfer:\n",
    "\n",
    "<img src=\"images/nvtx_domain.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "Detailed NVTX documentation can be found under the __[CUDA Profiler user guide](https://docs.nvidia.com/cuda/profiler-users-guide/index.html#nvtx)__.\n",
    "\n",
    "### Steps to follow\n",
    "To obtain the best performance from GPU and utilize the hardware, one should follow the cyclical process (analyze, parallelize, optimize). \n",
    "\n",
    "- **Analyze**: In this step, you first identify the element in pipeline that includes most of the computation and most of the execution time is spent. From here, you find the hotspots, evaluate the bottlenecks and start investigating GPU acceleration.\n",
    "\n",
    "- **Optimize**:  Improve the performance by implementing optimization strategies step by step in an iterative process including: identify optimization opportunity, apply and test the optimization method, verify and repeat the process.\n",
    "\n",
    "Note: The above optimization is done incrementally after investigating the profiler output.\n",
    "\n",
    "We will follow the optimization cycle for porting and improving the code performance.\n",
    "\n",
    "### Nsight System Tips\n",
    "\n",
    "- The timeline view of Nsight after running the DeepStream application has many rows. Each row representing the execution by seperate thread which may be launched by GStreamer engine for different stages of piepline. It is beneficial to pin the rows of importnance as shown in image below\n",
    "<img src=\"images/pinning_row.png\" width=\"80%\" height=\"80%\">\n",
    "\n",
    "\n",
    "\n",
    "## Using NSight Systems to generate a report and finding bottlenecks to solve\n",
    "\n",
    "Let us now profile the pipeline that we optimized in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nsys profile --force-overwrite true -o ../source_code/reports/report1 python3 ../source_code/utils/deepstream-no-osd-queue.py --num-sources 3 --prof True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click Here](../source_code/reports/report1.qdrep) to download the report file.\n",
    "\n",
    "Opening the above in NSight Compute we can notice that Stream Muxer is the key bottleneck and the pipeline is waiting for the Muxer to finish. \n",
    "\n",
    "![batch_size](images/batch_size.PNG)\n",
    "\n",
    "With this information let us now look at different parameters of Streammux to optimize the pipeline in next section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Streammux parameters \n",
    "\n",
    "In previous section we noticed that Streammux cannot provide buffers quick enough for NVInfer to process, which creates a bottleneck in this case.\n",
    "\n",
    "We can fix this by tweaking control parameters that we can find in the [nvstreammux documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux.html). \n",
    "\n",
    "![img](images/nvstreamux-control.png)\n",
    "\n",
    "Out of this, the most obvious ones to change are the Width, Height, and Batch_size. \n",
    "\n",
    "The current configurations for streammux are as follows \n",
    "\n",
    "```python\n",
    "# Set Input Width , Height and Batch Size \n",
    "streammux.set_property('width', 1920)\n",
    "streammux.set_property('height', 1080)\n",
    "streammux.set_property('batch-size', 1)\n",
    "```\n",
    "\n",
    "We set the Width and Height to that of the original video resolution to prevent it from scaling.\n",
    "\n",
    "```python\n",
    "# Set Input Width , Height  \n",
    "streammux.set_property('width', 1280)\n",
    "streammux.set_property('height', 720)\n",
    "```\n",
    "\n",
    "Let us also set the batch size to that of the number of input sources to make sure streammux works at full capacity. \n",
    "\n",
    "```python\n",
    "# Set Batch Size\n",
    "streammux.set_property('batch-size', num_sources)\n",
    "```\n",
    "\n",
    "Let us now benchmark and profile our application. The code with the changes are bundled in python file [here](../source_code/utils/deepstream-no-osd-queue-streammux.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux.py --num-sources 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance is better than before and hence we can fit more number of streams. Let us run for 16 frames now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux.py --num-sources 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Appending this to the table from previous notebook. \n",
    "\n",
    "|Pipeline|Relative time|\n",
    "|---|----|\n",
    "|Default Pipeline|baseline|\n",
    "|With Queues|~3x|\n",
    "|Without OSD |~3.1x|\n",
    "|With Queues and without OSD|~3.15x|\n",
    "|Streammux - Optimization|~4x|\n",
    "\n",
    "\n",
    "Let us now profile this application to further optimize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nsys profile --force-overwrite true -o ../source_code/reports/report2 python3 ../source_code/utils/deepstream-no-osd-queue-streammux.py --num-sources 16 --prof True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click Here](../source_code/reports/report2.qdrep) to download the report file.\n",
    "\n",
    "Let us now open this in NSight systems and view it.\n",
    "\n",
    "![Batch size](images/batch_size_nvinfer.png)\n",
    "\n",
    "\n",
    "We can notice that for processing one batch of the buffer, it takes a long time for the NVInfer, which is even larger for secondary inference.  Let us follow the same cycle and optimize the NVInfer paramters in next section to increase the throughput.\n",
    "\n",
    "#### Batch size across cascaded networks\n",
    "\n",
    "\n",
    "We can refer to NVInfer control parameters [here](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html). \n",
    "\n",
    "Just like for Streammux, let us now set the Batch size for the Primary and secondary inference. The code is present here for analyzing [here](../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py)\n",
    "\n",
    "For Primary Inference, let us set the batch size to be equal to the number of sources, and let us try with three different values for Secondary inference. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py --num-sources 3 --sgie-batch-size 3 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py --num-sources 3 --sgie-batch-size 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py --num-sources 3 --sgie-batch-size 27"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py --num-sources 3 --sgie-batch-size 81"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarising the above results, we notice that when we increase the batch size, the FPS increases. This is obvious as we process buffers in a batch, and this makes the pipeline more efficient. Still, when we increase the Secondary inference batch size to a much higher value, we notice that the FPS decreases,this is because the Secondary inference is stalled and made to wait for more buffers to process in a batch of higher value. Hence when we choose our Secondary batch size, we need to tweak it as per the application. E.g., Setting it to the average number of cars per frame for this application would work as a reasonable estimate.\n",
    "\n",
    "Adding this to our previous table : \n",
    "\n",
    "|Pipeline|Relative time|\n",
    "|---|----|\n",
    "|Default Pipeline|baseline|\n",
    "|With Queues|~3x|\n",
    "|Without OSD |~3.1x|\n",
    "|With Queues and without OSD|~3.15x|\n",
    "|Streammux - Increase batch size|~4x|\n",
    "|NVInfer - Set Primary & Secondary batch size|~6x|\n",
    "\n",
    "Let us run this for a higher number of streams and profile it for further optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nsys profile --force-overwrite true -o ../source_code/reports/report3 python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer.py --num-sources 16 --sgie-batch-size 27 --prof True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click Here](../source_code/reports/report3.qdrep) to download the report file.\n",
    "\n",
    "Let us now download and open the NSight systems report to review it.\n",
    "\n",
    "![img](images/inference.png)\n",
    "\n",
    "We can notice that we have significantly reduced the time taken to process per batch by setting Batch sizes for both the primary and secondary inferences. \n",
    "\n",
    "#### NVInfer \n",
    "\n",
    "Let us now try to reduce it further using the [configuration parameters](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer-file-configuration-specifications) available to us in NVInfer \n",
    "\n",
    "Some of the optional parameters that we can change are as follows. \n",
    "\n",
    "```txt\n",
    "# Optional properties for detectors:\n",
    "#   cluster-mode(Default=Group Rectangles), interval(Primary mode only, Default=0)\n",
    "#   custom-lib-path,\n",
    "#   parse-bbox-func-name\n",
    "\n",
    "\n",
    "# Other optional properties:\n",
    "#   net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),\n",
    "#   model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,\n",
    "#   mean-file, gie-unique-id(Default=0), offsets, process-mode (Default=1 i.e. primary),\n",
    "#   custom-lib-path, network-mode(Default=0 i.e FP32)\n",
    "```\n",
    "\n",
    "One crucial inference parameter here is the `network-mode`. We can use the `INT8` Quantized network to make our inference faster. \n",
    "\n",
    "The `network-mode` parameters are as follows :\n",
    "\n",
    "```txt\n",
    "Integer \n",
    "0: FP32 \n",
    "1: INT8 \n",
    "2: FP16\n",
    "```\n",
    "\n",
    "#### NVTracker\n",
    "\n",
    "One more important parameter that we can use to make our inference faster is the `interval` parameter. By setting the interval parameter, we set the number of frames to be skipped for inference.\n",
    "\n",
    "We will then use NVTracker to keep track of our object's location. For NVTracker, we can set the low-level tracker as per our applications.\n",
    "\n",
    "![img](images/nvtracker.png)\n",
    "\n",
    "We will use the Lightweight IOU tracker for our application. We can set the same using the following line in our tracker config file.\n",
    "\n",
    "```\n",
    "ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_iou.so\n",
    "```\n",
    "\n",
    "Let us now run our application and benchmark it. The code is bundled in sigle python file present [here](../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer-nvtracker.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer-nvtracker.py --num-sources 3 --sgie-batch-size 27 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now append this to our previous table and compare with the previous results.\n",
    "\n",
    "|Pipeline|Relative time|\n",
    "|---|----|\n",
    "|Default Pipeline|baseline|\n",
    "|With Queues|~3x|\n",
    "|Without OSD |~3.1x|\n",
    "|With Queues and without OSD|~3.15x|\n",
    "|Streammux - Increase batch size|~4x|\n",
    "|NVInfer - Set Primary & Secondary batch size|~6x|\n",
    "|NVTracker + INT8| 6.2x |\n",
    "\n",
    "Let us now try to run multiple streams with this configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-no-osd-queue-streammux-nvinfer-nvtracker.py --num-sources 30 --sgie-batch-size 120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "We started with baseline version of pipline and increased the FPS by almost 6x. The optimizations discussed in the first and second notebooks are some parameters that are easy to tweak to get maximum performance for our DeepStream pipeline.\n",
    "\n",
    "\n",
    "In the upcoming notebook, let us take another example and try to improve the application step-by-step.\n",
    "\n",
    "## Licensing\n",
    "  \n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Previous Notebook](Introduction_to_Performance_analysis.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&ensp;\n",
    "[1](Introduction_to_Performance_analysis.ipynb)\n",
    "[2]\n",
    "[3](Performance_Analysis_using_NSight_systems_Continued.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "[Next Notebook](Performance_Analysis_using_NSight_systems_Continued.ipynb)\n",
    "\n",
    "\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&ensp;\n",
    "[Home Page](Start_Here.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
