{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&ensp;\n",
    "[Home Page](Start_Here.ipynb)\n",
    "    \n",
    "    \n",
    "[Previous Notebook](Introduction_to_Performance_analysis.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&ensp;\n",
    "[1](Introduction_to_Performance_analysis.ipynb)\n",
    "[2](Performance_Analysis_using_NSight_systems.ipynb)\n",
    "[3]\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebooks, we have optimized the Multi-stream version of DeepStream Test App 2.In this notebook, we will work on a different pipeline to optimize it further.\n",
    "\n",
    "\n",
    "- [Case 2:COVID-19 Social Distancing Application.](#Case-2:-COVID-19-Social-Distancing-Application.)\n",
    "    - [Finding distance between 2 people](#Finding-distance-between-2-people)\n",
    "- [Solving the computational bottleneck](#Solving-the-computational-bottleneck)\n",
    "- [Jetson specific optimizations](#Jetson-specific-optimizations)\n",
    "- [Summary](#Summary)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Case 2: COVID-19 Social Distancing Application.\n",
    "\n",
    "\n",
    "The COVID-19 Social Distance application can be constructed from `deepstream-test-app1` by adding suitable Metadata processing to determine whether two people have come in close contact and violated the social distancing norms. \n",
    "\n",
    "![image](images/covid.png)\n",
    "\n",
    "##### Finding distance between 2 people\n",
    "\n",
    "As we view people from a camera, it is necessary to have a perspective correction as far people would look smaller and appear less distant in pixel space. So, it's important to approximate world-space distance between persons.\n",
    "\n",
    "\n",
    "We **assume** that avg human is of height 170 cm. We normalize bounding box height with this value. This normalized value is then further used to normalize pixel-space distance between objects.\n",
    "We define the distance between two persons given their BBOX centroid(x,y) and BBOX height (h) as follows.\n",
    "\n",
    "\n",
    "```python\n",
    "    # Pixel distance\n",
    "    dx = x2 - x1;\n",
    "    dy = y2 - y1;\n",
    "    # Pixel to real-world conversion using avg height as 170cm\n",
    "    lx = dx * 170 * (1/h1 + 1/h2) / 2;\n",
    "    ly = dy * 170 * (1/h1 + 1/h2) / 2;\n",
    "    l = sqrt(lx*lx + ly*ly);\n",
    "```\n",
    "\n",
    "Limitations: Above method tries to approximate 3d distance between persons from a 2d camera. As expected, this has limitations and works best only if the persons' body is perpendicular to the camera. These limitations can be removed if we use multiple cameras and camera calibration data to approximate persons' 3d location.\n",
    "\n",
    "Let us now start building our pipeline considering this assumtion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Import Required Libraries \n",
    "import sys\n",
    "sys.path.append('../source_code')\n",
    "\n",
    "import gi\n",
    "import time\n",
    "gi.require_version('Gst', '1.0')\n",
    "from gi.repository import GObject, Gst , GLib\n",
    "from common.bus_call import bus_call\n",
    "import pyds\n",
    "import math\n",
    "\n",
    "# Defining the Class Labels\n",
    "PGIE_CLASS_ID_VEHICLE = 0\n",
    "PGIE_CLASS_ID_BICYCLE = 1\n",
    "PGIE_CLASS_ID_PERSON = 2\n",
    "PGIE_CLASS_ID_ROADSIGN = 3\n",
    "\n",
    "# Defining the input output video file \n",
    "INPUT_VIDEO_NAME  = 'file:///opt/nvidia/deepstream/deepstream-5.0/python/source_code/dataset/wt.mp4'\n",
    "OUTPUT_VIDEO_NAME = \"../source_code/N3/ds_out.mp4\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def cb_newpad(decodebin, decoder_src_pad,data):\n",
    "    print(\"In cb_newpad\\n\")\n",
    "    caps=decoder_src_pad.get_current_caps()\n",
    "    gststruct=caps.get_structure(0)\n",
    "    gstname=gststruct.get_name()\n",
    "    source_bin=data\n",
    "    features=caps.get_features(0)\n",
    "\n",
    "    # Need to check if the pad created by the decodebin is for video and not\n",
    "    # audio.\n",
    "    print(\"gstname=\",gstname)\n",
    "    if(gstname.find(\"video\")!=-1):\n",
    "        # Link the decodebin pad only if decodebin has picked nvidia\n",
    "        # decoder plugin nvdec_*. We do this by checking if the pad caps contain\n",
    "        # NVMM memory features.\n",
    "        print(\"features=\",features)\n",
    "        if features.contains(\"memory:NVMM\"):\n",
    "            # Get the source bin ghost pad\n",
    "            bin_ghost_pad=source_bin.get_static_pad(\"src\")\n",
    "            if not bin_ghost_pad.set_target(decoder_src_pad):\n",
    "                sys.stderr.write(\"Failed to link decoder src pad to source bin ghost pad\\n\")\n",
    "        else:\n",
    "            sys.stderr.write(\" Error: Decodebin did not pick nvidia decoder plugin.\\n\")\n",
    "\n",
    "def decodebin_child_added(child_proxy,Object,name,user_data):\n",
    "    print(\"Decodebin child added:\", name, \"\\n\")\n",
    "    if(name.find(\"decodebin\") != -1):\n",
    "        Object.connect(\"child-added\",decodebin_child_added,user_data)   \n",
    "\n",
    "def create_source_bin(index,uri):\n",
    "    print(\"Creating source bin\")\n",
    "\n",
    "    # Create a source GstBin to abstract this bin's content from the rest of the\n",
    "    # pipeline\n",
    "    bin_name=\"source-bin-%02d\" %index\n",
    "    print(bin_name)\n",
    "    nbin=Gst.Bin.new(bin_name)\n",
    "    if not nbin:\n",
    "        sys.stderr.write(\" Unable to create source bin \\n\")\n",
    "\n",
    "    # Source element for reading from the uri.\n",
    "    # We will use decodebin and let it figure out the container format of the\n",
    "    # stream and the codec and plug the appropriate demux and decode plugins.\n",
    "    uri_decode_bin=Gst.ElementFactory.make(\"uridecodebin\", \"uri-decode-bin\")\n",
    "    if not uri_decode_bin:\n",
    "        sys.stderr.write(\" Unable to create uri decode bin \\n\")\n",
    "    # We set the input uri to the source element\n",
    "    uri_decode_bin.set_property(\"uri\",uri)\n",
    "    # Connect to the \"pad-added\" signal of the decodebin which generates a\n",
    "    # callback once a new pad for raw data has beed created by the decodebin\n",
    "    uri_decode_bin.connect(\"pad-added\",cb_newpad,nbin)\n",
    "    uri_decode_bin.connect(\"child-added\",decodebin_child_added,nbin)\n",
    "\n",
    "    # We need to create a ghost pad for the source bin which will act as a proxy\n",
    "    # for the video decoder src pad. The ghost pad will not have a target right\n",
    "    # now. Once the decode bin creates the video decoder and generates the\n",
    "    # cb_newpad callback, we will set the ghost pad target to the video decoder\n",
    "    # src pad.\n",
    "    Gst.Bin.add(nbin,uri_decode_bin)\n",
    "    bin_pad=nbin.add_pad(Gst.GhostPad.new_no_target(\"src\",Gst.PadDirection.SRC))\n",
    "    if not bin_pad:\n",
    "        sys.stderr.write(\" Failed to add ghost pad in source bin \\n\")\n",
    "        return None\n",
    "    return nbin\n",
    "\n",
    "## Make Element or Print Error and any other detail\n",
    "def make_elm_or_print_err(factoryname, name, printedname, detail=\"\"):\n",
    "  print(\"Creating\", printedname)\n",
    "  elm = Gst.ElementFactory.make(factoryname, name)\n",
    "  if not elm:\n",
    "     sys.stderr.write(\"Unable to create \" + printedname + \" \\n\")\n",
    "  if detail:\n",
    "     sys.stderr.write(detail)\n",
    "  return elm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Standard GStreamer initialization\n",
    "Gst.init(None)\n",
    "\n",
    "\n",
    "# Create gstreamer elements\n",
    "# Create Pipeline element that will form a connection of other elements\n",
    "print(\"Creating Pipeline \\n \")\n",
    "pipeline = Gst.Pipeline()\n",
    "\n",
    "if not pipeline:\n",
    "    sys.stderr.write(\" Unable to create Pipeline \\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "########### Create Elements required for the Pipeline ########### \n",
    "\n",
    "# Create nvstreammux instance to form batches from one or more sources.\n",
    "streammux = make_elm_or_print_err(\"nvstreammux\", \"Stream-muxer\",\"Stream-muxer\") \n",
    "\n",
    "pipeline.add(streammux)\n",
    "\n",
    "num_sources = 1 \n",
    "for i in range(num_sources):\n",
    "    print(\"Creating source_bin \",i,\" \\n \")\n",
    "    uri_name=INPUT_VIDEO_NAME\n",
    "    if uri_name.find(\"rtsp://\") == 0 :\n",
    "        is_live = True\n",
    "    source_bin=create_source_bin(i, uri_name)\n",
    "    if not source_bin:\n",
    "        sys.stderr.write(\"Unable to create source bin \\n\")\n",
    "    pipeline.add(source_bin)\n",
    "    padname=\"sink_%u\" %i\n",
    "    sinkpad = streammux.get_request_pad(padname) \n",
    "    if not sinkpad:\n",
    "        sys.stderr.write(\"Unable to create sink pad bin \\n\")\n",
    "    srcpad = source_bin.get_static_pad(\"src\")\n",
    "    if not srcpad:\n",
    "        sys.stderr.write(\"Unable to create src pad bin \\n\")\n",
    "    srcpad.link(sinkpad)\n",
    "\n",
    "\n",
    "# Use nvinfer to run inferencing on decoder's output, behaviour of inferencing is set through config file\n",
    "pgie = make_elm_or_print_err(\"nvinfer\", \"primary-inference\" ,\"pgie\")\n",
    "# Use convertor to convert from NV12 to RGBA as required by nvosd\n",
    "nvvidconv = make_elm_or_print_err(\"nvvideoconvert\", \"convertor\",\"nvvidconv\")\n",
    "# Create OSD to draw on the converted RGBA buffer\n",
    "nvosd = make_elm_or_print_err(\"nvdsosd\", \"onscreendisplay\",\"nvosd\")\n",
    "# Finally encode and save the osd output\n",
    "queue = make_elm_or_print_err(\"queue\", \"queue\", \"Queue\")\n",
    "# Use convertor to convert from NV12 to RGBA as required by nvosd\n",
    "nvvidconv2 = make_elm_or_print_err(\"nvvideoconvert\", \"convertor2\",\"nvvidconv2\")\n",
    "# Place an encoder instead of OSD to save as video file\n",
    "encoder = make_elm_or_print_err(\"avenc_mpeg4\", \"encoder\", \"Encoder\")\n",
    "# Parse output from Encoder \n",
    "codeparser = make_elm_or_print_err(\"mpeg4videoparse\", \"mpeg4-parser\", 'Code Parser')\n",
    "# Create a container\n",
    "container = make_elm_or_print_err(\"qtmux\", \"qtmux\", \"Container\")\n",
    "# Create Sink for storing the output \n",
    "sink = make_elm_or_print_err(\"filesink\", \"filesink\", \"Sink\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "############ Set properties for the Elements ############\n",
    "print(\"Playing file \",INPUT_VIDEO_NAME)\n",
    "# Set Input Width , Height and Batch Size \n",
    "streammux.set_property('width', 1920)\n",
    "streammux.set_property('height', 1080)\n",
    "streammux.set_property('batch-size', 1)\n",
    "# Timeout in microseconds to wait after the first buffer is available \n",
    "# to push the batch even if a complete batch is not formed.\n",
    "streammux.set_property('batched-push-timeout', 4000000)\n",
    "# Set Congifuration file for nvinfer \n",
    "pgie.set_property('config-file-path', \"../source_code/N3/dstest1_pgie_config.txt\")\n",
    "# Set Encoder bitrate for output video\n",
    "encoder.set_property(\"bitrate\", 2000000)\n",
    "# Set Output file name and disable sync and async\n",
    "sink.set_property(\"location\", OUTPUT_VIDEO_NAME)\n",
    "sink.set_property(\"sync\", 0)\n",
    "sink.set_property(\"async\", 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "########## Add and Link ELements in the Pipeline ########## \n",
    "\n",
    "print(\"Adding elements to Pipeline \\n\")\n",
    "\n",
    "pipeline.add(pgie)\n",
    "pipeline.add(nvvidconv)\n",
    "pipeline.add(nvosd)\n",
    "pipeline.add(queue)\n",
    "pipeline.add(nvvidconv2)\n",
    "pipeline.add(encoder)\n",
    "pipeline.add(codeparser)\n",
    "pipeline.add(container)\n",
    "pipeline.add(sink)\n",
    "\n",
    "\n",
    "# Linking elements to the Pipeline\n",
    "print(\"Linking elements to Pipeline \\n\")\n",
    "\n",
    "streammux.link(pgie)\n",
    "pgie.link(nvvidconv)\n",
    "nvvidconv.link(nvosd)\n",
    "nvosd.link(queue)\n",
    "queue.link(nvvidconv2)\n",
    "nvvidconv2.link(encoder)\n",
    "encoder.link(codeparser)\n",
    "codeparser.link(container)\n",
    "container.link(sink)\n",
    "\n",
    "# create an event loop and feed gstreamer bus mesages to it\n",
    "loop = GLib.MainLoop()\n",
    "bus = pipeline.get_bus()\n",
    "bus.add_signal_watch()\n",
    "bus.connect (\"message\", bus_call, loop)\n",
    "\n",
    "print(\"Created event loop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "############# Define Computation required for our pipeline #################\n",
    "\n",
    "def compute_dist(p1, p2):\n",
    "    \n",
    "    (x1, y1, h1) = p1;\n",
    "    (x2, y2, h2) = p2;\n",
    "    dx = x2 - x1;\n",
    "    dy = y2 - y1;\n",
    "\n",
    "    lx = dx * 170 * (1/h1 + 1/h2) / 2;\n",
    "    ly = dy * 170 * (1/h1 + 1/h2) / 2;\n",
    "\n",
    "    l = math.sqrt(lx*lx + ly*ly);\n",
    "    return l\n",
    "\n",
    "\n",
    "def get_min_distances(centroids):\n",
    "    mini=[]\n",
    "    for i in range(len(centroids)):\n",
    "        distance=[]\n",
    "        for j in range(len(centroids)):\n",
    "            distance.append(compute_dist(centroids[i],centroids[j]))\n",
    "        distance[i]=10000000\n",
    "        mini.append(min(distance))\n",
    "    return mini\n",
    "\n",
    "\n",
    "def visualize(objs):\n",
    "    violations = 0 \n",
    "    dist_threshold = 160  # Distance in cms\n",
    "    for obj in objs:\n",
    "        min_dist = obj[\"min_dist\"]\n",
    "        redness_factor = 1.5\n",
    "        r_channel = max(255 * (dist_threshold - min_dist) / dist_threshold, 0) * redness_factor\n",
    "        g_channel = 255 - r_channel\n",
    "        b_channel = 0\n",
    "        obj_meta = obj[\"obj_meta\"]\n",
    "        obj_meta.rect_params.border_color.red = r_channel\n",
    "        obj_meta.rect_params.border_color.green = g_channel\n",
    "        obj_meta.rect_params.border_color.blue = b_channel\n",
    "        obj[\"violated\"] = (min_dist < dist_threshold)\n",
    "        violations = violations + int(min_dist < dist_threshold)\n",
    "    return violations\n",
    "\n",
    "def get_centroid(rect):\n",
    "\n",
    "    xmin = rect.left\n",
    "    xmax = rect.left + rect.width\n",
    "    ymin = rect.top\n",
    "    ymax = rect.top + rect.height\n",
    "    centroid_x = (xmin + xmax) / 2\n",
    "    centroid_y = (ymin + ymax) / 2\n",
    "\n",
    "    return (centroid_x, centroid_y, rect.height)\n",
    "\n",
    "def compute_min_distances_cpp(objs):\n",
    "    centroids = [o[\"centroid\"] for o in objs]    \n",
    "    min_distances = get_min_distances(centroids)\n",
    "    for o in range(len(objs)):\n",
    "        objs[o][\"min_dist\"] = min_distances[o]\n",
    "\n",
    "\n",
    "\n",
    "############## Working with the Metadata ################\n",
    "\n",
    "def osd_sink_pad_buffer_probe(pad,info,u_data):\n",
    "    #Intiallizing object counter with 0.\n",
    "    obj_counter = {\n",
    "        PGIE_CLASS_ID_VEHICLE:0,\n",
    "        PGIE_CLASS_ID_PERSON:0,\n",
    "        PGIE_CLASS_ID_BICYCLE:0,\n",
    "        PGIE_CLASS_ID_ROADSIGN:0\n",
    "    }\n",
    "    # Set frame_number & rectangles to draw as 0 \n",
    "    frame_number=0\n",
    "    num_rects=0\n",
    "    \n",
    "    gst_buffer = info.get_buffer()\n",
    "    if not gst_buffer:\n",
    "        print(\"Unable to get GstBuffer \")\n",
    "        return\n",
    "\n",
    "    # Retrieve batch metadata from the gst_buffer\n",
    "    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the\n",
    "    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)\n",
    "    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))\n",
    "    l_frame = batch_meta.frame_meta_list\n",
    "    while l_frame is not None:\n",
    "        try:\n",
    "            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta\n",
    "            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)\n",
    "        except StopIteration:\n",
    "            break\n",
    "        objects=[]\n",
    "        # Get frame number , number of rectables to draw and object metadata\n",
    "        frame_number=frame_meta.frame_num\n",
    "        num_rects = frame_meta.num_obj_meta\n",
    "        l_obj=frame_meta.obj_meta_list\n",
    "        \n",
    "        while l_obj is not None:\n",
    "            try:\n",
    "                # Casting l_obj.data to pyds.NvDsObjectMeta\n",
    "                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)\n",
    "            except StopIteration:\n",
    "                break\n",
    "            # Increment Object class by 1 and Set Box border to Red color     \n",
    "            obj_counter[obj_meta.class_id] +=1\n",
    "            obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.0)\n",
    "            \n",
    "            if (obj_meta.class_id == PGIE_CLASS_ID_PERSON):\n",
    "                obj = {}\n",
    "                obj[\"tracker_id\"] = obj_meta.object_id\n",
    "                obj[\"unique_id\"] = obj_meta.unique_component_id\n",
    "                obj[\"centroid\"] = get_centroid(obj_meta.rect_params)\n",
    "                obj[\"obj_meta\"] = obj_meta\n",
    "                objects.append(obj)\n",
    "            else:\n",
    "                obj_meta.rect_params.border_width = 0\n",
    "\n",
    "            try: \n",
    "                l_obj=l_obj.next\n",
    "            except StopIteration:\n",
    "                break\n",
    "        # Get the number of violations \n",
    "        compute_min_distances_cpp(objects)\n",
    "        violations = visualize(objects)\n",
    "        ################## Setting Metadata Display configruation ############### \n",
    "        # Acquiring a display meta object.\n",
    "        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)\n",
    "        display_meta.num_labels = 1\n",
    "        py_nvosd_text_params = display_meta.text_params[0]\n",
    "        # Setting display text to be shown on screen\n",
    "        py_nvosd_text_params.display_text = \"Frame Number={} Number of Objects={} Vehicle_count={} Person_count={} Violations={}\".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON],violations)\n",
    "        # Now set the offsets where the string should appear\n",
    "        py_nvosd_text_params.x_offset = 10\n",
    "        py_nvosd_text_params.y_offset = 12\n",
    "        # Font , font-color and font-size\n",
    "        py_nvosd_text_params.font_params.font_name = \"Serif\"\n",
    "        py_nvosd_text_params.font_params.font_size = 10\n",
    "        # Set(red, green, blue, alpha); Set to White\n",
    "        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)\n",
    "        # Text background color\n",
    "        py_nvosd_text_params.set_bg_clr = 1\n",
    "        # Set(red, green, blue, alpha); set to Black\n",
    "        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)\n",
    "        # Using pyds.get_string() to get display_text as string to print in notebook\n",
    "        print(pyds.get_string(py_nvosd_text_params.display_text))\n",
    "        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)\n",
    "        \n",
    "        ############################################################################\n",
    "\n",
    "        try:\n",
    "            l_frame=l_frame.next\n",
    "        except StopIteration:\n",
    "            break\n",
    "    return Gst.PadProbeReturn.OK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Lets add probe to get informed of the meta data generated, we add probe to the sink pad  \n",
    "# of the osd element, since by that time, the buffer would have had got all the metadata.\n",
    "\n",
    "osdsinkpad = nvosd.get_static_pad(\"sink\")\n",
    "if not osdsinkpad:\n",
    "    sys.stderr.write(\" Unable to get sink pad of nvosd \\n\")\n",
    "    \n",
    "osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)\n",
    "\n",
    "print(\"Probe added\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# start play back and listen to events\n",
    "print(\"Starting pipeline \\n\")\n",
    "start_time = time.time()\n",
    "pipeline.set_state(Gst.State.PLAYING)\n",
    "try:\n",
    "    loop.run()\n",
    "except:\n",
    "    pass\n",
    "# cleanup\n",
    "pipeline.set_state(Gst.State.NULL)\n",
    "print(\"--- %s seconds ---\" % (time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Convert video profile to be compatible with Jupyter notebook\n",
    "!ffmpeg -loglevel panic -y -an -i ../source_code/N3/ds_out.mp4 -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -level 3 ../source_code/N3/output.mp4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Display the Output\n",
    "from IPython.display import HTML\n",
    "HTML(\"\"\"\n",
    " <video width=\"640\" height=\"480\" controls>\n",
    " <source src=\"../source_code/N3/output.mp4\"\n",
    " </video>\n",
    "\"\"\".format())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now run multiple streams concurrently and benchmark the performance we obtain from this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-covid-19.py --num-sources 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!nsys profile --force-overwrite true -o ../source_code/reports/report4 python3 ../source_code/utils/deepstream-covid-19.py --num-sources 32 --prof True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click Here](../source_code/reports/report4.qdrep) to download the report file.\n",
    "\n",
    "![report4](images/report4.PNG)\n",
    "\n",
    "#### Solving the computational bottleneck\n",
    "\n",
    "Here we can notice that the bottleneck is now shifted to the NV Decode. For Hardware capable of decoding multiple inputs concurrently, such as the A100, NV Decode would not be a bottleneck. We can see that the `queue3`, which works on the computation of the distance between people, would become the bottleneck as it takes a long time to execute. (In this case ~48 ms),in such cases, we can reduce the computation time to C++ or CUDA to make the computation faster. Here is one such example where we use C++ to run the distancing algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! cd ../source_code/distancing && cmake . && make "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python3 ../source_code/utils/deepstream-covid-19-cpp.py --num-sources 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!nsys profile --force-overwrite true -o ../source_code/reports/report5 python3 ../source_code/utils/deepstream-covid-19-cpp.py --num-sources 32 --prof True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Click Here](../source_code/reports/report5.qdrep) to download the report file.\n",
    "\n",
    "![report5](images/report5.PNG)\n",
    "\n",
    "Here we can notice that there has been a reduction in computation time when we shift it from Python to C++ (~48 ms to ~32ms), this can further be optimized in this case and even can be extended to CUDA if necessary.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jetson specific optimizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Power \n",
    "\n",
    "For Jetson devices, it is recommended to be set the mode to use Max power. \n",
    "\n",
    "The Max power mode can be enabled using the following command \n",
    "```bash\n",
    "$ sudo nvpmodel -m 0\n",
    "```\n",
    "The GPU clocks can be stepped to maximum using the following command \n",
    "\n",
    "```bash\n",
    "$ sudo jetson_clocks\n",
    "```\n",
    "\n",
    "For information about supported power modes, see \"Supported Modes and Power Efficiency\" in the power management topics of NVIDIA Tegra Linux Driver Package Development Guide, e.g., \"Power Management for Jetson AGX Xavier Devices.\"\n",
    "\n",
    "\n",
    "For Jetson devices, the details regarding the memory and compute usage can be queried using the following command.\n",
    "\n",
    "```bash\n",
    "$ tegrastats\n",
    "```\n",
    "\n",
    "This command cannot be run inside the Docker container and needs to be run in a separate terminal.\n",
    "\n",
    "#### Deep Learning Accelerators\n",
    "\n",
    "Jetson AGX Xavier and Jetson NX supports 2 DLA engines. DeepStream does support inferencing using GPU and DLAs in parallel. You can do this in separate processes or a single process. You will need three separate sets of configs, configured to run on GPU, DLA0, and DLA1.\n",
    "\n",
    "More details on this can be found [here](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Performance.html#running-applications-using-dla) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we learnt some techniques to optimize the Deepstream application and deal with computational bottlenecks that a may user can encounter and discussed one such way of solving them.\n",
    "\n",
    "## Licensing\n",
    "  \n",
    "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Previous Notebook](Introduction_to_Performance_analysis.ipynb)\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&ensp;\n",
    "[1](Introduction_to_Performance_analysis.ipynb)\n",
    "[2](Performance_Analysis_using_NSight_systems.ipynb)\n",
    "[3]\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;\n",
    "&emsp;&ensp;\n",
    "[Home Page](Start_Here.ipynb)\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
