{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "     \n", "     \n", "     \n", "     \n", "     \n", "   \n", "[Home Page](../START_HERE.ipynb)\n", "\n", "[Previous Notebook](03-Cudf_Exercise.ipynb)\n", "     \n", "     \n", "     \n", "     \n", "[1](01-Intro_to_cuDF.ipynb)\n", "[2](02-Intro_to_cuDF_UDFs.ipynb)\n", "[3](03-Cudf_Exercise.ipynb)\n", "[4]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Applying CuDF: Workbook\n", "\n", "Welcome to fourth cuDF tutorial notebook! This is a practical example that utilizes cuDF and cuPy, geared primarily for new users. The purpose of this tutorial is to introduce new users to a data science processing pipeline using RAPIDS on real life datasets. We will be working on a data science problem: US Accidents Prediction. This is a countrywide car accident dataset, which covers 49 states of the USA. The accident data are collected from February 2016 to June 2020, using two APIs that provide streaming traffic incident (or event) data. These APIs broadcast traffic data captured by a variety of entities, such as the US and state departments of transportation, law enforcement agencies, traffic cameras, and traffic sensors within the road-networks. Currently, there are about 3.5 million accident records in this dataset. \n", "\n", "\n", "## What should I do?\n", "\n", "Given below is a complete data science preprocessing pipeline for the dataset using Pandas and Numpy libraries. Using the methods and techniques from the previous notebooks, you have to convert this pipeline to a a RAPIDS implementation, using CuDF and CuPy. Don't forget to time your code cells and compare the performance with this original code, to understand why we are using RAPIDS. If you get stuck in the middle, feel free to refer to this sample solution. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Here is the list of exercises in the lab where you need to modify code:\n", "- Exercise 1
Loading the dataset from a csv file and store in a CuDF dataframe.\n", "- Exercise 2
Creating kernel functions to run the given function optimally on a GPU.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first step is downloading the dataset and putting it in the data directory, for using in this tutorial.\n", "Download the dataset here, and place it in (host/data) folder. Now we will import the necessary libraries." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import cudf\n", "import numpy as np\n", "import cupy as cp\n", "import math\n", "np.random.seed(12)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "First we need to load the dataset from the csv into CuDF dataframes, for the preprocessing steps. If you need help, refer to the Getting Data In and Out module from this [notebook](01-Intro_to_cuDF.ipynb/)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Modify the code in this cell\n", "\n", "# Use cudf to read csv\n", "%time df = \n", "print(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "First we will analyse the data and observe patterns that can help us process the data better for feeding to the machine learning algorithms in the future. By using the describe, we will generate the descriptive statistics for all the columns. Descriptive statistics include those that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TMCSeverityStart_LatStart_LngEnd_LatEnd_LngDistance(mi)NumberTemperature(F)Wind_Chill(F)Humidity(%)Pressure(in)Visibility(mi)Wind_Speed(mph)Precipitation(in)
count2.478818e+063.513617e+063.513617e+063.513617e+061.034799e+061.034799e+063.513617e+061.250753e+063.447885e+061.645368e+063.443930e+063.457735e+063.437761e+063.059008e+061.487743e+06
mean2.080226e+022.339929e+003.654194e+01-9.579151e+013.755758e+01-1.004560e+022.816170e-015.975383e+036.193512e+015.355730e+016.511427e+012.974463e+019.122644e+008.219025e+001.598300e-02
std2.076627e+015.521930e-014.883520e+001.736877e+014.861215e+001.852879e+011.550134e+001.496624e+041.862106e+012.377334e+012.275558e+018.319760e-012.885879e+005.262847e+001.928260e-01
min2.000000e+021.000000e+002.455527e+01-1.246238e+022.457011e+01-1.244978e+020.000000e+000.000000e+00-8.900000e+01-8.900000e+011.000000e+000.000000e+000.000000e+000.000000e+000.000000e+00
25%2.010000e+022.000000e+003.363784e+01-1.174418e+023.399477e+01-1.183440e+020.000000e+008.640000e+025.000000e+013.570000e+014.800000e+012.973000e+011.000000e+015.000000e+000.000000e+00
50%2.010000e+022.000000e+003.591687e+01-9.102601e+013.779736e+01-9.703438e+010.000000e+002.798000e+036.400000e+015.700000e+016.700000e+012.995000e+011.000000e+017.000000e+000.000000e+00
75%2.010000e+023.000000e+004.032217e+01-8.093299e+014.105139e+01-8.210168e+011.000000e-027.098000e+037.590000e+017.200000e+018.400000e+013.009000e+011.000000e+011.150000e+010.000000e+00
max4.060000e+024.000000e+004.900220e+01-6.711317e+014.907500e+01-6.710924e+013.336300e+029.999997e+061.706000e+021.150000e+021.000000e+025.774000e+011.400000e+029.840000e+022.500000e+01
\n", "
" ], "text/plain": [ " TMC Severity Start_Lat Start_Lng End_Lat \\\n", "count 2.478818e+06 3.513617e+06 3.513617e+06 3.513617e+06 1.034799e+06 \n", "mean 2.080226e+02 2.339929e+00 3.654194e+01 -9.579151e+01 3.755758e+01 \n", "std 2.076627e+01 5.521930e-01 4.883520e+00 1.736877e+01 4.861215e+00 \n", "min 2.000000e+02 1.000000e+00 2.455527e+01 -1.246238e+02 2.457011e+01 \n", "25% 2.010000e+02 2.000000e+00 3.363784e+01 -1.174418e+02 3.399477e+01 \n", "50% 2.010000e+02 2.000000e+00 3.591687e+01 -9.102601e+01 3.779736e+01 \n", "75% 2.010000e+02 3.000000e+00 4.032217e+01 -8.093299e+01 4.105139e+01 \n", "max 4.060000e+02 4.000000e+00 4.900220e+01 -6.711317e+01 4.907500e+01 \n", "\n", " End_Lng Distance(mi) Number Temperature(F) \\\n", "count 1.034799e+06 3.513617e+06 1.250753e+06 3.447885e+06 \n", "mean -1.004560e+02 2.816170e-01 5.975383e+03 6.193512e+01 \n", "std 1.852879e+01 1.550134e+00 1.496624e+04 1.862106e+01 \n", "min -1.244978e+02 0.000000e+00 0.000000e+00 -8.900000e+01 \n", "25% -1.183440e+02 0.000000e+00 8.640000e+02 5.000000e+01 \n", "50% -9.703438e+01 0.000000e+00 2.798000e+03 6.400000e+01 \n", "75% -8.210168e+01 1.000000e-02 7.098000e+03 7.590000e+01 \n", "max -6.710924e+01 3.336300e+02 9.999997e+06 1.706000e+02 \n", "\n", " Wind_Chill(F) Humidity(%) Pressure(in) Visibility(mi) \\\n", "count 1.645368e+06 3.443930e+06 3.457735e+06 3.437761e+06 \n", "mean 5.355730e+01 6.511427e+01 2.974463e+01 9.122644e+00 \n", "std 2.377334e+01 2.275558e+01 8.319760e-01 2.885879e+00 \n", "min -8.900000e+01 1.000000e+00 0.000000e+00 0.000000e+00 \n", "25% 3.570000e+01 4.800000e+01 2.973000e+01 1.000000e+01 \n", "50% 5.700000e+01 6.700000e+01 2.995000e+01 1.000000e+01 \n", "75% 7.200000e+01 8.400000e+01 3.009000e+01 1.000000e+01 \n", "max 1.150000e+02 1.000000e+02 5.774000e+01 1.400000e+02 \n", "\n", " Wind_Speed(mph) Precipitation(in) \n", "count 3.059008e+06 1.487743e+06 \n", "mean 8.219025e+00 1.598300e-02 \n", "std 5.262847e+00 1.928260e-01 \n", "min 0.000000e+00 0.000000e+00 \n", "25% 5.000000e+00 0.000000e+00 \n", "50% 7.000000e+00 0.000000e+00 \n", "75% 1.150000e+01 0.000000e+00 \n", "max 9.840000e+02 2.500000e+01 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will check the size of the dataset that is to be processed using the len function." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3513617" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will notice that the dataset has 3513616 rows and takes quite a lot of time to read from the file. As we go ahead with the preprocessing, computations will require more time to execute, and that's where the RAPIDS comes to the rescue!\n", "\n", "Now we use the info function to check the datatype of all the columns in the dataset." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 3513617 entries, 0 to 3513616\n", "Data columns (total 49 columns):\n", " # Column Dtype\n", "--- ------ -----\n", " 0 ID object\n", " 1 Source object\n", " 2 TMC float64\n", " 3 Severity int64\n", " 4 Start_Time object\n", " 5 End_Time object\n", " 6 Start_Lat float64\n", " 7 Start_Lng float64\n", " 8 End_Lat float64\n", " 9 End_Lng float64\n", " 10 Distance(mi) float64\n", " 11 Description object\n", " 12 Number float64\n", " 13 Street object\n", " 14 Side object\n", " 15 City object\n", " 16 County object\n", " 17 State object\n", " 18 Zipcode object\n", " 19 Country object\n", " 20 Timezone object\n", " 21 Airport_Code object\n", " 22 Weather_Timestamp object\n", " 23 Temperature(F) float64\n", " 24 Wind_Chill(F) float64\n", " 25 Humidity(%) float64\n", " 26 Pressure(in) float64\n", " 27 Visibility(mi) float64\n", " 28 Wind_Direction object\n", " 29 Wind_Speed(mph) float64\n", " 30 Precipitation(in) float64\n", " 31 Weather_Condition object\n", " 32 Amenity bool\n", " 33 Bump bool\n", " 34 Crossing bool\n", " 35 Give_Way bool\n", " 36 Junction bool\n", " 37 No_Exit bool\n", " 38 Railway bool\n", " 39 Roundabout bool\n", " 40 Station bool\n", " 41 Stop bool\n", " 42 Traffic_Calming bool\n", " 43 Traffic_Signal bool\n", " 44 Turning_Loop bool\n", " 45 Sunrise_Sunset object\n", " 46 Civil_Twilight object\n", " 47 Nautical_Twilight object\n", " 48 Astronomical_Twilight object\n", "dtypes: bool(13), float64(14), int64(1), object(21)\n", "memory usage: 1.4+ GB\n" ] } ], "source": [ "df.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will also check the number of missing values in the dataset, so that we can drop or fill in the missing values." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ID 0\n", "Source 0\n", "TMC 1034799\n", "Severity 0\n", "Start_Time 0\n", "End_Time 0\n", "Start_Lat 0\n", "Start_Lng 0\n", "End_Lat 2478818\n", "End_Lng 2478818\n", "Distance(mi) 0\n", "Description 1\n", "Number 2262864\n", "Street 0\n", "Side 0\n", "City 112\n", "County 0\n", "State 0\n", "Zipcode 1069\n", "Country 0\n", "Timezone 3880\n", "Airport_Code 6758\n", "Weather_Timestamp 43323\n", "Temperature(F) 65732\n", "Wind_Chill(F) 1868249\n", "Humidity(%) 69687\n", "Pressure(in) 55882\n", "Visibility(mi) 75856\n", "Wind_Direction 58874\n", "Wind_Speed(mph) 454609\n", "Precipitation(in) 2025874\n", "Weather_Condition 76138\n", "Amenity 0\n", "Bump 0\n", "Crossing 0\n", "Give_Way 0\n", "Junction 0\n", "No_Exit 0\n", "Railway 0\n", "Roundabout 0\n", "Station 0\n", "Stop 0\n", "Traffic_Calming 0\n", "Traffic_Signal 0\n", "Turning_Loop 0\n", "Sunrise_Sunset 115\n", "Civil_Twilight 115\n", "Nautical_Twilight 115\n", "Astronomical_Twilight 115\n", "dtype: uint64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.isna().sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many columns with null values, and we will fill them with random values or the mean from the column. We will drop some text columns, as we are not doing any natural language processing right now, but feel free to explore them on your own. We will also drop the columns with too many Nans as filling them will throw our accuracy." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "df = df.drop(columns = ['ID','Start_Time','End_Time','Street','Side','Description','Number','City','Country','Zipcode','Timezone','Airport_Code','Weather_Timestamp','Wind_Chill(F)','Wind_Direction','Wind_Speed(mph)','Precipitation(in)'])" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "#Here we are filling the TMC with mean.\n", "df['TMC'] = df['TMC'].fillna(df['TMC'].mean())\n", "df['End_Lat'] = df['End_Lat'].fillna(df['End_Lat'].mean())\n", "df['End_Lng'] = df['End_Lng'].fillna(df['End_Lng'].mean())\n", "df['Temperature(F)'] = df['Temperature(F)'].fillna(df['Temperature(F)'].mean())\n", "df['Humidity(%)'] = df['Humidity(%)'].fillna(df['Humidity(%)'].mean())\n", "df['Pressure(in)'] = df['Pressure(in)'].fillna(df['Pressure(in)'].mean())\n", "df['Visibility(mi)'] = df['Visibility(mi)'].fillna(df['Visibility(mi)'].mean())\n", "df['Humidity(%)'] = df['Humidity(%)'].fillna(df['Humidity(%)'].mean())\n", "df['Pressure(in)'] = df['Pressure(in)'].fillna(df['Pressure(in)'].mean())\n", "df['Visibility(mi)'] = df['Visibility(mi)'].fillna(df['Visibility(mi)'].mean())\n", "\n", "\n", "df['Weather_Condition'] = df['Weather_Condition'].fillna('Fair')\n", "df['Sunrise_Sunset'] = df['Sunrise_Sunset'].fillna('Day')\n", "df['Civil_Twilight'] = df['Civil_Twilight'].fillna('Day')\n", "df['Nautical_Twilight'] = df['Nautical_Twilight'].fillna('Day')\n", "df['Astronomical_Twilight'] = df['Astronomical_Twilight'].fillna('Day')\n", "df['Weather_Condition'] = df['Weather_Condition'].fillna('Fair')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now all the columns contain no Nan values and we can go ahead with the preprocessing." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", " \n", "As you have observed in the dataset we have the start and end coordinates,so let us apply Haversine distance formula to get the accident coverage distance. Take note of how these functions use the row-wise operations, something that we have learnt before. If you need help while creating the user defined functions refer to this [notebook](02-Intro_to_cuDF_UDFs.ipynb)." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "from math import cos, sin, asin, sqrt, pi, atan2\n", "from numba import cuda" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "#Modify the code in this cell\n", "\n", "def haversine_distance_kernel(Start_Lat, Start_Lng, End_Lat, End_Lng, out):\n", " \n", " for i, (x_1, y_1, x_2, y_2) in enumerate(zip(Start_Lat, Start_Lng, End_Lat, End_Lng)):\n", " \n", " #Perform the computations here and store the final value in out[i]\n", " \n", " out[i] = " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Modify the code in this cell\n", "\n", "%%time\n", "#Add the arguments to the apply_rows function for the haversine distance kernel\n", "df = df.apply_rows()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Wow! The code segment that previously took 7 minutes to compute, now gets executed in less than a second! " ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "#Modify the code in this cell\n", "\n", "def haversine_distance_kernel(Start_Lat, Start_Lng, End_Lat, End_Lng, out):\n", " \n", " for i, (x_1, y_1, x_2, y_2) in enumerate(zip(Start_Lat, Start_Lng, End_Lat, End_Lng)):\n", " \n", " #Perform the computations here and store the final value in out[i]\n", " \n", " out[i] = " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#Modify the code in this cell\n", "\n", "%%time\n", "#Add the arguments to the apply_chunks function for the haversine distance kernel\n", "outdf = df.apply_chunks()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save the dataframe in a csv for future use, and make sure you refer to our sample solution and compared your code's performance with it." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SourceTMCSeverityStart_LatStart_LngEnd_LatEnd_LngDistance(mi)Temperature(F)Humidity(%)...WC_ThunderstormWC_Thunderstorms and RainWC_Thunderstorms and SnowWC_TornadoWC_Volcanic AshWC_Widespread DustWC_Widespread Dust / WindyWC_Wintry MixWC_Wintry Mix / Windyout
0MapQuest201.0339.865147-84.05872337.557578-100.4559810.0136.991.0...0000000001443.524390
1MapQuest201.0239.928059-82.83118437.557578-100.4559810.0137.9100.0...0000000001548.467903
2MapQuest201.0239.063148-84.03260837.557578-100.4559810.0136.0100.0...0000000001440.697621
3MapQuest201.0339.747753-84.20558237.557578-100.4559810.0135.196.0...0000000001429.927497
4MapQuest201.0239.627781-84.18835437.557578-100.4559810.0136.089.0...0000000001430.383177
\n", "

5 rows × 1930 columns

\n", "
" ], "text/plain": [ " Source TMC Severity Start_Lat Start_Lng End_Lat End_Lng \\\n", "0 MapQuest 201.0 3 39.865147 -84.058723 37.557578 -100.455981 \n", "1 MapQuest 201.0 2 39.928059 -82.831184 37.557578 -100.455981 \n", "2 MapQuest 201.0 2 39.063148 -84.032608 37.557578 -100.455981 \n", "3 MapQuest 201.0 3 39.747753 -84.205582 37.557578 -100.455981 \n", "4 MapQuest 201.0 2 39.627781 -84.188354 37.557578 -100.455981 \n", "\n", " Distance(mi) Temperature(F) Humidity(%) ... WC_Thunderstorm \\\n", "0 0.01 36.9 91.0 ... 0 \n", "1 0.01 37.9 100.0 ... 0 \n", "2 0.01 36.0 100.0 ... 0 \n", "3 0.01 35.1 96.0 ... 0 \n", "4 0.01 36.0 89.0 ... 0 \n", "\n", " WC_Thunderstorms and Rain WC_Thunderstorms and Snow WC_Tornado \\\n", "0 0 0 0 \n", "1 0 0 0 \n", "2 0 0 0 \n", "3 0 0 0 \n", "4 0 0 0 \n", "\n", " WC_Volcanic Ash WC_Widespread Dust WC_Widespread Dust / Windy \\\n", "0 0 0 0 \n", "1 0 0 0 \n", "2 0 0 0 \n", "3 0 0 0 \n", "4 0 0 0 \n", "\n", " WC_Wintry Mix WC_Wintry Mix / Windy out \n", "0 0 0 1443.524390 \n", "1 0 0 1548.467903 \n", "2 0 0 1440.697621 \n", "3 0 0 1429.927497 \n", "4 0 0 1430.383177 \n", "\n", "[5 rows x 1930 columns]" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = df.dropna()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.to_csv(\"../../data/data_proc.csv\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion \n", "\n", "Thus we have successfully used CuDF and CuPy to process the accidents dataset, and converted the data to a form more suitable to apply machine learning algorithms. In the extra labs for future labs in CuML we will be using this processed dataset. You must have observed the parallels between the RAPIDS pipeline and traditional pipeline while writing your code. Try to experiment with the processing and making your code as efficient as possible." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# References\n", "\n", "- Moosavi, Sobhan, Mohammad Hossein Samavatian, Srinivasan Parthasarathy, and Rajiv Ramnath. “A Countrywide Traffic Accident Dataset.”, 2019.\n", "\n", "- Moosavi, Sobhan, Mohammad Hossein Samavatian, Srinivasan Parthasarathy, Radu Teodorescu, and Rajiv Ramnath. \"Accident Risk Prediction based on Heterogeneous Sparse Data: New Dataset and Insights.\" In proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, ACM, 2019.\n", "\n", "- If you need to refer to the dataset, you can download it [here](https://www.kaggle.com/sobhanmoosavi/us-accidents)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\"Creative

\n", "\n", "- This dataset is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Licensing\n", " \n", "This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Previous Notebook](03-Cudf_Exercise.ipynb)\n", "     \n", "     \n", "     \n", "     \n", "[1](01-Intro_to_cuDF.ipynb)\n", "[2](02-Intro_to_cuDF_UDFs.ipynb)\n", "[3](03-Cudf_Exercise.ipynb)\n", "[4]\n", "     \n", "     \n", "     \n", "     \n", "\n", "\n", "     \n", "     \n", "     \n", "     \n", "     \n", "   \n", "[Home Page](../START_HERE.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.2" } }, "nbformat": 4, "nbformat_minor": 4 }