{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading Images in Bulk from U.S. Naval Archives\n",
    "\n",
    "The Image Dataset can be found here [Here](https://www.nrlmry.navy.mil/tcdat/) \n",
    "\n",
    "The Text Dataset can be found and downloaded from [Here]( https://www.nhc.noaa.gov/data/#hurdat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python3\n",
    "#Libraries to Scrap Text Data and Recurse Through the Directory and Fetch only the Required Data\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import subprocess\n",
    "limit = 16 # 16 Concurrent parallel Downloads to Speed them up\n",
    "\n",
    "page_link = 'https://www.nrlmry.navy.mil/tcdat/'\n",
    "parallel=0\n",
    "parallel_links=[]\n",
    "textContent = []\n",
    "for i in range(20): # Loop through tc01 to tc19\n",
    "    new_page_link = page_link+'tc'+str(int(i/10))+str(int(i%10))+'/'\n",
    "    print(new_page_link)\n",
    "    links=[]\n",
    "    links.append(new_page_link+'ATL/') # Get Atlantic Cyclones \n",
    "    for j in links:  # Loop through all Cyclones and Download Only IR images in 1Km range\n",
    "        print(j)\n",
    "        page_response = requests.get(j,timeout=10)\n",
    "        page_content = BeautifulSoup(page_response.content, \"html.parser\")\n",
    "        for k in range(5,len(page_content.find_all(\"a\"))-1): \n",
    "            text_output = page_content.find_all(\"a\")[k].text\n",
    "            parallel_links.append(j+text_output+'ir/geo/1km/') \n",
    "            if(int(len(parallel_links)==limit)): # Get Batch of 16 and Put them to Download in Parallel \n",
    "                p1 = subprocess.Popen([\"echo\"]+parallel_links,stdout=subprocess.PIPE)\n",
    "                p2 = subprocess.Popen([\"xargs\" ,\"-n\",\"1\",\"-P\",str(int(len(parallel_links))),'wget','-r','-np','-e','robots=off','-R','index.html*'],stdin=p1.stdout, stdout=subprocess.PIPE,stderr=subprocess.PIPE)\n",
    "                p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.\n",
    "                output,error = p2.communicate()\n",
    "                parallel_links=[]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Warning !! \n",
    "\n",
    "#### The Cell Type has been changed to Markdown so that you don't accidentally run the code.\n",
    "\n",
    "#### It is not recommended to run the code until you fully understand the code and it's consequences , It can download huge amount of data (~ 10's of GB's ) thereby filling your computer's memory"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
