Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping\n",
    "\n",
    "Web scraping is a valuable tool for gathering information from the internet for data analysis projects.\n",
    "\n",
    "Today, I will demonstrate gathering information from the [International Space Station web scraping website](http://open-notify.org/) using python libraries and JSON.\n",
    "\n",
    "I will also be web scraping [Github](http://www.github.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping Status Codes\n",
    "\n",
    "Web scraping status codes can be helpful for debugging. Here are the codes:\n",
    "\n",
    "\n",
    "    200 — Everything went okay, and the server returned a result (if any).\n",
    "    201 — The request has been fulfilled and has resulted in one or more resources being created.\n",
    "    204 — The server has successfully fulfilled the request and that there is no additional content to send in the response payload body.\n",
    "    301 — The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or when an endpoint's name has changed.\n",
    "    400 — The server thinks you made a bad request. This can happen when you don't send the information the API requires to process your request (among other things).\n",
    "    401 — The server thinks you're not authenticated. This happens when you don't send the right credentials to access an API (we'll talk about this in a later mission).\n",
    "    403 — The resource you're trying to access is forbidden, and you don't have the right permissions to see it.\n",
    "    404 — The server didn't find the resource you tried to access.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping the International Space Station"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.get(\"http://api.open-notify.org/iss-now.json\")\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get('http://api.open-notify.org/iss-pass.json').status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This one requires two parameters. [The website](http://open-notify.org/Open-Notify-API/ISS-Pass-Times/) shows this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Including the latitude and longitude\n",
    "st_louis = {'lat': 36.270, 'lon': -90.1994}\n",
    "response1 = requests.get('http://api.open-notify.org/iss-pass.json', params=st_louis)\n",
    "response1.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'{\\n  \"message\": \"success\", \\n  \"request\": {\\n    \"altitude\": 100, \\n    \"datetime\": 1617881095, \\n    \"latitude\": 36.27, \\n    \"longitude\": -90.1994, \\n    \"passes\": 5\\n  }, \\n  \"response\": [\\n    {\\n      \"duration\": 612, \\n      \"risetime\": 1617897786\\n    }, \\n    {\\n      \"duration\": 635, \\n      \"risetime\": 1617903578\\n    }, \\n    {\\n      \"duration\": 499, \\n      \"risetime\": 1617909507\\n    }, \\n    {\\n      \"duration\": 448, \\n      \"risetime\": 1617915431\\n    }, \\n    {\\n      \"duration\": 580, \\n      \"risetime\": 1617921249\\n    }\\n  ]\\n}\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response1.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bytes"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response1.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `.json()` method will turn the bytes content into a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'message': 'success',\n",
       " 'request': {'altitude': 100,\n",
       "  'datetime': 1617881095,\n",
       "  'latitude': 36.27,\n",
       "  'longitude': -90.1994,\n",
       "  'passes': 5},\n",
       " 'response': [{'duration': 612, 'risetime': 1617897786},\n",
       "  {'duration': 635, 'risetime': 1617903578},\n",
       "  {'duration': 499, 'risetime': 1617909507},\n",
       "  {'duration': 448, 'risetime': 1617915431},\n",
       "  {'duration': 580, 'risetime': 1617921249}]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response1.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response1.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Realtime Observations\n",
    "\n",
    "The API scrapes the data in realtime. Notice the `'timestamp'` value increases by about 1 each time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'message': 'success', 'iss_position': {'latitude': '45.3559', 'longitude': '12.4882'}, 'timestamp': 1617881096}\n",
      "{'message': 'success', 'iss_position': {'latitude': '45.3862', 'longitude': '12.5655'}, 'timestamp': 1617881097}\n",
      "{'message': 'success', 'iss_position': {'latitude': '45.4164', 'longitude': '12.6428'}, 'timestamp': 1617881098}\n",
      "{'message': 'success', 'iss_position': {'latitude': '45.4618', 'longitude': '12.7590'}, 'timestamp': 1617881100}\n",
      "{'message': 'success', 'iss_position': {'latitude': '45.4919', 'longitude': '12.8366'}, 'timestamp': 1617881101}\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    time.sleep(1)\n",
    "    response = requests.get(\"http://api.open-notify.org/iss-now.json\")\n",
    "    content=response.json()\n",
    "    print(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `headers` property tells us valuable information. the `Content-Type` key in the `headers` dictionary is one of the most important, showing us how to decode the response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Server : nginx/1.10.3\n",
      "Date : Thu, 08 Apr 2021 11:24:55 GMT\n",
      "Content-Type : application/json\n",
      "Content-Length : 522\n",
      "Connection : keep-alive\n",
      "Via : 1.1 vegur\n"
     ]
    }
   ],
   "source": [
    "for key in response1.headers.keys():\n",
    "    print(key, ':', response1.headers[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As is expected, this content type is `application/json`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`astros.json` tells us how many astronauts are currently in space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'message': 'success',\n",
       " 'number': 7,\n",
       " 'people': [{'craft': 'ISS', 'name': 'Sergey Ryzhikov'},\n",
       "  {'craft': 'ISS', 'name': 'Kate Rubins'},\n",
       "  {'craft': 'ISS', 'name': 'Sergey Kud-Sverchkov'},\n",
       "  {'craft': 'ISS', 'name': 'Mike Hopkins'},\n",
       "  {'craft': 'ISS', 'name': 'Victor Glover'},\n",
       "  {'craft': 'ISS', 'name': 'Shannon Walker'},\n",
       "  {'craft': 'ISS', 'name': 'Soichi Noguchi'}]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "astros = requests.get('http://api.open-notify.org/astros.json')\n",
    "astros.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 7 people in space as of this writing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping Github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "headers = {'Authorization': 'token ###'}\n",
    "response = requests.get('https://api.github.com/users/stephentaul22', headers=headers)\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'login': 'stephentaul22',\n",
       " 'id': 52689411,\n",
       " 'node_id': 'MDQ6VXNlcjUyNjg5NDEx',\n",
       " 'avatar_url': 'https://avatars.githubusercontent.com/u/52689411?v=4',\n",
       " 'gravatar_id': '',\n",
       " 'url': 'https://api.github.com/users/stephentaul22',\n",
       " 'html_url': 'https://github.com/stephentaul22',\n",
       " 'followers_url': 'https://api.github.com/users/stephentaul22/followers',\n",
       " 'following_url': 'https://api.github.com/users/stephentaul22/following{/other_user}',\n",
       " 'gists_url': 'https://api.github.com/users/stephentaul22/gists{/gist_id}',\n",
       " 'starred_url': 'https://api.github.com/users/stephentaul22/starred{/owner}{/repo}',\n",
       " 'subscriptions_url': 'https://api.github.com/users/stephentaul22/subscriptions',\n",
       " 'organizations_url': 'https://api.github.com/users/stephentaul22/orgs',\n",
       " 'repos_url': 'https://api.github.com/users/stephentaul22/repos',\n",
       " 'events_url': 'https://api.github.com/users/stephentaul22/events{/privacy}',\n",
       " 'received_events_url': 'https://api.github.com/users/stephentaul22/received_events',\n",
       " 'type': 'User',\n",
       " 'site_admin': False,\n",
       " 'name': 'Stephen Taul',\n",
       " 'company': None,\n",
       " 'blog': 'www.linkedin.com/in/stephentaul',\n",
       " 'location': 'USA',\n",
       " 'email': None,\n",
       " 'hireable': None,\n",
       " 'bio': 'Data Scientist',\n",
       " 'twitter_username': None,\n",
       " 'public_repos': 21,\n",
       " 'public_gists': 1,\n",
       " 'followers': 0,\n",
       " 'following': 2,\n",
       " 'created_at': '2019-07-09T04:49:47Z',\n",
       " 'updated_at': '2021-04-08T11:17:00Z'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing my Repositories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing-CIA-Factbook-Data-Using-SQL\n",
      "Answering-Business-Questions-using-SQL\n",
      "api-post-test2\n",
      "App-Profile-Recommendation\n",
      "Building-a-Database-for-Crime-Reports\n",
      "Building-a-Spam-Filter-with-Naive-Bayes-Algorithm\n",
      "Building-Fast-Queries-on-a-CSV\n",
      "Cleaning-and-Analyzing-Employee-Exit-Surveys\n",
      "Exploring-Ebay-Car-Sale-Data\n",
      "Exploring-Hacker-News-Posts\n",
      "Exploring-The-World-Happiness-Reports-by-Region\n",
      "Finding-the-Best-Markets-to-Advertise-In\n",
      "hello-world\n",
      "Investigating-COVID-19-Virus-Trends\n",
      "Investigating-Movie-Ratings\n",
      "Mobile-App-for-Lottery-Addiction\n",
      "Predicting-Car-Prices\n",
      "Star-Wars-Survey\n",
      "Visualizing-Earnings-Based-on-College-Majors\n",
      "Visualizing-the-Gender-Gap-in-College-Degrees\n",
      "Winning-Jeopardy\n"
     ]
    }
   ],
   "source": [
    "repos = requests.get('https://api.github.com/users/stephentaul22/repos', headers=headers).json()\n",
    "for i in range(len(repos)):\n",
    "    print(repos[i]['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Starred Repositories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building-a-Database-for-Crime-Reports\n",
      "Building-Fast-Queries-on-a-CSV\n",
      "Predicting-Car-Prices\n",
      "Exploring-The-World-Happiness-Reports-by-Region\n",
      "Climate-ChangePrediction\n",
      "Answering-Business-Questions-using-SQL\n",
      "Winning-Jeopardy\n",
      "Building-a-Spam-Filter-with-Naive-Bayes-Algorithm\n"
     ]
    }
   ],
   "source": [
    "starred = requests.get('https://api.github.com/users/stephentaul22/starred', headers=headers).json()\n",
    "for i in range(len(starred)):\n",
    "    print(starred[i]['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Token Trick\n",
    "\n",
    "The account username is not necessary since we have the token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'login': 'stephentaul22',\n",
       " 'id': 52689411,\n",
       " 'node_id': 'MDQ6VXNlcjUyNjg5NDEx',\n",
       " 'avatar_url': 'https://avatars.githubusercontent.com/u/52689411?v=4',\n",
       " 'gravatar_id': '',\n",
       " 'url': 'https://api.github.com/users/stephentaul22',\n",
       " 'html_url': 'https://github.com/stephentaul22',\n",
       " 'followers_url': 'https://api.github.com/users/stephentaul22/followers',\n",
       " 'following_url': 'https://api.github.com/users/stephentaul22/following{/other_user}',\n",
       " 'gists_url': 'https://api.github.com/users/stephentaul22/gists{/gist_id}',\n",
       " 'starred_url': 'https://api.github.com/users/stephentaul22/starred{/owner}{/repo}',\n",
       " 'subscriptions_url': 'https://api.github.com/users/stephentaul22/subscriptions',\n",
       " 'organizations_url': 'https://api.github.com/users/stephentaul22/orgs',\n",
       " 'repos_url': 'https://api.github.com/users/stephentaul22/repos',\n",
       " 'events_url': 'https://api.github.com/users/stephentaul22/events{/privacy}',\n",
       " 'received_events_url': 'https://api.github.com/users/stephentaul22/received_events',\n",
       " 'type': 'User',\n",
       " 'site_admin': False,\n",
       " 'name': 'Stephen Taul',\n",
       " 'company': None,\n",
       " 'blog': 'www.linkedin.com/in/stephentaul',\n",
       " 'location': 'USA',\n",
       " 'email': None,\n",
       " 'hireable': None,\n",
       " 'bio': 'Data Scientist',\n",
       " 'twitter_username': None,\n",
       " 'public_repos': 21,\n",
       " 'public_gists': 1,\n",
       " 'followers': 0,\n",
       " 'following': 2,\n",
       " 'created_at': '2019-07-09T04:49:47Z',\n",
       " 'updated_at': '2021-04-08T11:17:00Z'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get('https://api.github.com/user', headers=headers).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `post` requests\n",
    "\n",
    "`post` requests send information, whereas `get` requests receive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "201"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.post('https://api.github.com/user/repos', headers=headers, json={'name': 'api-post-test'})\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `patch` requests \n",
    "\n",
    "`patch` requests update information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "404"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "payload = {'name': 'api-post-test',\n",
    "           'description': 'Api Patch'}\n",
    "response = requests.patch('https://api.github.com/repos/stephentaul22/api-post-test', headers=headers, json=payload)\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `delete` requests\n",
    "\n",
    "`delete` requests will remove an object from the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "204"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.delete('https://api.github.com/repos/stephentaul22/api-post-test', headers=headers)\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verifying that the repo is deleted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.get('https://api.github.com/repos/stephentaul22/api-post-test', headers=headers)\n",
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this project I demonstrated a few useful ways to perform web scraping using the python `requests` API library. This is a simple introduction and the techniques here can be applied to larger, more detailed projects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}