Lecture_3_preoprocessing/Lecture_3_HTML_parsing.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:158c6f2a34137ff9dd4b9fe4df467ce05ae6e13a4dd1b7f470b88be9662df554"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import urllib2\n",
      "response = urllib2.urlopen('http://gozips.uakron.edu/~fbao5/')\n",
      "html=response.read()\n",
      "print html"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<html>\n",
        "     <meta charset=\"UTF-8\">\n",
        "     <head>\n",
        "          <title>Prof. Dr. Forrest Sheng Bao</title>\n",
        "     </head>\n",
        "\n",
        "   <body>\n",
        "   <p></p>\n",
        "   <p></p>\n",
        "   <p></p>\n",
        "   <p>I have moved my homepage to <a href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a>.</p> \n",
        "   <p>Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. </p>\n",
        "   <p>I believe that a necessary condition for academic freedom is the freedom of speech. </p> \n",
        "   <p> <b>See also:</b> \n",
        "   <ul>\n",
        "     <li> <a href=\"https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei\">Die Gedanken sind frei (The thoughts are free.)</a></li>\n",
        "     <li> <a href=\"http://web.stanford.edu/dept/pres-provost/president/speeches/951005dieluft.html\">Der Luft der Freiheit weht (the wind of freedom blows)</a></li>\n",
        "     <li><a href=\"https://en.wikipedia.org/wiki/Freedom_of_thought\">Freedom of thought</a></li>\n",
        "     <li><a href=\"https://en.wikipedia.org/wiki/Academic_freedom\">Academic freedom </a></li>\n",
        "     <li><a href=\"https://en.wikipedia.org/wiki/Internet_censorship_in_China\">Internet censorship in China</a></li>\n",
        "     <li><a href=\"https://googleblog.blogspot.com/2010/01/new-approach-to-china.html\">A new approach to China</a></li>\n",
        "   </ul>\n",
        "\n",
        "\n",
        "</p>\n",
        "\n",
        "<p>The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein</p>\n",
        "\n",
        "   </body>\n",
        "\n",
        "\n",
        "\n",
        "     <!--body>\n",
        "      <p>This website could be outdated. For full and up-to-date information about my research, please visit: <a href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a></p>\n",
        "          <hr>\n",
        "\n",
        "<div>Forrest Sheng Bao, Assistant Professor<br>\n",
        "Dept. of Electrical and Computer Engineering<br>\n",
        "University of Akron, Akron, Ohio, USA<br>\n",
        "</div>\n",
        "<p><a href=\"ForrestShengBao.pdf\">Curriculum Vitae (PDF, local)</a>  &nbsp;&nbsp;  <a href=\"https://docs.google.com/file/d/0B6mdryaP3bK2ZDZmNjYwOGItOWFkZi00YTY5LWFmZjEtYTdmM2IyNDNlZGY2/edit?usp=sharing\" target=\"_blank\">Curriculum Vitae</a> (PDF, Google Docs) <a href=\"http://scholar.google.com/citations?user=XWpFf0IAAAAJ&amp;hl=en\" target=\"_blank\">Full publication list</a> (Google Scholar)\n",
        "</p>\n",
        "\n",
        "<div dir=\"ltr\">\n",
        "<h2>Research Interests</h2>\n",
        "<p>Artificial Intelligence, Signal/Image Processing, Wireless Networking, Bioinformatics</p>\n",
        "<h2>Research Grants</h2>\n",
        "<img src=\"http://www.nsf.gov/images/logos/nsf1.gif\" width=\"198\" border=\"0\" height=\"200\">\n",
        "<img src=\"https://news.boisestate.edu/update/files/2015/04/AFOSRlogo.jpg\" width=\"180\" border=\"0\" height=\"180\">\n",
        "<img src=\"https://assets.onestore.ms/cdnfiles/onestorerolling-1606-01000/shell/v3/images/logo/microsoft.png\" border=\"0\">\n",
        "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/de/US-FederalAviationAdmin-Seal.svg/720px-US-FederalAviationAdmin-Seal.svg.png\" border=\"0\" width=\"180\" border=\"0\" height=\"180\">\n",
        "<ul>\n",
        "<li><b>National Science Foundation (NSF)</b>, Award ID 1616216, <a href=\"http://www.nsf.gov/awardsearch/showAward?AWD_ID=1616216\" target=\"_blank\" rel=\"nofollow\">Collaborative Research: Productivity Prediction of Microbial Cell Factories using Machine Learning and Knowledge Engineering</a>, 50% of total $470,000 budget, 2016-2019.</li>\n",
        "<li><b>Federal Aviation Administration (FAA)</b>, Opportunity No. 15-C-TTHP-100PM71515, Center of Excellence for Technical Training and Human Performance, a multi-institution multi-PI center of $5M minimum budget 2016-2021 and possible additional funding for another 5 years. PI in Curriculum Architecture Gap Analysis (first year total $90,000 and possible future forwarding grant), a project using Natural Language Processing to analyze the redundancy and inconsistency between training documents. </li>\n",
        "</ul>\n",
        "\n",
        "\n",
        "<h2>Selected Publications</h2>\n",
        "<h3>Journal papers</h3>\n",
        "<ul>\n",
        "<li>Duo Chen, Shuren Wang, <u>Forrest S. Bao</u>, EEG-based Epileptic Focus Localization Using Discrete Wavelet Transform, <i>IEEE Transactions on Neural Engineering and Rehabilitation Engineering</i>, accepted, 2016 (2014 IF: 3.188)</li>\n",
        "<li><u>Forrest S. Bao</u>, Wujun Zhou, Wu Jiang, Yinfei Yang, and Qian Chen, <a href=\"http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7433923\" target=\"_blank\" rel=\"nofollow\">Coverage-Based Lossy Node Localization for Wireless Sensor Networks</a>, <i>IEEE Sensors Journal</i>, vol. 16, no. 11, 2016 (2014 IF: 1.762)</li>\n",
        "<li>Stephen G. Wu, Yuxuan Wang, Wu Jiang, Tolutola Oyetunde, Ruilian Yao, Kazuyuki Shimizu, Yinjie J. Tang, and <u>Forrest S. Bao</u>, <a href=\"http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004838\" target=\"_blank\" rel=\"nofollow\">Rapid Prediction of Bacterial Heterotrophic Fluxomics using Machine Learning and Constraint Programming</a>, <i>PLoS Computational Biology</i>, 12(4): e1004838, Apr. 19, 2016 (2014 IF: 4.62)</li>\n",
        "<li>Duo Chen, Jiasong Wang, Ming Yan, <u>Forrest S. Bao</u>, <a href=\"https://sites.google.com/site/forrestbao/Preprint_CPNR.pdf?attredirects=0&amp;d=1\" target=\"_blank\">A Complex Prime Numerical Representation of Amino Acids for Protein Function Comparison</a>, <i>Journal of Computational Biology</i>, vol. 23, no. 8, pp. 669-677, 2016 (2014 IF:1.737)</li>\n",
        "<li>Haitao Zhu, Jinlong Zhu, <u>Forrest S. Bao</u> <i>et al.</i>, Statistical parametric mapping for analyzing interictal magnetoencephalography in patients with left frontal lobe epilepsy, <i>Seizure</i>, vol. 34, pp. 38--43, January 2016 (2014 IF: 1.822)</li>\n",
        "<li>Shi Chen and <u>Forrest S. Bao</u>, <a href=\"http://www.sciencedirect.com/science/article/pii/S0304380015003555\" target=\"_blank\" rel=\"nofollow\">Linking Body Size and Energetics with Predation Strategies: a Game Theoretic Modeling Framework</a>, <i>Ecological Modeling</i>, vol. 316, pp. 81--86, November 2015 (2014 IF: 2.321)</li>\n",
        "<li><u>Forrest S. Bao</u>, and Yuanlin Zhang, <a href=\"http://arxiv.org/abs/0906.0205\">A Survey of Tree Convex Sets Test</a>, <i><a href=\"http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8640.2012.00418.x/abstract\">Computational Intelligence</a></i>, Volume 28, Issue 3, pages 358--372, August 2012 (2014 IF: 0.673)</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, <a href=\"http://www.hindawi.com/journals/cin/2011/406391/\">PyEEG: An Open Source Python Module for EEG/MEG Feature Extraction</a>, <i>Computational Intelligence and Neuroscience</i>, 2011 (2014 IF: 0.596)</li>\n",
        "<li>Xiao-Peng Qi, <u>Forrest S. Bao</u> and Zhixin Xie, <a href=\"http://www.plosone.org/article/info:doi/10.1371/journal.pone.0004971\">Small RNA Deep Sequencing Reveals Role for Arabidopsis thaliana RNA-dependent RNA Polymerases in Viral siRNA Biogenesis</a>, <a href=\"http://www.plosone.org\">PLoS One</a> 4(3): e4971, Mar. 24, 2009  (2014 IF: 3.234)</li></ul>\n",
        "<h3>Peer-reviewed conference papers</h3>\n",
        "<ul>\n",
        "<li>Duo Chen, Shuren Wan, <u>Forrest S. Bao</u>, <a href=\"http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7359914\" target=\"_blank\" rel=\"nofollow\">EEG-based seizure detection using discrete wavelet transform through full-level decomposition</a>, IEEE BIBM 2015, Washington, DC</li>\n",
        "<li>Duo Chen, Shuren Wan, <u>Forrest S. Bao</u>, Epileptic Focus Localization Using EEG Based On Discrete Wavelet Transform Through Full-Level Decomposition, <a href=\"http://mlsp2015.conwiz.dk/home.htm\">IEEE MLSP 2015</a>, Boston, MA</li>\n",
        "<li>Yaowei Yan, Nghi Huu Tran, <u>Forrest S. Bao</u>, Gossiping Along the path: A Direction-Biased Routing Scheme for Wireless Ad Hoc Networks, <a href=\"http://globecom2015.ieee-globecom.org/\">IEEE GLOBECOM 2015</a>, San Diego, CA</li>\n",
        "<li>Yinfei Yang, Yaowei Yan, Minghui Qiu, <u>Forrst S. Bao</u>, Semantic Analysis and Helpfulness Prediction of Text for Online Product Reviews, <a href=\"http://acl2015.org/\">ACL-2015</a>, Beijing, China</li>\n",
        "<li>Yaowei Yan, Chris Gutierrez, Jeriah Jn-Charles, <u>Forrest S. Bao</u> and Yuanlin Zhang, Accelerating SAT Solving by Common Subclause Elimination, <i>AAAI 2015</i></li>\n",
        "<li><u>Forrest S. Bao</u> <i>et al.</i>,Coverage-Based Lossy Node Localization in Wireless Sensor Networks Using Chi-Square Test, IEEE WCNC, 2014, Istanbul, Turkey </li>\n",
        "<li>Shiqi Zhang, Mohan Sridharan and Forrest S. Bao, \"<a href=\"http://redwood.cs.ttu.edu/~smohan/Papers/icdl12_aspPomdp.pdf\" target=\"_blank\">Integrating Non-Monotonic Logic Programming and Hierarchical POMDPs on Mobile Robots,</a>\" IEEE ICDL-EpiRob, 2012 (Paper of Excellence award)</li>\n",
        "<li>Sai Rohit Gouravajhala, David Wang, Lunal Khuon, <u>Forrest S. Bao</u>, Sarvesh Kulkarni, <a href=\"http://forrest.bao.googlepages.com/HealthInno2012.pdf?attredirects=0\" target=\"_blank\">Implementing a Real-Time Seizure Detection Application on a Smartphone as part of a Multi-Modal Platform</a>, <a href=\"http://healthinnov.embs.org/2012conf/\"><i>IEEE HIC 2012</i></a>, Houston, TX</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, Automated extraction of nested sulcus features from human brain MRI data, <a href=\"http://embc2012.embs.org/\"><i>IEEE EMBC 2012</i></a>, San Diego, CA</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, <a href=\"http://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/4833/5395\">Temporally Expressive Planning Based on Answer Set Programming with Constraints</a>, <i>AAAI-12</i> (2 pages), Toronto, ON</li>\n",
        "<li>Shiqi Zhang, <u>Forrest S. Bao</u>, Mohan Sridharan, <a href=\"http://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/4799/5420\">Combining Probabilistic Planning and Logic Programming on Mobile Robots</a>, <i>AAAI-12</i> (2 pages), Toronto, ON</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, <a href=\"http://www.aaai.org/ocs/index.php/AAAI/AAAI11/paper/view/3546\">Medical Treatment Conflict Resolving in Answer Set Programming</a>, <i>AAAI-11</i> (2 pages), San Francisco, CA</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, Fast Phased Small RNA Cycle Counting Algorithms, <a href=\"http://bioinformatics-bioengineering.org/\"><i>IEEE BIBE 2010</i></a>, Philadelphia, PA</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, <a href=\"http://forrest.bao.googlepages.com/EMBC2009_Forrest_Bao.pdf\">Automated Epilepsy Diagnosis Using Interictal Scalp EEG</a>, <a href=\"http://www.embc09.org/\"><i>IEEE EMBC 2009</i></a>, Minneapolis, MN</li>\n",
        "<li><u>Forrest S. Bao</u>, <i>et al.</i>, <a href=\"http://arxiv.org/abs/0804.3361\">A New Approach to Automated Epileptic Diagnosis Using EEG and Probabilistic Neural Network</a>, <a href=\"http://dblab.cs.wright.edu/ictai2008/\"><i>IEEE ICTAI 2008</i></a>, Dayton, OH</li>\n",
        "<li>Lei Yuan, <u>Forrest S. Bao</u> and Guanming Lu, Automatic Analysis of Facial Expressions of Acute Pain in Neonates Using Boosted Gabor Features, <a href=\"http://dblab.cs.wright.edu/ictai2008/\"><i>IEEE ICTAI 2008</i></a>, Dayton, OH</li>\n",
        "<li>Zhendong Zhao, Lei Yuan, Yuxuan Wang, <u>Forrest S. Bao</u>,  <i>et al.</i>, <a href=\"http://arxiv.org/pdf/0706.0585\">A Novel Model of Working Set Selection for SMO Decomposition Methods</a>, <a href=\"http://ictai07.ceid.upatras.gr/\"><i>IEEE ICTAI 2007</i></a>, Greece</li>\n",
        "<li>Yu-Xuan Wang and <u>Forrest S. Bao</u>, <a href=\"http://forrest.bao.googlepages.com/WCA_ETS.pdf\">An Entropy-based Weighted Clustering Algorithm and Its Optimization for Ad hoc Networks</a>, <a href=\"http://www.gel.usherbrooke.ca/WiMob2007/\"><i>IEEE WiMob 2007</i></a>, White Plains, NY</li></ul>\n",
        "<h2>Selected Services</h2>\n",
        "<ul><li>Junior Associate Editor: <a href=\"http://health.embs.org/author/fbao/\" target=\"_blank\">IEEE Journal of Translational Engineering in Health and Medicine</a>, 2012 -- 2016<br>\n",
        "</li>\n",
        "<li>Session Chair: AI in Medicine, 20th IEEE Int'l Conference on Tools with Artificial Intelligence (ICTAI'08)</li>\n",
        "<li>Program Committee member: 23rd Int'l Joint Conference on Artificial Intelligence (IJCAI 2013), IEEE 27th-29th Int'l Symposium on Computer-Based Medical System (<a href=\"http://sites.ieee.org/cbms/\" target=\"_blank\" rel=\"nofollow\">CBMS 2014, </a>CBMS 2015, CBMS 2016), 30th-31st AAAI Conf. on AI (<a href=\"http://www.aaai.org/Conferences/AAAI/aaai16.php\" target=\"_blank\" rel=\"nofollow\">AAAI-16</a>, <a href=\"http://www.aaai.org/Conferences/AAAI/aaai17.php\" target=\"_blank\" rel=\"nofollow\">AAAI-17</a>), IEEE GLOBECOM 2016</li>\n",
        "<li>Journal Reviewer: IEEE Transactions on Image Processing, IEEE/ACM Transactions on Computational Biology and Bioinformatics, Wireless Communications &amp; Mobile Computing (John Wiley &amp; Sons), Neural Processing Letters, Computational Intelligence, Brain Connectivity, Journal of Neuroscience Methods</li>\n",
        "<li>Conference Reviewer: IEEE Wireless Communications and Networking Conference (WCNC) 2007, IEEE Int'l Conf. on Mobile Ad-hoc and Sensor Systems (MASS) 2008, IEEE 35th Int'l Conf. on Acoustics, Speech, and Signal Processing (ICASSP) 2010, Human Brain Mapping (HBM) 2011, 25th Conference on Artificial Intelligence (AAAI-11), Human Brain Mapping (HBM) 2013</li>\n",
        "<li>Judge: 2013 FIRST Robotics Hub City (Lubbock, TX) Regional Competitions<br>\n",
        "</li></ul>\n",
        "\n",
        "<h2>News Release</h2>\n",
        "<ul><li>A Neural Net That Diagnoses Epilepsy, <i>MIT Technology Review website</i>, <a href=\"http://www.technologyreview.com/blog/arxiv/23465/\">http://www.technologyreview.com/blog/arxiv/23465/</a></li>\n",
        "<li>New Diagnostic Tool for Automatic Epilepsy Diagnosis, <i>epilepsyUSA</i> (the official magazine of Epilepsy Foundation of America), 2009, No. 5, pp.9-10, <a href=\"http://www.epilepsyfoundation.org/resources/epilepsyusa/upload/USA-2009-Issue5-final.pdf\">http://www.epilepsyfoundation.org/resources/epilepsyusa/upload/USA-2009-Issue5-final.pdf</a></li></ul>\n",
        "<hr>\n",
        "<p>Contact me: FORREST.BAO [At] GMAIL.COM</p>\n",
        "<p>I also have a blog: <a href=\"http://forrestbao.blogspot.com/\" title=\"http://forrestbao.blogspot.com/\">http://forrestbao.blogspot.com/</a></p>\n",
        "<p><b>For up-to-date and detailed information about my research, please visit my full homepage <a href=\"https://sites.google.com/site/forrestba\">https://sites.google.com/site/forrestbao</a></b></p>\n",
        "</div>\n",
        "<div dir=\"ltr\">&nbsp;</div>\n",
        "\n",
        "\n",
        "</body-->\n",
        "\n",
        "</html>\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import bs4"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "soup = bs4.BeautifulSoup(html, 'html.parser')\n",
      "#print soup.prettify()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# If u want text only but not the parsing tree, do: \n",
      "print soup.get_text()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "\n",
        "Prof. Dr. Forrest Sheng Bao\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "I have moved my homepage to https://sites.google.com/site/forrestbao.\n",
        "Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. \n",
        "I believe that a necessary condition for academic freedom is the freedom of speech. \n",
        " See also:\n",
        "\n",
        " Die Gedanken sind frei (The thoughts are free.)\n",
        " Der Luft der Freiheit weht (the wind of freedom blows)\n",
        "Freedom of thought\n",
        "Academic freedom \n",
        "Internet censorship in China\n",
        "A new approach to China\n",
        "\n",
        "\n",
        "The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein\n",
        "\n",
        "\n",
        "\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "all_paras = soup.find_all('p') # find all nodes of tag p under node soup\n",
      "print \"\\n\".join(map(str, all_paras))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<p></p>\n",
        "<p></p>\n",
        "<p></p>\n",
        "<p>I have moved my homepage to <a href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a>.</p>\n",
        "<p>Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. </p>\n",
        "<p>I believe that a necessary condition for academic freedom is the freedom of speech. </p>\n",
        "<p> <b>See also:</b>\n",
        "<ul>\n",
        "<li> <a href=\"https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei\">Die Gedanken sind frei (The thoughts are free.)</a></li>\n",
        "<li> <a href=\"http://web.stanford.edu/dept/pres-provost/president/speeches/951005dieluft.html\">Der Luft der Freiheit weht (the wind of freedom blows)</a></li>\n",
        "<li><a href=\"https://en.wikipedia.org/wiki/Freedom_of_thought\">Freedom of thought</a></li>\n",
        "<li><a href=\"https://en.wikipedia.org/wiki/Academic_freedom\">Academic freedom </a></li>\n",
        "<li><a href=\"https://en.wikipedia.org/wiki/Internet_censorship_in_China\">Internet censorship in China</a></li>\n",
        "<li><a href=\"https://googleblog.blogspot.com/2010/01/new-approach-to-china.html\">A new approach to China</a></li>\n",
        "</ul>\n",
        "</p>\n",
        "<p>The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein</p>\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for num, tag in enumerate(all_paras):\n",
      "    print num+1, tag.name, tag.attrs, tag.string\n",
      "# Question: Why don't we see \"See also\" or any hyperlinks in the results below? \n",
      "# Because...."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 p {} None\n",
        "2 p {} None\n",
        "3 p {} None\n",
        "4 p {} None\n",
        "5 p {} Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. \n",
        "6 p {} I believe that a necessary condition for academic freedom is the freedom of speech. \n",
        "7 p {} None\n",
        "8 p {} The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "soup.b\n",
      "# Using a tag name as an attribute will give you only the first tag by that name"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "<b>See also:</b>"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Going down, Solution 1 \n",
      "# use .contents to access all childern of a node AS A LIST\n",
      "print [child.name for child in soup.contents]\n",
      "print [child.name for child in soup.html.contents]\n",
      "print [child.name for child in soup.body.contents]\n",
      "# Note that head is not a child of <html> why? "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[u'html', None]\n",
        "[None, u'meta']\n",
        "[None, u'p', None, u'p', None, u'p', None, u'p', None, u'p', None, u'p', None, u'p', None, u'p', None]\n"
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# The example below extracts all paragraphs that contains a hyperlink\n",
      "for child in soup.body.contents: \n",
      "    if child.name == \"p\" and len(child.contents) > 0: \n",
      "        print len(child.contents), child.find_all('a')  \n",
      "        # note here that we find all `a' tags under each paragraph p "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "3 [<a href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a>]\n",
        "1 []\n",
        "1 []\n",
        "5 [<a href=\"https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei\">Die Gedanken sind frei (The thoughts are free.)</a>, <a href=\"http://web.stanford.edu/dept/pres-provost/president/speeches/951005dieluft.html\">Der Luft der Freiheit weht (the wind of freedom blows)</a>, <a href=\"https://en.wikipedia.org/wiki/Freedom_of_thought\">Freedom of thought</a>, <a href=\"https://en.wikipedia.org/wiki/Academic_freedom\">Academic freedom </a>, <a href=\"https://en.wikipedia.org/wiki/Internet_censorship_in_China\">Internet censorship in China</a>, <a href=\"https://googleblog.blogspot.com/2010/01/new-approach-to-china.html\">A new approach to China</a>]\n",
        "1 []\n"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# So how do we get the string \"see also\"?\n",
      "for p in soup.body.find_all('p'):\n",
      "    for child in p.contents:\n",
      "        if child.name == \"b\":\n",
      "            print child.string\n",
      "# Only a node without child has the attribute .string. \n",
      "# Note here that the Python interpreter compared an ASCII string with a UTF-8 string. "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "See also:\n"
       ]
      }
     ],
     "prompt_number": 92
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# or a quicker solution without going down the parsing tree is: \n",
      "print soup.find_all('b')\n",
      "\n",
      "#Note that becuse the b tag is under <html>, <body>, and some <p> tags, you can run .find_all from anyo those levels on the parsin tree to find the b tag. \n",
      "print soup.body.find_all('b')\n",
      "print soup.html.find_all('b')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[<b>See also:</b>]\n",
        "[<b>See also:</b>]\n",
        "[<b>See also:</b>]\n"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Going down, Solution 2\n",
      "# Instead of getting them as a list, you can iterate over a tag\u2019s children using the .children generator\n",
      "print soup.body.children"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<listiterator object at 0x7fa5ec13f7d0>\n"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# How to move \"horizontally and right\" on the parsing tree \n",
      "print soup.p.next_siblings  # You are smart enough to figure out what soup.p.next_sibling (singular) wil be \n",
      "\n",
      "print [(sb.name, sb.string) for sb in soup.p.next_siblings]\n",
      "# Remember that the parsing error is because the HTML code has syntax error at <meta> "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<generator object next_siblings at 0x7fd3940ee730>\n",
        "[(None, u'\\n'), (u'p', None), (None, u'\\n'), (u'p', None), (None, u'\\n'), (u'p', None), (None, u'\\n'), (u'p', u'Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. '), (None, u'\\n'), (u'p', u'I believe that a necessary condition for academic freedom is the freedom of speech. '), (None, u'\\n'), (u'p', None), (None, u'\\n'), (u'p', u\"The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein\"), (None, u'\\n')]\n"
       ]
      }
     ],
     "prompt_number": 74
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# You can also move \"horizontally and left\" on the parsing tree by using \"previous sibling\"\n",
      "print soup.p.next_sibling.previous_sibling == soup.p"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "True\n"
       ]
      }
     ],
     "prompt_number": 111
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Naturally, .previous_siblings will generate all previous siblings \n",
      "last_para = soup.find_all('p')[-1]\n",
      "print last_para\n",
      "\n",
      "all_pres = list(last_para.previous_siblings)\n",
      "all_pres  # Note that it is in reversed order\n",
      "\n",
      "#soup.find_all('p')[:-1][::-1]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<p>The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it. -- Albert Einstein</p>\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 126,
       "text": [
        "[u'\\n',\n",
        " <p> <b>See also:</b>\\n<ul>\\n<li> <a href=\"https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei\">Die Gedanken sind frei (The thoughts are free.)</a></li>\\n<li> <a href=\"http://web.stanford.edu/dept/pres-provost/president/speeches/951005dieluft.html\">Der Luft der Freiheit weht (the wind of freedom blows)</a></li>\\n<li><a href=\"https://en.wikipedia.org/wiki/Freedom_of_thought\">Freedom of thought</a></li>\\n<li><a href=\"https://en.wikipedia.org/wiki/Academic_freedom\">Academic freedom </a></li>\\n<li><a href=\"https://en.wikipedia.org/wiki/Internet_censorship_in_China\">Internet censorship in China</a></li>\\n<li><a href=\"https://googleblog.blogspot.com/2010/01/new-approach-to-china.html\">A new approach to China</a></li>\\n</ul>\\n</p>,\n",
        " u'\\n',\n",
        " <p>I believe that a necessary condition for academic freedom is the freedom of speech. </p>,\n",
        " u'\\n',\n",
        " <p>Of course, due to Internet censorship in some countries, my homepage hosted at Google may not be accessible. </p>,\n",
        " u'\\n',\n",
        " <p>I have moved my homepage to <a href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a>.</p>,\n",
        " u'\\n',\n",
        " <p></p>,\n",
        " u'\\n',\n",
        " <p></p>,\n",
        " u'\\n',\n",
        " <p></p>,\n",
        " u'\\n']"
       ]
      }
     ],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Let's try something slightly harder: find the sibling after the <p> that has <b>see also</b> as a child\n",
      "star = None\n",
      "for p in soup.find_all('p'):\n",
      "    if p == None:\n",
      "        continue \n",
      "    if len(p.contents) > 0:# and  p.contents[0].name == \"b\":\n",
      "        children_names = [child.name for chiled in p.contents]\n",
      "        if 'b' in children_names:\n",
      "            star = p\n",
      "print star.next_sibling\n",
      "# What the heck? star was never found? "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "AttributeError",
       "evalue": "'NoneType' object has no attribute 'next_sibling'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-108-bd253f5c7788>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;34m'b'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mchildren_names\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m             \u001b[0mstar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0;32mprint\u001b[0m \u001b[0mstar\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnext_sibling\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m \u001b[0;31m# What the heck? Nothing?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'next_sibling'"
       ]
      }
     ],
     "prompt_number": 108
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# A typo! chiled? the variable \"child\" was a previous one. Global variable is dangerous! Do not write one piece of lenghty code. \n",
      "# Now, let's do it right. \n",
      "star = None\n",
      "for p in soup.find_all('p'):\n",
      "    if p == None:\n",
      "        continue \n",
      "    if len(p.contents) > 0:# and  p.contents[0].name == \"b\":\n",
      "        children_names = [child.name for child in p.contents]\n",
      "        if 'b' in children_names:\n",
      "            print \"Eureka!\"\n",
      "            star = p  # Let's find the <p> tag that has a <b> tag child"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Eureka!\n"
       ]
      }
     ],
     "prompt_number": 102
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# You can access the attributes of each node by using .attrs, which gives you a dictionary \n",
      "for links in soup.find_all('a'):\n",
      "    print links.attrs"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{u'href': u'https://sites.google.com/site/forrestbao'}\n",
        "{u'href': u'https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei'}\n",
        "{u'href': u'http://web.stanford.edu/dept/pres-provost/president/speeches/951005dieluft.html'}\n",
        "{u'href': u'https://en.wikipedia.org/wiki/Freedom_of_thought'}\n",
        "{u'href': u'https://en.wikipedia.org/wiki/Academic_freedom'}\n",
        "{u'href': u'https://en.wikipedia.org/wiki/Internet_censorship_in_China'}\n",
        "{u'href': u'https://googleblog.blogspot.com/2010/01/new-approach-to-china.html'}\n"
       ]
      }
     ],
     "prompt_number": 109
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# For more, read the document yourself. "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 76,
       "text": [
        "[u'I have moved my homepage to ',\n",
        " <a class=\"hahaha\" href=\"https://sites.google.com/site/forrestbao\">https://sites.google.com/site/forrestbao</a>,\n",
        " u'.']"
       ]
      }
     ],
     "prompt_number": 76
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 126
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}