Web Scraping

For the last few classes, our programs read strings and local files. Web scraping downloads a page as a string and extracts the parts we want.

Web scraping combines four previous topics: HTML represents pages, CSS selectors identify parts of a page, file operations read and write data, and exceptions handle failures.

Some sites alter their markup or block requests to resist scraping.

CommitStrip comic titled 'Data Wars': one team of developers plans to scrape a site by editing headers, using PhantomJS, and routing calls through different routes; the opposing team plans to stop them by randomly changing the markup, creating honey pots, and adding captchas.

In this course, we will scrape public pages that permit ordinary access.

Getting a page from the web

The requests library downloads web pages in Python. It is not built in, so install it once from the terminal:

$ pip3 install requests

Pass a URL to requests.get, then read the page from the response’s .text attribute:

>>> import requests
>>> response = requests.get('https://www.gutenberg.org/files/345/345-h/345-h.htm')
>>> type(response.text)
<class 'str'>

That URL points to the full text of Dracula on Project Gutenberg, and response.text is a str containing the page’s HTML. Both response.text and open(filename).read() return strings, so they support the same operations: len, .lower(), .find, slicing, and in.

When the request fails

Network requests can fail because a page has moved, a server is down, or the Wi-Fi connection drops. Use try/except to handle these failures.

Every HTTP response carries a status code, a number that reports the request’s outcome. Read it from the .status_code attribute:

>>> response = requests.get('https://www.gutenberg.org/files/345/345-h/345-h.htm')
>>> response.status_code
200
>>> response = requests.get('https://www.gutenberg.org/nonexistent-page')
>>> response.status_code
404

200 means the request succeeded; 404 means the server has no such page. In general, codes in the 200s indicate success, 300s indicate redirects, 400s indicate a problem with the request, and 500s indicate a server error. MDN lists all HTTP status codes.

Check the status code before using the response text. Wrap the request in try/except so one failed request cannot stop a program that processes many pages:

try:
    response = requests.get(url)
    if response.status_code == 200:
        text = response.text
    else:
        print('bad status:', response.status_code)
except requests.exceptions.RequestException as e:
    print('request failed:', e)

The program uses the response text only when the status is 200 and prints a warning for other statuses. If the request cannot complete because of a missing network connection or bad hostname, the except block handles the exception. This pattern lets a scraper continue after one of many requests fails.

A command-line tool

To run a program on different pages without editing its code, accept the URL as input. The built-in argparse library reads values off the command line for you.

The following small wget program gets a page from the web and saves it to a file. Define its two command-line arguments with argparse:

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--url')
parser.add_argument('--filename')
args = parser.parse_args()

After parse_args, args.url and args.filename hold whatever the user typed. Use those values with requests and the open/write pattern from the files topic:

import requests
response = requests.get(args.url)

with open(args.filename, 'w', encoding='utf-8') as f:
    f.write(response.text)

open(..., encoding='utf-8') writes the response text with the encoding used in the files topic. Run the program from the terminal:

$ python3 wget.py --url https://www.gutenberg.org/files/345/345-h/345-h.htm --filename dracula.html

Loading and processing text

Programs in this unit separate loading text from processing it. A string literal, a file (open), the command line (argparse), and the web (requests) can all provide a plain str.

Choose a loader based on where the text lives and a processor based on what you want from it. Because those choices are independent, loaders and processors can be combined. Two processors are useful here: JSON for structured lists and dictionaries, and Beautiful Soup for web pages.

JSON

Web servers often return structured data for programs: lists of records with the same fields. The standard text format for structured data is JSON, which resembles Python lists and dictionaries written as text:

jsontext = '''
[
    { "text": "hello", "username": "Trump" },
    { "text": "world", "username": "Obama" },
    { "text": "hola",  "username": "Obama" },
    { "text": "mundo", "username": "Trump" }
]
'''

That string is a list of four records, each a dictionary with a "text" and a "username". The built-in json library parses that text with json.loads (“load string”):

>>> import json
>>> data = json.loads(jsontext)
>>> type(data)
<class 'list'>
>>> data[0]
{'text': 'hello', 'username': 'Trump'}
>>> data[0]['username']
'Trump'

json.loads returns an ordinary Python list of dictionaries, which we index and loop over like any other. In this week’s Analyzing Trump Tweets lab, you’ll load JSON files holding every tweet Trump sent from 2009 to 2018, join them into one list, and count how often words like Obama and Russia appear.

Obama, OBAMA, and obama are three different strings. Lower-case text before searching to count every variant:

>>> 'obama' in 'talking about OBAMA today'.lower()
True

The lab uses this case-insensitive check for its word counts.

Beautiful Soup

The CSS selectors used to style pages, such as div > .price and a[href], can also extract data from pages you did not write.

A downloaded page is an HTML string whose tags form the document tree from the CSS lesson. The Beautiful Soup library (imported from bs4) parses that tree and selects tags with CSS selectors. It is not built in, so we install it once:

$ pip3 install bs4

Pass BeautifulSoup the HTML string and the name of a parser to create a searchable soup object. Its .select method takes a CSS selector string and returns a list of every matching tag:

>>> from bs4 import BeautifulSoup
>>> html = '<div><b>this is HTML</b> inside a <em>Python <b>String!</b></em></div>'
>>> soup = BeautifulSoup(html, 'html.parser')
>>> soup.select('b')
[<b>this is HTML</b>, <b>String!</b>]

soup.select('b') found both <b> tags at any depth, as the CSS type selector b would. The selectors from the CSS lesson work here unchanged: .price for a class, #header for an id, div > a for a direct child, and a[href] for an attribute.

After .select finds tags, .text returns the text between a tag’s opening and closing, and a key such as tag['href'] returns an attribute’s value. For example, consider this HTML from an online store:

>>> html = '''
... <div class="product">
...     <a href="/item/123">Wireless Mouse</a>
...     <span class="price">$19.99</span>
... </div>
... <div class="product">
...     <a href="/item/456">Mechanical Keyboard</a>
...     <span class="price">$49.99</span>
... </div>
... '''

We select the prices by class and read each one’s .text:

>>> soup = BeautifulSoup(html, 'html.parser')
>>> for tag in soup.select('.price'):
...     print(tag.text)
...
$19.99
$49.99

We select the links and read both their text and their href:

>>> for link in soup.select('div.product > a'):
...     print(link['href'], link.text)
...
/item/123 Wireless Mouse
/item/456 Mechanical Keyboard

.select chooses the tags, .text and ['href'] extract their values, and a for loop processes the list. The scraping project uses this loop to extract names and prices from an eBay results page. Automate the Boring Stuff’s chapter on web scraping documents more uses of requests and bs4.

Counting matches, again

Before scraping a page, predict how many HTML tags a selector will match. The CSS unit used document.querySelectorAll in the browser console; in Python, use len:

>>> len(soup.select('.price'))
2

Predict the count, then run len(soup.select(...)) and see if you were right. The two practice files in this folder, practice quiz 1 and practice quiz 2, each contain HTML and selectors that print their match counts. You can also read the counts in the answers. Check two details: a space between selectors means any descendant, while > means a direct child; adding result lists with + keeps duplicates that a single grouped selector would merge.

Is scraping legal?

In hiQ v. LinkedIn, U.S. courts held that scraping data that is already public, with no login and no password, is generally not a violation of the main federal anti-hacking law, the Computer Fraud and Abuse Act. The Electronic Frontier Foundation, which argued that position, explains the case.

A public page does not provide blanket permission to scrape it. A site’s terms of service can forbid scraping as a matter of contract, copyright governs how you use collected material, and thousands of rapid requests can overload a server and deny service to other users. Follow three rules: scrape public pages rather than private ones, go slowly, and don’t republish someone else’s data as your own. The lab and project use public pages at a limited request rate.

Combining the tools

The web-scraping project combines these tools: argparse reads a search term, requests downloads the eBay results, bs4 extracts each item’s name and price, and json saves the list to a file.

This week’s Analyzing Trump Tweets lab focuses on JSON. Many of its loops iterate over a list, test each item, and keep a running count. Later in the term, we will learn shortcuts for writing this pattern in one line.