Lab: Selector Golf

Due: Wednesday, October 14 at 11:59pm (one week after it is assigned) Worth: 8 points

This is the second of this week’s two labs; the other is Analyzing Trump Tweets. That lab covers JSON data that already arrives as lists and dictionaries. This one uses Beautiful Soup and CSS selectors to extract data from an HTML page.

The selectors from the CSS reading, such as div > .price and a[href], can also extract data from pages you did not write. In this lab, you will write the function that Project 2 uses to turn a page of listings into a list of dictionaries.

Download: lab_selectors.py and the practice page sample_listings.html

From coloring to counting

Every function in this lab begins by passing an HTML string to BeautifulSoup, which returns a soup object that .select can search:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
soup.select('.price')      # a list of every tag matching the selector

.select takes a CSS selector string, the same kind you wrote in your stylesheets, and returns a list of every tag that matches, top to bottom. The selector chooses which tags, and you loop over the resulting list to extract the values you want. Beautiful Soup uses the CSS selector syntax unchanged: .select('div > a') matches the tags that div > a { ... } would style.

Download sample_listings.html into the same folder as your code. It is a small, made-up search-results page with five items. Each has a title, price, condition, and link, and three advertise free shipping. It represents the eBay page you will scrape in Project 2, but is small enough to inspect in full. Open it in your browser and in your editor, and keep it in front of you for the rest of the lab.

Making soup

Read the file into a string and parse it:

>>> from bs4 import BeautifulSoup
>>> html = open('sample_listings.html').read()
>>> soup = BeautifulSoup(html, 'html.parser')

The first line uses open(...).read() from the files unit, so html is an ordinary str. Use .select to find all the prices, then read each one’s text:

>>> for tag in soup.select('.price'):
...     print(tag.text)
...
$49.99
$12.50
$8.99
$23.00
$34.95

.select found five prices in page order. .text pulled the words from between each tag’s opening and closing.

Text inside HTML tags may include whitespace, which you will handle below with .strip().

Selector golf

Before extracting data, predict how many tags a selector matches and then check your answer. In the CSS unit, you checked with document.querySelectorAll in the browser console. For this lab, use len in Python:

>>> len(soup.select('.price'))
5

Play a round of selector golf on sample_listings.html. For each selector below, predict the count first, then run len(soup.select(...)) and see if you were right. No peeking at the page’s source until you have committed to a number.

soup.select('.item')            # every listing
soup.select('.badge')           # every badge, of any kind
soup.select('.free-shipping')   # just the free-shipping badges
soup.select('.sponsored')       # careful with this one
soup.select('#results > li')    # direct children of the results list
soup.select('.item + .item')    # each item that follows another item
soup.select('li.item > a')      # the title link inside each item

.sponsored matches two elements, not one: the sponsored <li> and the <span class="badge sponsored"> inside it, because a class can live on any tag and this page happens to use the name twice. .item + .item matches four, not five, because the first item has no preceding item. If your prediction missed either, review how class selectors and adjacent-sibling selectors work.

The first two functions in lab_selectors.py use this pattern. count_matches returns len(soup.select(selector)), and select_texts returns the .text of every match.

From tags to fields

Scraping extracts data inside tags. Two operations handle most fields. tag.text gives you the words between a tag’s opening and closing, and indexing with a key like tag['href'] gives you an attribute’s value:

>>> link = soup.select('a.title')[0]
>>> link.text
'Mechanical Keyboard, RGB, Brown Switches'
>>> link['href']
'/itm/101'

select_attr, extract_titles, and first_link use these two operations. Real pages put newlines and indentation inside tags, so tag.text often includes surrounding spaces. The extractors in this lab call .strip() to remove them.

Prices need one extra step, which Project 2 will grade. HTML represents a price as the string '$49.99', but you must store money as a whole number of cents, as an int, never a float. Convert '$49.99' to the integer 4999: strip the $, and turn the dollars into cents.

>>> dollars = '$49.99'.replace('$', '')
>>> int(round(float(dollars) * 100))
4999

extract_prices_cents returns a list of integers.

One dict per item

Represent each listing as a dictionary containing its name, price, and link. The page becomes a list of those dictionaries.

Loop over the items and call .select inside each item to find its title and price:

soup = BeautifulSoup(html, 'html.parser')
listings = []
for item in soup.select('.item'):
    title = item.select('.title')[0]
    price = item.select('.price')[0]
    listings.append({
        'name': title.text.strip(),
        'price_cents': ...,     # the price as an int of cents
        'url': title['href'],
    })

item.select('.title') searches within one listing, so the loop does not mix fields from different items. extract_listings uses this pattern:

>>> extract_listings(open('sample_listings.html').read())
[{'name': 'Mechanical Keyboard, RGB, Brown Switches', 'price_cents': 4999, 'url': '/itm/101'}, ...]

Project 2 uses this same pattern with additional fields and pages.

The eBay project adds status, shipping, free_returns, and items_sold to each dictionary, processes ten pages instead of one string, and saves the finished list with the json library. The extraction pattern remains the same: select the items, loop over them, extract each field with .text and [...], build a dictionary, and collect the dictionaries. count_free_shipping practices one of the extra fields.

Submitting

This lab is auto-graded. Open lab_selectors.py, fill in one function body at a time, and run the doctests from your terminal until the command produces no output:

$ python3 -m doctest lab_selectors.py

When a test fails, the command prints the example with its Expected and Got values. Add -v to display passing tests too. Fix each failure and rerun the tests.

Commit the finished file and push it to GitHub:

$ git add lab_selectors.py
$ git commit -m 'complete lab_selectors.py'
$ git push

Your repository runs the doctests automatically on every push. That action is preliminary feedback, not the grade. When it is green, submit the repository and branch to the Gradescope Programming Assignment. Gradescope runs instructor-owned pytest cases against lab_selectors.py, including unseen HTML, and its result is authoritative. Fix, push, and resubmit until those tests pass.