Lab: Selector Golf

Due: Wednesday, October 14 at 11:59pm (one week after it is assigned) Worth: 8 points

This week’s other lab, Analyzing Trump Tweets, covers JSON data that already arrives as lists and dictionaries. This lab uses Beautiful Soup and CSS selectors to extract data from an HTML page.

The selectors from the CSS reading, such as div > .price and a[href], can also extract data from pages you did not write. In this lab, you will write the function that Project 2 uses to turn a page of listings into a list of dictionaries.

Starter repository: github.com/rtealwitter/lab-selectors

Start

Select Use this template to create an independent repository named lab-selectors under your account, then clone that copy. It contains lab_selectors.py and the practice page sample_listings.html.

From Coloring to Counting

Every function in this lab begins by passing an HTML string to BeautifulSoup, which returns a soup object that .select can search:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
soup.select('.price')      # a list of every tag matching the selector

.select takes a CSS selector string, the same kind you wrote in your stylesheets, and returns a list of every tag that matches, top to bottom. The selector chooses which tags, and you loop over the resulting list to extract the values you want. Beautiful Soup uses the CSS selector syntax unchanged: .select('div > a') matches the tags that div > a { ... } would style.

Open the starter’s sample_listings.html. It is a small, made-up search-results page with five items. Each has a title, price, condition, and link, and three advertise free shipping. It represents the eBay page you will scrape in Project 2, but is small enough to inspect in full. Open it in your browser and in your editor, and keep it in front of you for the rest of the lab.

Making Soup

Read the file into a string and parse it:

>>> from bs4 import BeautifulSoup
>>> html = open('sample_listings.html').read()
>>> soup = BeautifulSoup(html, 'html.parser')

The first line uses open(...).read() from the files unit, so html is an ordinary str. Use .select to find all the prices, then read each one’s text:

>>> for tag in soup.select('.price'):
...     print(tag.text)
...
$49.99
$12.50
$8.99
$23.00
$34.95

.select found five prices in page order. .text pulled the words from between each tag’s opening and closing.

Selector Golf

Before extracting data, predict how many tags a selector matches and then check your answer. In the CSS unit, you checked with document.querySelectorAll in the browser console. For this lab, use len in Python:

>>> len(soup.select('.price'))
5

Play a round of selector golf on sample_listings.html. For each selector below, predict the count first, then run len(soup.select(...)) and see if you were right. No peeking at the page’s source until you have committed to a number.

soup.select('.item')            # every listing
soup.select('.badge')           # every badge, of any kind
soup.select('.free-shipping')   # just the free-shipping badges
soup.select('.sponsored')       # careful with this one
soup.select('#results > li')    # direct children of the results list
soup.select('.item + .item')    # each item that follows another item
soup.select('li.item > a')      # the title link inside each item

.sponsored matches two elements, not one: the sponsored <li> and the <span class="badge sponsored"> inside it, because a class can live on any tag and this page happens to use the name twice. .item + .item matches four, not five, because the first item has no preceding item. If your prediction missed either, review how class selectors and adjacent-sibling selectors work.

The first two functions in lab_selectors.py use this pattern. count_matches returns len(soup.select(selector)), and select_texts returns the .text of every match.

From Tags to Fields

Scraping extracts data inside tags. Two operations handle most fields. tag.text gives you the words between a tag’s opening and closing, and indexing with a key like tag['href'] gives you an attribute’s value:

>>> link = soup.select('a.title')[0]
>>> link.text
'Mechanical Keyboard, RGB, Brown Switches'
>>> link['href']
'/itm/101'

select_attr, extract_titles, and first_link use these two operations. Real pages put newlines and indentation inside tags, so tag.text often includes surrounding spaces. The extractors in this lab call .strip() to remove them.

Prices need one extra step, which Project 2 will grade. HTML represents a price as the string '$49.99', but you must store money as a whole number of cents, as an int, never a float. Convert '$49.99' to the integer 4999: strip the $, and turn the dollars into cents.

>>> dollars = '$49.99'.replace('$', '')
>>> int(round(float(dollars) * 100))
4999

extract_prices_cents returns a list of integers.

One dict per Item

Represent each listing as a dictionary containing its name, price, and link. The page becomes a list of those dictionaries.

Loop over the items and call .select inside each item to find its title and price:

soup = BeautifulSoup(html, 'html.parser')
listings = []
for item in soup.select('.item'):
    title = item.select('.title')[0]
    price = item.select('.price')[0]
    listings.append({
        'name': title.text.strip(),
        'price_cents': ...,     # the price as an int of cents
        'url': title['href'],
    })

item.select('.title') searches within one listing, so the loop does not mix fields from different items. extract_listings uses this pattern:

>>> extract_listings(open('sample_listings.html').read())
[{'name': 'Mechanical Keyboard, RGB, Brown Switches', 'price_cents': 4999, 'url': '/itm/101'}, ...]

The eBay project adds status, shipping, free_returns, and items_sold to each dictionary, processes ten pages instead of one string, and saves the finished list with the json library. The extraction pattern remains the same: select the items, loop over them, extract each field with .text and [...], build a dictionary, and collect the dictionaries. count_free_shipping practices one of the extra fields.

Submit

This lab is auto-graded. Open lab_selectors.py, fill in one function body at a time, and run the doctests from your terminal until the command produces no output:

$ python3 -m doctest lab_selectors.py

When a test fails, the command prints the example with its Expected and Got values. Add -v to display passing tests too. Fix each failure and rerun the tests.

Commit and sync lab_selectors.py, then on Gradescope choose GitHub and submit your lab-selectors repository and the branch containing your commit.