Lab: Web Data: Extract, Analyze, Visualize

Due: Wednesday, October 14 at 11:59pm · Worth: 16 points

Which phrases from a merchandise catalog appear most often in an archive of Trump’s tweets? We will extract the catalog’s product names, choose phrases to investigate, and make a chart of their frequency. First we’ll check our extractors on a five-item page that we can read by hand.

The catalog is fictional. The tweet archive contains some repeated records; count each record as stored. Our results describe this dataset, so they cannot establish merchandise sales or public opinion. The extractors and tweet analysis earn 8 points each, independently.

Start

Create lab-web-data from the starter template, clone your copy, and open it in VS Code. See course setup for help with VS Code and Git, and set up a Python environment in this repository. Then install its dependencies:

$ python -m pip install -r requirements.txt

A starter ZIP is also available. Unzip it, open the inner lab-web-data folder, and use Initialize Repository and Publish to GitHub in VS Code. Keep your work in this one repository.

Check a Small Page

Open sample_listings.html in your browser and editor. Find the first item’s title, price, and link in both views. These are the fields your program will collect.

Create one scratch file, inspect_listings.py, to try selectors before putting them into functions. Use the reading’s saved-page example to load sample_listings.html and create a BeautifulSoup object. Print the text of every .price match. Check the results against the five prices in the browser.

For each selector below, predict how many tags it matches before running len(soup.select(selector)). Explain any difference by finding the matching tags in the HTML.

.item
.badge
.free-shipping
.sponsored
#results > li
.item + .item
li.item > a

Two useful checks: does .sponsored match only listings? Does the first item have a preceding sibling? Keep using this scratch file when a selector surprises you.

Build the Extractors

Complete the eight functions in lab_selectors.py. Their docstrings give the inputs, expected results, and examples you can run:

$ python -m doctest lab_selectors.py

Run this after each function. No output means its published examples pass. The reading covers text, attributes, and CSS selectors; use those operations to work out each function.

Watch the return types and whitespace:

select_texts preserves each tag’s text, including surrounding spaces. extract_titles strips surrounding spaces.
extract_prices_cents returns integers: '$49.99' becomes 4999. The starter has a conversion hint.
first_link returns None when there is no link.

Keep Each Item Together

In extract_listings, loop over the .item tags and search inside each item for its title and price. For example, item.select('.title') searches only that item’s descendants. Searching the whole page again inside the loop could pair one item’s title with another item’s price.

Use inspect_listings.py to call your function on the sample page. It should return five dictionaries; the first should be:

{'name': 'Mechanical Keyboard, RGB, Brown Switches', 'price_cents': 4999, 'url': '/itm/101'}

Compare the last dictionary with the last item in the browser, too. In Project 2, you will reuse this pattern on eBay listings, with price in place of price_cents and additional fields for condition, shipping, returns, and sales.

Choose Catalog Phrases

Change the scratch file’s input to sample_merchandise.html and print the names from your extracted records. Pick at least three phrases you want to look for in the tweet archive. The catalog suggests wall, America, jobs, freedom, MAGA, swamp, mainstream media, and border security; you may choose other phrases.

Keep the five required searches as well: Obama, Trump, Mexico, Russia, and Fake News. Before counting, predict which of your phrases will occur most often. If your extractor is unfinished, read the catalog in your browser and continue with the analysis. The scratch-file checks and catalog inspection are practice, with no additional points or files to submit.

Get the Archive

The tweet archive supplies ten master_*.json.zip files for 2009 through 2018. Save download_tweets.py inside lab-web-data, open it to see which files it fetches, and run:

$ python download_tweets.py

The helper downloads a fixed snapshot and unzips it. Check that master_2009.json through master_2018.json are directly inside your repository folder. You can also download and unzip those files yourself; use the full master_ files. If a download fails, rerun the helper. It replaces the same files.

Count the Phrases

Write lab_tweets.py from scratch. Build and check one step at a time:

Load the ten JSON files and combine their lists of tweets. The reading’s JSON archive example shows how to load records from a file. Print the total; this snapshot contains 35402 records.
Inspect one record before choosing its message field. Use full_text when present, otherwise text.
Count tweets containing each required phrase and your three choices, ignoring capitalization. Lowercase both the phrase and message before using in. Count a tweet once per phrase, even if it repeats that phrase. Print each phrase’s count.
Check two results: Trump occurs in 13659 records and Obama in 2738. If yours differ, check the input files, message field, and capitalization before continuing.
Calculate each phrase’s percentage of all records. Use your program’s counts and total so the calculation also works on other data.

A tweet can contain several of the phrases. What does that mean for the sum of their percentages?

Print or save a Markdown table, with right-aligned phrases and percentages showing two digits before and after the decimal:

| phrase            | percent of tweets |
| ----------------- | ----------------- |
|              daca | 00.05             |
|         fake news | 00.88             |
|  mainstream media | 00.03             |
|            mexico | 00.45             |
|             obama | 07.73             |
|            russia | 00.91             |
|             trump | 38.58             |
|              wall | 00.68             |

The optional format_table.py experiment compares f-string alignment and number formats. For example, f'{7.73:05.2f}' produces '07.73'. Use the values you calculate for your own phrases.

Plot and Explain the Results

Use matplotlib to make a bar chart of your phrases and their counts or percentages. We have not covered plotting in class; use the bar-chart documentation to learn the needed arguments. Choose bar or barh, label the phrases, and save the figure with plt.savefig('tweets.png'). plt.show() displays a window but does not save a file for submission.

Put your program’s Markdown table and saved image in README.md, with a short description of what they show. Compare the result with your prediction. One sentence of description is enough.

Run python lab_tweets.py from the repository folder, open the saved image, and check that the README displays both the table and image. Gradescope also runs your script on different tweets using the same ten filenames, so every count, percentage, and bar must come from the input files. Use Python’s standard library and matplotlib for this script.

Optional Exploration (Ungraded)

The archive maintainer links a successor archive. Try newer data or investigate a field other than the message text. Keep the ten-file input interface working for your graded submission. These explorations earn no additional points.

Submit

Commit and sync lab_selectors.py, lab_tweets.py, README.md, and the saved plot image. On Gradescope, choose GitHub and submit your repository and branch to lab-web-data. You may resubmit as you finish more steps. The first lab explains the commit, sync, and submission workflow.