Project 2: Scraping eBay

ebay-dl.py is a command-line program that scrapes eBay search results. Given a search term, it downloads the first pages of results, extracts fields such as name, price, and shipping, and saves the listings as structured JSON.

Like the Markdown compiler, this program converts data from one format to another. The input is eBay’s HTML, and the output is JSON that another program can use.

An optional roughly two-hour pair-programming video builds most of the code and demonstrates debugging and writing doctests.

Due: Wednesday, October 21 at 11:59pm, see the schedule.

Start

Create an empty GitHub repository named ebay-scraper under your account, clone it, and do all project work inside it.

Learning Objectives

1. understand how web scraping works
2. complete a Python project from scratch
3. integrate your Python knowledge with what you know about HTML and JSON

Build the Scraper

The project has three parts: write the scraper, run it on a few real searches, and publish the results.

Part 1: the scraper. Create a Python file named ebay-dl.py. It should:

• Use the argparse library to read a search term from the command line.
• Use the requests library to download the first 10 pages of results for that search term.
• Use bs4 (Beautiful Soup) to extract every item in the search results.
• Build a Python list of the extracted items, where each entry is a dictionary with these keys:
  • name: the name of the item.
  • price: the price of the item in cents, stored as an integer. Never use floats to store monetary values, because floats can’t be represented exactly in computers. If several prices are listed (for example, $54.99 to $79.99), you may pick either one.
  • status: a string stating the item’s condition, such as "Brand New", "Refurbished", or "Pre-owned".
  • shipping: the cost of shipping in cents, stored as an integer; if the item has free shipping, this value is 0.
  • free_returns: a boolean value for whether the item has free returns.
  • items_sold: the number of items sold, as an integer.
• Use the json library to save the list as a JSON file named SEARCH_TERM.json, where SEARCH_TERM is replaced by the search term passed in on the command line.

Note: Not every eBay listing has every field above. For a missing field, keep the associated key and set its value to None.

Note: eBay actively tries to block scrapers, so a bare requests.get may return a “robot check” page instead of real results. Set a browser-like User-Agent header (search the requests docs for how) and pause between requests. You only need a few searches. If a page still comes back blocked, wait and try again or switch networks.

Part 2: run it on real searches. Run your ebay-dl.py file on three search terms of your choice, generating three different JSON files. At least one of these search terms must contain a space (for example, drill press, stuffed animal, or claremont mckenna).

Part 3: publish it. Your GitHub repository must contain:

• your ebay-dl.py file;
• your three JSON files; and
• a README.md file with:
  • an explanation of what ebay-dl.py does;
  • instructions for running ebay-dl.py, using Markdown code block(s), not inline code, to show the exact commands that generate the three JSON files in your repo; and
  • a link to this project page.

Grading Rubric

This project is worth 23 points. Your grade starts at 23/23, and you lose the listed points for each checkbox you do not complete.

• Use the argparse library to pass information from the command line (4 pts)
• Use the requests library to download the webpage (4 pts)
• Use bs4 to extract the items from the webpage (4 pts)
• Output a JSON file (4 pts)
• Extract items from every page you downloaded (7 pts)

Parts 2 and 3 (running on three searches and publishing the repo with its README.md) are how you package and submit the project; the points above are earned on the Part 1 criteria, checked against the JSON your published repo produces.

Gradescope runs the scraper against ten instructor-owned HTML fixtures and replaces requests.get with an offline fake. That means everyone receives the same listings, eBay is not contacted during grading, and a temporary robot-check page cannot change your score. The tests check the command-line search term (4 points), requests for result pages 1 through 10 and their User-Agent (4 points), Beautiful Soup extraction (4 points), JSON schema and types (4 points), and evidence that every downloaded page contributed listings (7 points). The fixtures include multiple listings per page, missing fields, free shipping, and a price range, following the rules above. Separate feedback checks remind you about the submitted search files and README without changing the 23-point rubric.

Extra Credit

• Add a CSV option (+1 point)

1. Modify ebay-dl.py so that it accepts a new command-line flag, --csv. Whenever this flag is specified, the output file should be saved in CSV format instead of JSON format.
2. Generate three CSV files in addition to the three JSON files, and include them in your repo.
3. Update your README.md file to include instructions and examples for using the --csv flag.

Submit

If you completed the CSV extra credit, add this file at the repository root:

# submission.toml
[features]
csv = true

The CSV test runs only when you claim it.

Commit and sync ebay-dl.py, README.md, and the three JSON files. On Gradescope, choose GitHub and submit your ebay-scraper repository and the branch containing your commit.