Project 2: Scraping eBay

Dilbert comic: a marketing manager says he doesn't see why engineers get paid more than marketing professionals; Dilbert replies that engineers designed and built every important part of modern civilization while all marketing did was misrepresent it; the manager insists 'you need both,' and Dilbert answers 'you really don't.'

ebay-dl.py is a command-line program that scrapes eBay search results. Given a search term, it downloads the first pages of results, extracts fields such as name, price, and shipping, and saves the listings as structured JSON.

Like the Markdown compiler, this program converts data from one format to another. The input is eBay’s HTML, and the output is JSON that another program can use.

Walkthrough (optional). This project is adapted from Mike Izbicki’s course. His roughly two-hour pair-programming video builds most of the code and demonstrates debugging and writing doctests.

Due: Wednesday, October 21 at 11:59pm, see the schedule.

Learning objectives

  1. understand how web scraping works
  2. complete a Python project from scratch
  3. integrate your Python knowledge with what you know about HTML and JSON

Instructions

The project has three parts: write the scraper, run it on a few real searches, and publish the result as a GitHub repository.

Part 1: the scraper. Create a Python file named ebay-dl.py. It should:

  1. Use the argparse library to read a search term from the command line.
  2. Use the requests library to download the first 10 pages of results for that search term.
  3. Use bs4 (Beautiful Soup) to extract every item in the search results.
  4. Build a Python list of the extracted items, where each entry is a dictionary with the following keys:
    1. name: the name of the item.
    2. price: the price of the item in cents, stored as an integer. Never use floats to store monetary values, because floats can’t be represented exactly in computers. If several prices are listed (for example, $54.99 to $79.99), you may pick either one.
    3. status: a string stating the item’s condition, such as "Brand New", "Refurbished", or "Pre-owned".
    4. shipping: the cost of shipping in cents, stored as an integer; if the item has free shipping, this value is 0.
    5. free_returns: a boolean value for whether the item has free returns.
    6. items_sold: the number of items sold, as an integer.
  5. Use the json library to save the list as a JSON file named SEARCH_TERM.json, where SEARCH_TERM is replaced by the search term passed in on the command line.

Note: Not every eBay listing has every field above. For a missing field, keep the associated key and set its value to None.

Note: eBay actively tries to block scrapers, so a bare requests.get may return a “robot check” page instead of real results. Set a browser-like User-Agent header (search the requests docs for how) and pause between requests. You only need a few searches. If a page still comes back blocked, wait and try again or switch networks.

Part 2: run it on real searches. Run your ebay-dl.py file on three search terms of your choice, generating three different JSON files. At least one of these search terms must contain a space (for example, drill press, stuffed animal, or claremont mckenna).

Part 3: publish it. Create a GitHub repository that:

  1. contains your ebay-dl.py file,
  2. contains your three JSON files, and
  3. contains a README.md file explaining:
    1. what your ebay-dl.py file does,
    2. how to run your ebay-dl.py file, using Markdown code block(s) (and not inline code) to show the exact commands that generate the three JSON files in your repo, and
    3. a link to this project page.

Grading rubric

This project is worth 23 points. Your grade starts at 23/23, and you lose the listed points for each checkbox you do not complete.

Parts 2 and 3 (running on three searches and publishing the repo with its README.md) are how you package and submit the project; the points above are earned on the Part 1 criteria, checked against the JSON your published repo produces.

Gradescope runs the scraper against ten instructor-owned HTML fixtures and replaces requests.get with an offline fake. That means everyone receives the same listings, eBay is not contacted during grading, and a temporary robot-check page cannot change your score. The tests check the command-line search term (4 points), ten HTTP requests and their User-Agent (4 points), Beautiful Soup extraction (4 points), JSON schema and types (4 points), and evidence that every downloaded page contributed listings (7 points).

Extra credit

  1. Modify ebay-dl.py so that it accepts a new command-line flag, --csv. Whenever this flag is specified, the output file should be saved in CSV format instead of JSON format.
  2. Generate three CSV files in addition to the three JSON files, and include them in your repo.
  3. Update your README.md file to include instructions and examples for using the --csv flag.

Submission

Push your work to GitHub, then submit your repository and branch to the Project 2 programming assignment on Gradescope. Gradescope grades the exact commit you select and shows the result of each rubric group.

If you completed the CSV extra credit, add this file at the repository root:

# submission.toml
[features]
csv = true

The CSV test runs only when you claim it.