Project 2: Scraping eBay

Due: Wednesday, October 21 at 11:59pm · Worth: 23 points (+1 extra credit; maximum 24/23)

ebay-dl.py is a command-line program that scrapes eBay search results. Given a search term, it downloads the first pages of results, extracts fields such as name, price, and shipping, and saves the listings as structured JSON.

Like the Markdown compiler, this program converts data from one format to another. The input is eBay’s HTML, and the output is JSON that another program can use.

An optional roughly two-hour pair-programming video builds most of the code and demonstrates debugging and writing doctests.

Start

Create an empty GitHub repository named ebay-scraper under your account, clone it, and do all project work inside it. In this repository, set up its Python environment, then install the two libraries:

$ python -m pip install requests beautifulsoup4

Learning Objectives

• Download an HTML page and extract selected fields from it.
• Build and test a Python command-line program from scratch.
• Save scraped records as JSON and load them back into Python.

Build the Scraper

Start with a scraper that can save one search, then use it to collect three searches someone else can reproduce.

The scraper. Create a Python file named ebay-dl.py. It should:

• Read a search term from the command line. Use argparse as in the reading, with a named option such as --search; args.search then holds the search term.
• Use the requests library to download the first 10 pages of results for that search term.
• Use bs4 (Beautiful Soup) to extract every item in the search results.
• Build a Python list of the extracted items, where each entry is a dictionary with these keys:
  • name: the name of the item.
  • price: the price of the item in cents, stored as an integer. Never use floats to store monetary values, because floats can’t be represented exactly in computers. If several prices are listed (for example, $54.99 to $79.99), you may pick either one.
  • status: a string stating the item’s condition, such as "Brand New", "Refurbished", or "Pre-owned".
  • shipping: the cost of shipping in cents, stored as an integer; if the item has free shipping, this value is 0.
  • free_returns: a boolean value for whether the item has free returns.
  • items_sold: the number of items sold, as an integer.
• Use the json library to save the list as a JSON file named SEARCH_TERM.json, where SEARCH_TERM is replaced by the search term passed in on the command line.

Note: Not every eBay listing has every field above. For a missing field, keep the associated key and set its value to None.

Start with one page before adding the ten-page loop. Pass the search term in the _nkw query parameter and the page number in _pgn, for example https://www.ebay.com/sch/i.html?_nkw=garden+tools&_pgn=1. Set a browser-like User-Agent header and a timeout on each request, and pause between pages. Inspect both the HTTP status and the returned HTML: a successful HTTP response can still contain a robot check instead of listings. If you receive HTTP 403, HTTP 429, a robot check, or HTML without listing cards, stop live requests and use the practice pages below. An empty list does not establish that your extraction works.

Practice Pages When eBay Blocks Requests

Download practice-pages.zip into your repository and extract it:

$ python -m zipfile -e practice-pages.zip .

The practice-pages folder contains page-1.html through page-10.html. These are artificial listings, with known prices, missing fields, and distinct items on every page. Open page-1.html in your browser and editor to inspect the listing structure.

Separate downloading a page from extracting its listings. Your extraction function should take an HTML string so you can give it either response.text or the contents of a saved file. Add an optional --pages-dir argument (the named-argument example shows how argparse handles flags): when supplied, read page-1.html through page-10.html from that folder as UTF-8 instead of making requests. When omitted, keep the normal ten-page requests download; Gradescope exercises that default interface with its own offline responses.

$ python ebay-dl.py "garden tools" --pages-dir practice-pages

The practice output should contain 30 listings. The first trowel costs 1195 cents and ships free; the first planter costs 1250 or 1800 cents, with 425-cent shipping. The mystery-seed listings retain every key with None for missing values, which JSON writes as null. Check that names from page 10 appear too. Live eBay can change its HTML; keep support for this published practice structure when experimenting with other selectors.

Three saved searches. Run your ebay-dl.py file on three search terms of your choice, generating three different JSON files. At least one of these search terms must contain a space (for example, drill press, stuffed animal, or claremont mckenna). If live access is blocked, run each search with --pages-dir practice-pages and save the resulting practice output instead. The practice listings stay the same for every search term; the filename still comes from the supplied term. State in your README which outputs use artificial practice data and include the exact commands you ran. For an optional investigation, choose a budget and use the saved prices and shipping costs to compare which practice listings fit it. Keep that exploration separate from the required output schema.

A reproducible collection. Your GitHub repository must contain:

• your ebay-dl.py file;
• your three JSON files; and
• a README.md file with:
  • an explanation of what ebay-dl.py does;
  • instructions for running ebay-dl.py, using Markdown code block(s), not inline code, to show the exact commands that generate the three JSON files in your repo; and
  • a link to this project page.

Grading Rubric

This project is worth 23 points. Gradescope checks each item separately and adds the points earned.

• Use the supplied command-line search term in requests and the output filename (4 points)
• Use the requests library to download the webpage (4 points)
• Use bs4 to extract the items from the webpage (4 points)
• Output a JSON file (4 points)
• Extract items from every page you downloaded (7 points)

The three saved searches and README let someone else try your program. The points above come from running your submitted scraper and checking the JSON it produces.

Gradescope runs the scraper against ten instructor-owned HTML fixtures and replaces requests.get with an offline fake. That means everyone receives the same listings, eBay is not contacted during grading, and a temporary robot-check page cannot change your score. The tests check the command-line search term (4 points), requests for result pages 1 through 10 and their User-Agent (4 points), Beautiful Soup extraction (4 points), JSON schema and types (4 points), and evidence that every downloaded page contributed listings (7 points). The fixtures include multiple listings per page, missing fields, free shipping, and a price range, following the rules above. The saved search files and README are self-checks and do not change the 23-point rubric.

Extra Credit

• Add a CSV option (+1 point)

1. Modify ebay-dl.py so that it accepts a new command-line flag, --csv. Whenever this flag is specified, the output file should be saved in CSV format instead of JSON format.
2. Generate three CSV files in addition to the three JSON files, and include them in your repo.
3. Update your README.md file to include instructions and examples for using the --csv flag.

Automatic Grading

All 23 base points and the 1-point CSV bonus are automatic. The grader checks command-line behavior, so either sys.argv or argparse can earn the CLI points. Support python ebay-dl.py "search term" with one positional search term, or a single clearly named --search, --query, or --term option. Do not require subcommands for the graded interface. The grader substitutes ten local HTML pages for network responses. Saved searches and README examples remain practice; their historical provenance is not graded.

Submit

If you completed the CSV extra credit, add this file at the repository root:

# submission.toml
[features]
csv = true

The CSV test runs only when you claim it.

Commit and sync ebay-dl.py, README.md, and the three JSON files. On Gradescope, choose GitHub and submit your ebay-scraper repository and the branch containing your commit.