Lab: Rosetta Stone

Due: Wednesday, December 2 at 11:59pm (moved past Thanksgiving break) Worth: 8 points

Starter code: github.com/rtealwitter/lab-formats

The Rosetta Stone carries the same decree in three scripts so that different audiences could read it. Data also needs forms that different programs can read. A running program keeps dicts, lists, and strings in memory, but saving or sending that data requires converting it to text. That conversion is called serialization, and reading the text back in is called parsing.

Several formats can represent the same data, but its structure limits the choices. The same list of tweets can be stored as JSON, YAML, TOML, or CSV. The choice depends on who reads it next: another program, a person editing a config file, or a spreadsheet. This lab moves the tweets and configs from the reading among all four formats and demonstrates the limitations of each.

Getting the code

1. Fork the starter repository, github.com/rtealwitter/lab-formats, to your own account with the Fork button.

2. Clone your fork and open the folder in VS Code:

   $ git clone https://github.com/<your-username>/lab-formats
   $ cd lab-formats

3. Two of the functions use PyYAML, which is not part of the standard library, so install it once:

   $ pip install pyyaml

The json, csv, and tomllib modules are all built in, so there is nothing else to install. (One exception: tomllib only joined the standard library in Python 3.11, so if you are on 3.10 or earlier, also run pip install tomli, which is the very same module under its old name. The starter imports whichever one you have.)

The file is lab_formats.py, about ten short functions with blank bodies and doctests. The loop is the same as every doctest lab: read a function’s doctests, write its body, run python3 -m doctest lab_formats.py, and repeat until the command falls silent.

JSON, both directions

Start with JSON. from_json takes a JSON string and returns Python objects; to_json takes Python objects and returns a JSON string. The reading called them “load string” and “dump string”; the s on the end marks the string versions. json.loads and json.dumps work on strings; json.load and json.dump (no s) work on open files. Choose the function based on whether the input is a string or an open file.

Your to_json should pass indent=2. Indented JSON makes the structure of a response easier to inspect. Pass sort_keys=True as well, so the output does not depend on the order the dict happened to be built in, which is what makes the doctest deterministic:

>>> print(to_json({'username': 'Trump', 'text': 'hello'}))
{
  "text": "hello",
  "username": "Trump"
}

roundtrip_json checks whether the two functions are inverses: dump an object to text, load it back, and compare the result with the original. Most data survives untouched, but the doctest also pins one case that does not. JSON object keys are always strings, so an integer key like 1 comes back as the string '1' and the round-trip is no longer equal. count_keys returns the number of top-level keys in a dict. A value that is itself a list still counts as one key.

YAML for humans

JSON requires double quotes on every key, does not support comments, and rejects trailing commas. YAML is designed to be edited by people, and the GitHub Actions workflow grading this lab is written in it. yaml_to_obj calls yaml.safe_load, which returns the same ordinary dicts and lists as json.loads:

>>> yaml_to_obj('title: My Blog') == {'title': 'My Blog'}
True

Always safe_load, never plain load: the unsafe version can build arbitrary Python objects out of a document, which is a security hole the moment the document came from someone else.

The Norway problem. YAML infers the type of every bare value, and it sometimes guesses wrong. The famous case is that the bare word no is read as the boolean False, so a spreadsheet of country codes silently turns Norway’s code NO into False with no error to warn you. norway_value tests this behavior: it parses country: no and returns the value YAML built.

>>> yaml_to_obj('country: no')
{'country': False}

To inspect the types YAML constructed, run print(to_json(config)).

Meme: yaml.safe_load('country: no') returns {'country': False}. YAML interpreted the name of a country as a Boolean.

TOML for config

TOML uses stricter type rules than YAML. Many Python projects store configuration in a pyproject.toml file. Python 3.11 added a reader to the standard library, tomllib, and it behaves like json.load. toml_get parses a TOML string with tomllib.loads and returns one top-level value:

>>> pyproject = '''
... name = "introcs"
... version = "1.0"
... port = 8080
... '''
>>> toml_get(pyproject, 'port')
8080

The standard library only reads TOML; writing it back out needs a third-party package. tomllib takes types exactly as written, so unlike YAML there is no Norway problem: no stays the string it looks like. Values under a [section] header sit one level deeper in the returned dict. toml_get fetches only top-level keys.

CSV for tables

The other three formats can all nest, holding a list inside a dict inside a list as deep as you like. Plenty of data is not nested at all: a flat table of rows and columns, the kind of thing that belongs in a spreadsheet. CSV represents a flat table, and three functions cover it. csv_to_rows reads CSV text into a list of dicts with csv.DictReader, one dict per row keyed by the header, which is exactly the shape our tweets already have:

>>> tweets = '''text,username
... hello,Trump
... world,Obama'''
>>> csv_to_rows(tweets) == [
...     {'text': 'hello', 'username': 'Trump'},
...     {'text': 'world', 'username': 'Obama'}]
True

rows_to_csv goes the other way, turning a list of row-dicts back into CSV text with csv.DictWriter. Writing CSV is extra credit on Project 2, where you save the list of scraped listings as a spreadsheet instead of JSON. json_to_csv_rows parses a JSON array of flat records into the row form CSV expects.

CSV has no types: every value comes back as a string, so the number 8080 arrives as '8080'. It also cannot nest, so it cannot replace JSON for structured data. To store a list of hashtags in CSV, you must flatten it into one cell or use a format such as JSON instead.

🖼️ Meme: Excel opens your gene-list CSV and helpfully rewrites the gene SEPT2 as a calendar date. The Norway problem has cousins in every corner of the type system.

Files and databases

The docchat and agent projects send API requests as JSON and parse JSON responses. Loading a JSON or CSV file requires memory proportional to its size. A database can filter rows on disk without loading the full dataset. The next class introduces SQL for writing those queries.

Submitting

Work one function at a time, and run the tests from your terminal as you go:

$ python3 -m doctest lab_formats.py

Silence means every test passed; while a test is failing, the output shows exactly what it Expected and what it Got, so you always know which function to fix next. Your fork’s GitHub Action runs the same doctests on every push, so commit and push once the file is silent:

$ git add lab_formats.py
$ git commit -m 'complete lab_formats.py'
$ git push

Enable Actions on your fork and use its result as preliminary feedback. When it is green, submit the repository and branch to the Gradescope Programming Assignment. Gradescope runs instructor-owned pytest cases against lab_formats.py, including unseen JSON, YAML, TOML, and quoted CSV values, and its result is authoritative. There is no partial credit, so fix, push, and resubmit until those tests pass.