JSON & Alternatives

The scraper from web scraping and the LLM programs behind docchat exchange data with other systems. When an API uses JSON, its response arrives as text that json.loads converts into Python objects.

A running program keeps its data in memory as dicts, lists, numbers, and strings. To save that data in a file or send it to another program, we need a text representation that both programs can parse. JSON (JavaScript Object Notation) is a common format for exchanging structured data between programs.

The shape of JSON

JSON looks familiar because its text representation resembles Python’s dicts and lists. This JSON document holds a list of tweets, each with text and username fields:

[
    {"text": "hello", "username": "Trump"},
    {"text": "world", "username": "Obama"},
    {"text": "hola",  "username": "Obama"}
]

The square brackets are a list (JSON calls it an array), each pair of curly braces is a dict (JSON calls it an object), and the keys and text are strings in double quotes. The mapping to Python is direct:

• an object {...} becomes a Python dict
• an array [...] becomes a Python list
• a string, always in double quotes, becomes a str
• a number becomes an int or a float
• true and false become True and False
• null becomes None

Python will refuse to read JSON that violates these rules:

• strings and keys use double quotes, never single
• the booleans and null are lowercase: true, not True
• there is no trailing comma after the last item
• there are no comments anywhere

Reading JSON

In our programs, JSON arrives as a string containing a web response body or a file’s contents. json.loads (“load string”) converts that string into Python objects:

>>> import json
>>> jsontext = '''
... [
...     {"text": "hello", "username": "Trump"},
...     {"text": "world", "username": "Obama"},
...     {"text": "hola",  "username": "Obama"}
... ]
... '''
>>> data = json.loads(jsontext)
>>> type(data)
<class 'list'>
>>> data[0]
{'text': 'hello', 'username': 'Trump'}
>>> data[0]['username']
'Trump'

json.loads returns an ordinary Python list of dicts, which we can index, look up by key, and loop over. Python prints the strings with single quotes because the JSON double-quote rule applies only to the text representation.

The tweets dataset from web scraping uses a folder of master_*.json files, each holding a list of tweets. After loading a file, we can count tweets from one account with Python:

>>> sum(1 for tweet in data if tweet['username'] == 'Obama')
2

When the JSON lives in a file instead of a string, use json.load (no s) on an open file, the same way we read text files in strings and files:

with open('tweets.json') as f:
    data = json.load(f)

loads reads a string; load reads an open file.

Writing JSON

json.dumps (“dump string”) turns Python objects into a JSON string:

>>> config = {'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}
>>> print(json.dumps(config, indent=2))
{
  "title": "My Blog",
  "port": 8080,
  "debug": true,
  "tags": [
    "python",
    "web"
  ]
}

The indent=2 argument produces a readable block. Without it, json.dumps produces compact JSON for storage or transmission.

When you are unsure of the shape of nested data, print it as indented JSON:

print(json.dumps(data, indent=2))

json.dumps accepts nested combinations of dicts, lists, strings, numbers, booleans, and None.

To write to a file instead of a string, use json.dump (no s) with an open file:

with open('config.json', 'w') as f:
    json.dump(config, f, indent=2)

dumps returns a string; dump writes to an open file. Python’s None becomes JSON null:

>>> print(json.dumps({'author': None}))
{"author": null}

JSON in APIs

When docchat asks an LLM a question, the API’s response body is JSON. After the client library parses it, the model’s answer is one string inside nested objects and arrays:

reply = response['choices'][0]['message']['content']

The docchat and agents projects send JSON to a model and read JSON responses. The Twitter clone’s backend will also exchange JSON with the browser.

YAML

JSON has no comments, requires double quotes on every key, and rejects trailing commas. Hand-edited configuration files often use YAML instead.

The same config in YAML is:

# a comment, at last
title: My Blog
port: 8080
debug: true
tags:
  - python
  - web

YAML uses one key: value per line, indentation for nesting, and - for list items. It also permits # comments. Loading it into Python needs a third-party library, pyyaml, and the function is yaml.safe_load:

>>> import yaml
>>> config = yaml.safe_load(open('config.yaml'))
>>> config
{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

JSON syntax is also valid YAML 1.2, which is the joke in this meme:

The GitHub Actions file that runs your labs’ preliminary public checks is YAML, as are Docker Compose and Kubernetes configuration files. Gradescope runs the separate instructor-owned tests that determine the grade. YAML infers the types of unquoted values, which can produce unexpected results. For example, the bare word no is read as the boolean False:

>>> yaml.safe_load('country: no')
{'country': False}

This is known as the Norway problem: a file listing country codes can turn the code NO into False without an error. To inspect YAML’s inferred values, load the file and run print(json.dumps(config, indent=2)).

TOML

TOML (Tom’s Obvious Minimal Language) is a configuration format with stricter type rules than YAML. In TOML, the config is:

title = "My Blog"
port = 8080
debug = true
tags = ["python", "web"]

TOML uses key = value, quotes strings, and groups related keys under [section] headers when needed. Its syntax resembles older .ini files. Python 3.11 added the tomllib reader to the standard library:

>>> import tomllib
>>> with open('config.toml', 'rb') as f:
...     config = tomllib.load(f)
>>> config
{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

Note the 'rb': tomllib reads raw bytes rather than text. The standard library only reads TOML; writing it back out needs a third-party package like tomli-w. Modern Python projects commonly use TOML in pyproject.toml to declare dependencies, and Rust uses it for Cargo configuration.

TOML’s name calls it obvious and minimal, though new versions have added more features:

CSV

JSON, YAML, and TOML can represent nested data. For a plain table of rows and columns, CSV (comma-separated values) is simpler. Our tweets have no nesting, so CSV can store one tweet per row:

text,username
hello,Trump
world,Obama
hola,Obama

The first line names the columns, and every line after it is one row, with a comma between the fields. Spreadsheet programs can open this compact format. Python reads and writes it with the built-in csv module, and csv.DictReader returns one dict per row, keyed by the header:

>>> import csv
>>> with open('tweets.csv', newline='') as f:
...     for row in csv.DictReader(f):
...         print(row)
{'text': 'hello', 'username': 'Trump'}
{'text': 'world', 'username': 'Obama'}
{'text': 'hola', 'username': 'Obama'}

CSV cannot represent a list inside a cell without another encoding convention. A tweet containing a list of hashtags must be flattened or stored in a format such as JSON.

Which format when

Each format has a common use:

• JSON for data moving between programs: API responses, scraper output, anything crossing a network. The default for machine-to-machine.
• YAML for configuration a human edits, when the tool you are feeding expects it: GitHub Actions, Docker, Kubernetes.
• TOML for configuration a human edits in the Python and Rust worlds: pyproject.toml and its neighbors.
• CSV for flat tables headed to a spreadsheet or a data-analysis library.

This week’s quiz covers these config formats and is open-note and on paper like the others. Use the practice quiz and its answers to study.

Files and databases

Our JSON examples load the whole file into memory and loop over its contents. The memory required grows with the size of the file. A database can keep the data on disk, and SQL queries select rows without loading the full dataset.