JSON & Alternatives

We want to share our tweet-processing program with a colleague. They need saved tweet records, settings they can edit, and a table they can open in a spreadsheet.

A Python dictionary exists in memory while a program runs. To save it in a file or send it to another program, we can convert it to JSON (JavaScript Object Notation).

JSON Syntax

Start by saving the records in tweets.json. JSON looks familiar because its text representation resembles Python’s dicts and lists. Save the following document as tweets.json in your practice folder before running the file examples. It holds a list of tweets, each with text and username fields:

[
    {"text": "hello", "username": "Trump"},
    {"text": "world", "username": "Obama"},
    {"text": "hola",  "username": "Obama"}
]

The square brackets are a list (JSON calls it an array), each pair of curly braces is a dict (JSON calls it an object), and the keys and text are strings in double quotes. Python reads the JSON types as follows:

  • an object {...} becomes a Python dict
  • an array [...] becomes a Python list
  • a string, always in double quotes, becomes a str
  • a number becomes an int or a float
  • true and false become True and False
  • null becomes None

Python will refuse to read JSON that violates these rules:

  • strings and keys use double quotes, never single
  • the booleans and null are lowercase: true, not True
  • there is no trailing comma after the last item
  • there are no comments anywhere

Loading JSON

Create handoff.py beside tweets.json and add the Python blocks below in order. Before running it, set up its Python environment in this practice folder. Run python handoff.py from that folder after each addition. The output beneath each block shows what its new lines print. The completed handoff.py, tweets.json, config.yaml, and config.toml can also be saved together in a practice folder. The YAML section needs PyYAML, and the TOML section needs Python 3.11 or later.

In our programs, JSON arrives as a string containing a web response body or a file’s contents. json.loads (“load string”) converts that string into Python objects:

import json

with open('tweets.json', encoding='utf-8') as f:
    jsontext = f.read()

data = json.loads(jsontext)
print(type(data))
print(data[0])
print(data[0]['username'])

The output is:

<class 'list'>
{'text': 'hello', 'username': 'Trump'}
Trump

json.loads returns an ordinary Python list of dicts, which we can index, look up by key, and loop over. The printed dictionary uses single quotes around its strings. The JSON double-quote rule applies to JSON text. Printing the username itself displays Trump without quotes. Read data[0]['username'] one operation at a time: data[0] selects the first dictionary, and ['username'] selects a value from that dictionary. If a lookup fails, inspect the intermediate value and its type before adding another pair of brackets.

Parsing checks whether the text is valid JSON. Our code must also check whether the records have the fields it needs. {"username": "Ada"} is valid JSON even if your program also expects a text field. Invalid JSON raises JSONDecodeError. A missing key after a successful parse raises KeyError when you look it up.

The tweets dataset from web scraping uses a folder of master_*.json files, each holding a list of tweets. After loading a file, we can count tweets from one account with Python:

print(sum(1 for tweet in data if tweet['username'] == 'Obama'))

The output is:

2

The expression inside sum contributes one for each matching tweet, so the result counts records rather than adding their field values.

We read the file into a string before parsing it. json.load (no s) can combine those two steps, using the open file from strings and files. Append this equivalent read to check the shorter form:

with open('tweets.json', encoding='utf-8') as f:
    data = json.load(f)

loads reads a string. load reads an open file. Neither function takes a filename and opens it for you: json.loads('tweets.json') would try to parse the letters of the filename as JSON.

Writing JSON

Saving the records lets the next run read them again. Write the current data list back to tweets.json with json.dump:

with open('tweets.json', 'w', encoding='utf-8') as f:
    json.dump(data, f, indent=2)

That produces the same three-record JSON document we started with. The program also needs saved settings. Suppose the tweet report appears on a site titled “My Blog”. Its port, debug setting, and tags should be editable without changing Python. json.dumps (“dump string”) turns those Python settings into a JSON string:

config = {'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}
print(json.dumps(config, indent=2))

The output is:

{
  "title": "My Blog",
  "port": 8080,
  "debug": true,
  "tags": [
    "python",
    "web"
  ]
}

The indent=2 argument produces a readable block. Without it, json.dumps produces compact JSON for storage or transmission.

When you are unsure of the shape of nested data, print it as indented JSON:

print(json.dumps(data, indent=2))

The output is:

[
  {
    "text": "hello",
    "username": "Trump"
  },
  {
    "text": "world",
    "username": "Obama"
  },
  {
    "text": "hola",
    "username": "Obama"
  }
]

json.dumps accepts nested combinations of dicts, lists, strings, numbers, booleans, and None.

To write to a file instead of a string, use json.dump (no s) with an open file:

with open('config.json', 'w', encoding='utf-8') as f:
    json.dump(config, f, indent=2)

dumps returns a string. dump writes to an open file. Together, loading and dumping let a program read saved records, modify ordinary Python objects, and store the result for the next run. Python’s None becomes JSON null:

print(json.dumps({'author': None}))

The output is:

{"author": null}

JSON in APIs

Files are only one way to exchange these records. A tweet-processing tool could also receive data from an API, where it must inspect the response shape before selecting fields. We have already used this mechanism: when docchat asks an LLM a question, the API’s response body is JSON. After the client library parses it, the model’s answer is one string inside nested objects and arrays:

response = {'choices': [{'message': {'content': 'Two tweets are from Obama.'}}]}
reply = response['choices'][0]['message']['content']
print(reply)

The output is:

Two tweets are from Obama.

Here response is a small local example of a parsed response body, so this script needs no API key. The Groq client in the LLM reading wraps the same kind of data in objects, so it uses dots such as completion.choices[0].message.content instead. Use the access style provided by the value you actually received.

The agents project uses the same exchange. The Twitter clone’s backend will also exchange JSON with the browser.

YAML

Our colleague can load the records, but they would also like to leave an explanation next to each setting. JSON has no comments, requires double quotes on every key, and rejects trailing commas. Hand-edited configuration files often use YAML instead. The name stands for YAML Ain’t Markup Language: YAML stores data, while a markup language such as HTML labels document content.

Save the same settings as config.yaml beside tweets.json:

# a comment, at last
title: My Blog
port: 8080
debug: true
tags:
  - python
  - web

YAML uses one key: value per line, indentation for nesting, and - for list items. It also permits # comments. Loading it into Python needs the third-party library PyYAML. As in the LLM project, record the dependency in requirements.txt:

pyyaml

In the same terminal, with this folder’s environment active, install the dependency:

$ python -m pip install -r requirements.txt

If you open another terminal, activate the same .venv there before installing or running Python. Append the YAML loader to handoff.py, keeping the earlier code that defines data:

import yaml

with open('config.yaml', encoding='utf-8') as f:
    config = yaml.safe_load(f)
print(config)

The output is:

{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

JSON syntax is also valid YAML 1.2, which is the joke in this meme:

'Change My Mind' meme: a man seated at a table behind a sign that reads 'YAML is just Python-like JSON.'

GitHub Actions, Docker Compose, and Kubernetes use YAML for configuration. The optional final-week automation exercise uses a YAML file to run a test on GitHub. YAML parsers can infer types from unquoted values, which can produce unexpected results. PyYAML’s safe_load uses older boolean rules under which the bare word no is read as False:

print(yaml.safe_load('country: no'))

The output is:

{'country': False}

This is known as the Norway problem: a file listing country codes can turn the code NO into False without an error. Quote a value such as "no" when it should stay text. YAML 1.2 narrows these boolean rules, but the parser in use determines what your program receives. To inspect YAML’s inferred values, load the file and run print(json.dumps(config, indent=2)).

The same boolean conversion applies to dictionary keys. Try these two strings in handoff.py:

print(yaml.safe_load('on: push'))
print(yaml.safe_load('"on": push'))
print(yaml.safe_load('version: 3.10'))
print(yaml.safe_load('version: "3.10"'))

They print:

{True: 'push'}
{'on': 'push'}
{'version': 3.1}
{'version': '3.10'}

Quote a key or value when its spelling must survive as text, such as a version number. A list nested under a key becomes a Python list, so select an item by index before looking up a field within it. For example, yaml.safe_load('users: [{name: Ada}, {name: Grace}]')['users'][1]['name'] gives 'Grace'.

Settings can also hold several lines of text. Append this example:

messages = yaml.safe_load("""
welcome: |
  Hello!
  Welcome to the blog.
summary: >
  Posts from
  this week.
""")
print(repr(messages['welcome']))
print(repr(messages['summary']))

It prints 'Hello!\nWelcome to the blog.\n' and 'Posts from this week.\n'. The | keeps these line breaks. The > folds them into spaces, and both forms retain a final newline here. repr makes those newline characters visible.

TOML

The unquoted country code became a boolean in YAML. TOML requires us to quote strings, including country codes. TOML (Tom’s Obvious Minimal Language) is a configuration format with stricter type rules than YAML. Save the TOML version as config.toml in the same folder:

title = "My Blog"
port = 8080
debug = true
tags = ["python", "web"]

TOML uses key = value, quotes strings, and groups related keys under [section] headers when needed. Its syntax resembles older .ini files. Python 3.11 added the tomllib reader to the standard library:

import tomllib

with open('config.toml', 'rb') as f:
    config = tomllib.load(f)
print(config)

The output is:

{'title': 'My Blog', 'port': 8080, 'debug': True, 'tags': ['python', 'web']}

Note the 'rb': tomllib reads raw bytes rather than text. The standard library only reads TOML. Writing it back out needs a third-party package like tomli-w. Modern Python projects commonly use TOML in pyproject.toml to declare dependencies, and Rust uses it for Cargo configuration.

TOML’s name calls it obvious and minimal, though new versions have added more features:

Trojan horse meme: the wooden horse is labeled 'TOML v1.1' and the soldiers hidden inside it are labeled 'JSON'.

Nested TOML Settings

Our colleague also needs an owner and separate export settings. A section header creates a nested dictionary, and a dotted header adds another level. Append this string-based example to handoff.py:

settings = tomllib.loads('''
[owner]
name = "Ada"

[export.csv]
columns = ["username", "text"]
options = {header = true, delimiter = ","}
''')
print(settings['owner']['name'])
print(settings['export']['csv']['columns'][0])
print(settings['export']['csv']['options']['header'])

It prints Ada, username, and True. tomllib.loads reads a string, while tomllib.load reads an open binary file. The inline table in braces also becomes a dictionary. Keys that follow a section header belong to that section until the next header, regardless of blank lines. Use string keys for dictionaries and integer positions for lists.

CSV Export

Our colleague wants to inspect the tweets in a spreadsheet. JSON, YAML, and TOML can represent nested data. For a plain table of rows and columns, CSV (comma-separated values) is simpler. Our tweets have no nesting, so CSV can store one tweet per row:

text,username
hello,Trump
world,Obama
hola,Obama

The first line names the columns, and every line after it is one row, with a comma between the fields. Spreadsheet programs can open this compact format. Export the loaded tweet records with csv.DictWriter, giving it the column order explicitly:

import csv

with open('tweets.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.DictWriter(f, fieldnames=['text', 'username'])
    writer.writeheader()
    writer.writerows(data)

The file contains the header and three rows shown above. The writer handles quoting for us, and newline='' lets the CSV module handle line endings. Python reads and writes it with the built-in csv module, and csv.DictReader returns one dict per row, keyed by the header:

with open('tweets.csv', newline='', encoding='utf-8') as f:
    for row in csv.DictReader(f):
        print(row)

The output is:

{'text': 'hello', 'username': 'Trump'}
{'text': 'world', 'username': 'Obama'}
{'text': 'hola', 'username': 'Obama'}

CSV cannot represent a list inside a cell without another encoding convention. A tweet containing a list of hashtags must be flattened or stored in a format such as JSON.

A field can also contain a comma, such as the tweet hello, world. CSV quotes that entire field to distinguish the comma in the text from the comma separating columns:

text,username
"hello, world",Ada

Use csv.DictReader or csv.reader instead of splitting each line at every comma. The library understands quoted fields and embedded newlines. CSV fields normally arrive as strings, so a numeric-looking value still needs a conversion such as int(...) before arithmetic.

CSV Text in Memory

An API or helper function may hand us CSV text before it has been saved to a file. io.StringIO gives that string the same read/write interface as an open text file. Append this small export to handoff.py:

import io

csv_buffer = io.StringIO()
writer = csv.DictWriter(csv_buffer, fieldnames=['text', 'username'],
                        lineterminator='\n')
writer.writeheader()
writer.writerow(data[0])
csv_text = csv_buffer.getvalue()
print(csv_text, end='')
print(list(csv.DictReader(io.StringIO(csv_text))))

The output is:

text,username
hello,Trump
[{'text': 'hello', 'username': 'Trump'}]

The writer writes into csv_buffer, and .getvalue() returns the accumulated string. Passing csv_text to a new StringIO lets DictReader read the same rows without opening a disk file. The lab’s CSV helpers use these two directions.

Choosing a Format

Choose the format the receiving tool expects. Each format has a common use:

  • JSON for API responses and saved records such as scraper output.
  • YAML for tools that expect it, such as GitHub Actions, Docker Compose, and Kubernetes.
  • TOML for configuration files such as Python’s pyproject.toml.
  • CSV for flat tables opened in a spreadsheet or data-analysis library.

Files and Databases

Finding one account’s tweets becomes slower as the collection grows. Our JSON examples load the whole file into memory and loop over its contents. The memory required grows with the size of the file. A database can keep the data on disk, and SQL queries select rows without loading the full dataset. The next reading gives the same kind of records a persistent message history that we can query and update one row at a time.