Syntactic Sugar

Exceptions, web scraping, and the tools you wrote for docchat use the accumulator loop: start with an empty list, walk over some items, and append the ones you want.

Syntactic sugar is a shorter notation for a common pattern. It changes how code is written without adding new power to the language, and every sugared expression has a longer equivalent.

Toy Story meme: Buzz gestures to the horizon while telling Woody 'syntactic sugar, syntactic sugar everywhere.'

Translating the short form into the long one is called desugaring. When a piece of sugar is unclear, write out the loop it represents.

Sugar we have already used

You have used syntactic sugar since your first loop because a for loop is itself sugar for a while loop. A counting for loop over range looks like this:

for i in range(5):
    print(i)

It desugars into a while loop that manages the counter by hand:

i = 0
while i < 5:
    print(i)
    i = i + 1

Both print 0 through 4. The for version avoids counter bookkeeping: initializing the counter at 0, incrementing it by 1, and choosing between < and <=.

'Change My Mind' meme: a man sits at a table on a campus walkway behind a sign reading 'for loop is just a syntactic sugar for a while loop. CHANGE MY MIND.'

These constructs form a hierarchy: a while loop is more powerful than a for loop, which is more powerful than a list comprehension. “More powerful” means it can express more. A while loop can repeat on any condition at all; a for loop can only walk through a sequence; a comprehension can only build a list. A more restricted construct states your intent more clearly. For example, a comprehension tells the reader that the code is building a collection. Use the least powerful construct that does the job, and use while when repetition depends on a condition rather than a sequence.

Back in the factorial function we wrote result = result * i. Python lets you shorten that to result *= i, and +=, -=, and friends work the same way. These operators remove repetition without adding new behavior.

List comprehensions

An accumulator loop often takes every item in a list and transforms it into a new item. Here we greet everyone in a list of names:

names = ['alice', 'bob', 'charlie', 'dave', 'eve']
greetings = []
for name in names:
    greetings.append('hello ' + name)

The loop walks over names and appends one greeting per name. For a different transformation, only 'hello ' + name needs to change.

A list comprehension writes this pattern in a single line:

greetings = ['hello ' + name for name in names]

It produces the same list, which we can check in the REPL:

>>> ['hello ' + name for name in names]
['hello alice', 'hello bob', 'hello charlie', 'hello dave', 'hello eve']

Read it from the inside out: for each name in names, compute 'hello ' + name and collect the results into a list. The general shape is [COMPUTATION for VARIABLE in LIST], and it desugars into the original loop:

accumulator = []
for VARIABLE in LIST:
    accumulator.append(COMPUTATION)

To read an unfamiliar comprehension, expand it into these three lines.

Tuxedo Winnie the Pooh meme, ordinary Pooh beside a three-line accumulator loop that appends to a list, and fancy tuxedo Pooh beside the same result written as a one-line list comprehension.

For example, a comprehension can calculate the squares of the first ten numbers:

>>> [x*x for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

For each x from 0 to 9, compute x*x; range(10) supplies the sequence to walk. A useful rule for reading other people’s code: any time you see the keyword for inside square brackets, you are looking at a comprehension.

Filtering

Often you need only the items that pass a test. The scraper used this pattern to keep matching rows. A comprehension filters by adding an if to the end. Here we keep only the short words in a sentence:

>>> sentence = 'This is an example sentence with a few words in it.'
>>> [word.lower() for word in sentence.split() if len(word) <= 2]
['is', 'an', 'a', 'in']

split() breaks the sentence into words, the if len(word) <= 2 keeps only the short ones, and word.lower() transforms each survivor. The shape is now [COMPUTATION for VARIABLE in LIST if CONDITION], and it desugars into a loop with an if nested inside:

accumulator = []
for VARIABLE in LIST:
    if CONDITION:
        accumulator.append(COMPUTATION)

The condition can be any expression Python reads as true or false. Python treats 0, empty strings, and empty collections as false; nonzero numbers and nonempty strings and collections are true. That is why x % 2 works as an is-odd test: the remainder is 1 (true) for odd numbers and 0 (false) for even ones. We can keep the squares of the odd numbers:

>>> [x*x for x in range(10) if x % 2]
[1, 9, 25, 49, 81]

Comprehensions can filter a list of records in one line: [t for t in tweets if 'trump' in t['text'].lower()] selects every tweet that mentions Trump.

When to stop

Comprehensions can nest inside each other and chain several for clauses to build or flatten lists in a single line. These forms are often difficult to read. A nested comprehension can build a list of lists:

>>> [[i for i in range(x)] for x in [2, 3, 4] if x % 2 == 0]
[[0, 1], [0, 1, 2, 3]]

Nested comprehensions desugar from the outside in, and stacked for clauses read from left to right. If your own code requires desugaring to understand, use a plain loop.

'All the things' meme: a wild-eyed stick figure charging forward, captioned 'list comprehension all the things.'

Use a comprehension for one transformation with an optional filter. Use a loop for a nested comprehension, a second for, or substantial work in the body.

Dictionary comprehensions and other sugar

A dictionary comprehension builds a dict with {key: value for ...}:

>>> {name: len(name) for name in names}
{'alice': 5, 'bob': 3, 'charlie': 7, 'dave': 4, 'eve': 3}

For each name we make one entry mapping the name to its length. Dictionary comprehensions replace the square brackets with curly braces and the computation with a key: value pair.

F-strings are sugar for joining strings, and a with block that closes a file is sugar for a try/finally block. The Real Python guide to list comprehensions, the DataCamp tutorial on dictionary comprehensions, and Corey Schafer’s video also cover set comprehensions and generators.

War dialing example

This week’s war dialing lab scans all 1024 IP addresses the DPRK owns and counts how many run a web server. It reuses requests and try/except from the web-scraping and exceptions weeks. The accumulator loop walks the list of IPs and keeps the ones that answer:

dprk_ips_with_servers = []
for ip in dprk_ips:
    if is_server_at_hostname(ip):
        dprk_ips_with_servers.append(ip)

One transformation and one filter can be written as a list comprehension:

dprk_ips_with_servers = [ip for ip in dprk_ips if is_server_at_hostname(ip)]

Complete this code in the lab.