Strings and Files

So far, our programs have used values written directly in the code. Automating tasks requires data from elsewhere, such as a folder of files, a webpage, or a spreadsheet. The open function gives a program access to files for reading and writing. Text encodings convert a file’s bytes into characters and explain why opening a text file can fail.

Reading a file

Start with a two-line English poem in poem.txt, stored next to the program. The open function returns a connection to the file, and the file’s read method returns its contents as one string:

>>> f = open('poem.txt')
>>> contents = f.read()
>>> f.close()
>>> print(contents)
the cat sat
on the warm mat

open('poem.txt') returns a file object, a handle to the file that we stored in f. Calling f.read() reads the whole file and returns one string, including newlines, which we saved in contents. The f.close() line releases the operating-system resource used by the file. A program that forgets to close thousands of files can run out of these resources.

Use a with block so Python closes the file automatically:

>>> with open('poem.txt') as f:
...     contents = f.read()

The with block opens the file, runs the indented code, and closes the file when the block ends, even if the code inside raises an exception. contents has the same value as before, but no explicit close is needed.

You can also loop over a file object to process one line at a time:

>>> with open('poem.txt') as f:
...     for line in f:
...         print(repr(line))
'the cat sat\n'
'on the warm mat\n'

Each line is printed with repr to make its trailing \n newline visible. print normally renders that character as a line break. See Al Sweigart’s Automate the Boring Stuff, Chapter 9 for more file operations.

Paths

open('poem.txt') searches for the file relative to the current working directory. If the file is not there, open raises a FileNotFoundError:

>>> open('nope.txt')
FileNotFoundError: [Errno 2] No such file or directory: 'nope.txt'

A path identifies a file’s location and can be absolute or relative.

An absolute path gives the file’s exact location starting from the top of the drive. On macOS and Linux it starts with a /, like /Users/alice/csci40/poem.txt; on Windows it starts with a drive letter, like C:\Users\alice\csci40\poem.txt. An absolute path names the same file regardless of the working directory.

poem.txt and data/poem.txt are relative paths. Python interprets them relative to the folder where the program is running, called the working directory. The os.getcwd() function returns that directory as an absolute path:

>>> import os
>>> os.getcwd()          # cwd = "current working directory"
'/Users/alice/csci40'

Python resolves a relative path against the working directory. With the working directory above, open('poem.txt') opens /Users/alice/csci40/poem.txt. A FileNotFoundError means no file exists at the resulting path.

Use these three terminal commands to inspect and change the working directory:

$ pwd                 # print working directory: where am I?
$ ls                  # list the files here
$ cd data             # change directory: move into the "data" folder

(On Windows the first two are cd with no argument and dir.) The special name .. means “the folder one level up,” so cd .. moves to the parent folder. When a lab tells you to run something “from inside the repo,” cd into that folder so your relative paths resolve correctly.

Writing a file

To write results to a file, pass open a second argument called the mode. The mode 'w' opens the file for writing:

>>> with open('shopping.txt', 'w') as f:
...     f.write('apples\n')
...     f.write('bread\n')
7
6

Those two numbers are the REPL echoing what each f.write returns, the count of characters written (7 for apples\n, 6 for bread\n); you can ignore them. The write method does not add newlines for you, so we put the \n in ourselves; leave them out and both words land on one line. Opening a file with 'w' creates it if it does not exist. If the file already exists, opening it erases its contents. Check the result:

>>> with open('shopping.txt') as f:
...     print(f.read())
apples
bread

The default mode, with no letter, is 'r' for read, so open('poem.txt') and open('poem.txt', 'r') are the same call. Project 1 uses this read-change-write sequence: it opens a .md file, rewrites its contents as HTML, and writes an .html file.

Text is numbers

A file stores bytes, whole numbers from 0 to 255. To store text, software uses agreed-upon mappings between numbers and characters.

ASCII was standardized in 1963 to cover the characters on an American typewriter: the letter A is 65, B is 66, a space is 32, and so on up to 127. ord maps a character to its number, and chr performs the reverse lookup:

>>> ord('A')
65
>>> chr(65)
'A'

ASCII covers only 128 symbols and cannot represent 友, ñ, م, or 😊.

Unicode aims to assign every character its own number, called a code point. It includes characters used in languages around the world, along with symbols and emoji. Unicode retains ASCII’s assignments, so A is still 65. ord and chr also work with Unicode:

>>> ord('友')
21451
>>> ord('😊')
128522

Python supports Unicode in strings and even permits variable names in other alphabets.

Code points and glyphs

A code point is a number; the picture drawn for that number is chosen by the font on your screen, and different systems draw the same number differently.

The pistol emoji has code point U+1F52B. For years every platform drew it as a realistic revolver, and then in 2016 Apple swapped their picture for a bright green water pistol. Nothing about the stored number changed, only the drawing, so the same message with the same code point could arrive as a threat on one phone and a toy on another. (Emojipedia has the whole saga.) This xkcd uses the distinction between code points and glyphs to propose a “vomiting modifier” that could attach to any emoji:

The vomiting modifier is fictional, but Unicode has real combining characters that attach to the character before them. A symbol you see as one letter can therefore be stored as two code points, which affects ordinary text as well as emoji. You can write any code point directly with the \u escape followed by its number in hex, so '\u0301' is the combining acute accent, code point U+0301. Take the accented á: it can be one precomposed code point, or a plain a followed by that combining accent, and the two are not equal:

>>> len('á')            # precomposed: one code point
1
>>> len('a\u0301')      # 'a' plus a combining accent: two code points
2
>>> 'á' == 'a\u0301'
False

Both strings render identically, but == compares code points, and the second string contains two. Two names that look the same can therefore fail an equality check. Normalization rewrites a string into a standard form. The NFC form uses precomposed code points where possible:

>>> import unicodedata
>>> unicodedata.normalize('NFC', 'a\u0301') == 'á'
True

Normalize strings from different sources before comparing them, because Python will not do it for you.

Different characters that look alike also create security problems. The Latin a and the Cyrillic а are different code points that most fonts draw identically:

>>> 'apple' == 'аpple'      # the second word starts with a Cyrillic 'а'
False

An attacker can register аpple.com with a Cyrillic а; it looks like apple.com but points to a different domain. This is called a homoglyph attack, and it is why your browser sometimes shows a foreign domain as an xn-- string instead of Unicode characters. A related problem occurs with the “smart quotes” produced by word processors. A character such as '\u2019' has a different code point from the straight ' Python expects, so pasting code from Google Docs can produce a hard-to-spot SyntaxError. Write code in a code editor such as VS Code rather than a document editor.

Encodings

Unicode includes code points above one million, but a byte holds only 0 through 255. An encoding converts a sequence of code points into a sequence of bytes and back. Two string methods do the conversion, .encode for string to bytes and .decode for bytes to string:

>>> 'hello world'.encode('utf-8')
b'hello world'

The result, with its leading b, is a bytes object: raw numbers rather than text. For plain English it looks unchanged because UTF-8 uses one byte per ASCII character and reuses ASCII’s numbers. Existing ASCII text is therefore valid UTF-8. UTF-8 is variable-length, so characters outside ASCII use more than one byte. Count the bytes three Chinese characters take under three Unicode encodings:

>>> len('计算机'.encode('utf-8'))
9
>>> len('计算机'.encode('utf-16'))
8
>>> len('计算机'.encode('utf-32'))
16

For these three characters, UTF-8 uses 9 bytes. UTF-16 uses 6 plus a 2-byte byte-order mark, and UTF-32 uses 12 plus a 4-byte byte-order mark. The byte-order mark records how to interpret the byte sequence. All three can represent every Unicode character; ASCII cannot represent 计算机. UTF-8 became the web’s dominant encoding:

The chart shows UTF-8 overtaking older web encodings within a decade. Use UTF-8 unless another encoding is specified. See Real Python’s guide to Unicode and encodings for more detail.

Files and encodings

When you open a text file, Python decodes its bytes into a string. Set the encoding explicitly, especially for text beyond plain English:

>>> with open('poem.txt', encoding='utf-8') as f:
...     contents = f.read()

Decoding bytes with the wrong encoding can raise a UnicodeDecodeError. These two bytes represent a valid character in China’s GB2312 encoding but not in Taiwan’s Big5:

>>> b'\xc8\xcb'.decode('gb2312')
'人'
>>> b'\xc8\xcb'.decode('big5')
UnicodeDecodeError: 'big5' codec can't decode byte 0xc8 in position 0: illegal multibyte sequence

b'\xc8\xcb' is a bytes object written with two \x escapes, the raw numbers 0xc8 and 0xcb, with no encoding attached. Under GB2312 they decode to 人 (“person”); under Big5 they do not form a valid byte sequence. Arbitrary bytes do not reliably identify their encoding. Code that reads them must know or detect the encoding separately. UnicodeDecodeError usually means that the selected encoding does not match the file.

Using these tools

Downloading a Video runs a file-downloading script, and File Encodings decodes historical documents stored in Chinese and Portuguese encodings. There is also a practice quiz for this material; work it on paper, the way the real quizzes are taken, and only then open the answers.

Web pages also arrive as bytes that we decode, usually as UTF-8, before extracting data. The next reading uses try and except to keep one decoding failure from stopping a script that processes thousands of files.