Lab: Files: Download and Decode

Due: Wednesday, September 30 at 11:59pm Worth: 6 points

This week’s lab works with two kinds of files: a downloaded video and an encoded text message. The video steps earn 2 points and the decoded-message steps earn 4 points, in the same submission. You can complete either part first.

Start

Open the starter repository and select Use this template to create lab-files under your own account. Clone your copy and open it in VS Code. See course setup for the commit, sync, and submission workflow. A starter ZIP is also available: unzip it, open the inner lab-files folder, then use Initialize Repository and Publish to GitHub in VS Code. Use this one repository for every part below; do not create separate repositories for the parts.

Part A: Download a Video

Columbus meme captioned 'I didn't steal your code... I DISCOVERED it.'

Python makes it easy to download and run programs, called scripts, that other people have written. In this lab, you will run a script for downloading videos from YouTube. You will write your own script in the next project.

Finding the Working Repo

There are many GitHub repos with code for downloading YouTube videos. Here are two:

Compare the projects’ recent test results in their Actions tabs. A README may also show a status badge:

Test results change over time, and passing tests do not guarantee that every YouTube download works. Use yt-dlp for the remaining steps so the folder names match these instructions. From the parent folder containing your assignment repository, clone the downloader beside it, then step into the downloader folder:

$ git clone https://github.com/yt-dlp/yt-dlp.git
$ cd yt-dlp

Running the Code

Recall that ls lists the files in the current folder. If the clone worked, you will see something like this:

$ ls
bundle           LICENSE         README.md                 yt-dlp.cmd
Changelog.md     Maintainers.md  supportedsites.md         yt-dlp.sh
CONTRIBUTING.md  Makefile        test
CONTRIBUTORS     public.key      THIRD_PARTY_LICENSES.txt
devscripts       pyproject.toml  yt_dlp

Your exact output may differ as the project changes; look for pyproject.toml and the yt_dlp folder. Thousands of people have contributed to this project. Later in the course, you will learn how to contribute to a project like it and add that work to your GitHub profile.

Commands marked $ run in your terminal; do not type the $. If you see Python’s >>> prompt, run exit() first. Check python3 --version against the project’s supported Python versions. From the yt-dlp folder, create a virtual environment:

$ python3 -m venv .venv

Activate it with source .venv/bin/activate on macOS/Linux, or .venv\Scripts\Activate.ps1 in Windows PowerShell. If PowerShell blocks activation, use a Command Prompt terminal and run .venv\Scripts\activate.bat. Install the downloader’s dependencies:

$ python -m pip install -e ".[default]"

Current YouTube support also uses a JavaScript runtime. Follow the project’s YouTube JavaScript setup instructions to install the recommended runtime, Deno, then reopen your terminal and reactivate the environment. Run deno --version to check the installation.

Python projects commonly use a file named __main__.py as an entry point. It usually sits in a subfolder named after the project, which here is yt_dlp. Run it with:

$ python yt_dlp/__main__.py

Usage: yt-dlp [OPTIONS] URL [URL...]

yt-dlp: error: You must provide at least one URL.
Type yt-dlp --help to see a list of all options.

yt_dlp/__main__.py is a relative path: the file’s name together with the subfolder it lives in. Drop the subfolder and Python cannot find the file:

$ python __main__.py
python3: can't open file '__main__.py': [Errno 2] No such file or directory

This is a missing-file error caused by omitting one folder from the path; Python cannot even start the script. Even when called correctly, the program displays an error message rather than opening a window. Python scripts usually have a command-line interface (CLI), where you type the options as part of the command, rather than the graphical user interface (GUI) you are used to.

To download a video, pass the script a URL. Use https://www.youtube.com/watch?v=dQw4w9WgXcQ for this lab. Paste the URL at the end of the command:

$ python yt_dlp/__main__.py -f "best[height<=360][ext=mp4]/best[height<=360]" -o "../lab-files/video.%(ext)s" "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

The -f option requests a single file containing video and audio at up to 360p, avoiding a separate merge step. The -o option saves it directly in your sibling submission repository as video.mp4 or video.webm. If you used a different folder name, change that output path to match. Low resolution keeps the file small enough to commit: GitHub blocks files larger than 100 MiB. Submit the video itself, not a Git LFS pointer.

If you see a certificate error, a sign-in/bot check, or “Requested format is not available,” show the instructor the exact error. YouTube changes can break a download even when you followed the steps correctly. Do not spend the lab repeatedly retrying the same command.

Most command-line scripts print detailed instructions when you pass --help:

$ python yt_dlp/__main__.py --help

Run it and you will see that yt-dlp has options for setting audio and video quality, grabbing whole playlists, and routing the download through a proxy.

Reusing Open-Source Tools

'Wait, it's all yt-dlp?' meme: two astronauts in orbit look at a collage of YouTube-downloader websites, with the caption 'always has been.'

yt-dlp is a fork of youtube-dl: its developers built on an existing project’s source code. In this lab you use that shared code through its command-line interface.

Open the downloaded video to confirm it plays the assigned clip. Commit the video itself, not the downloader’s source or a Git LFS pointer. Gradescope checks that a video is present and can decode its first five frames; video identity and full playback are self-checks.

Part B: Read and Decode Text

Run the Python examples from your lab-files folder so the relative paths work. Type python3 to enter the >>> prompt and exit() to return to your terminal. Keep the original files in files/ and part2/ unchanged.

Reading Historical Documents

In 1972, President Richard Nixon flew to China.

1972 black-and-white photograph of President Nixon shaking hands with Chinese Premier Zhou Enlai on an airport tarmac.

The Shanghai Communiqué was a joint statement by the United States and the People’s Republic of China, famous for first stating the US “One China” policy. The official English version was stored as ASCII text, and you have a copy in files/shanghai_communique.english. Open it in VS Code to confirm you can read it, then read it from Python:

>>> f = open('files/shanghai_communique.english', encoding='ascii')
>>> text_english = f.read()
>>> print(text_english[:1000])

The document scrolls past starting with JOINT COMMUNIQUE ... February 28, 1972 ... Shanghai. The relative path includes the folder: shanghai_communique.english lives inside files, so its path is files/shanghai_communique.english. Drop the folder and you get a FileNotFoundError. The encoding='ascii' argument tells open which table converts the file’s bytes into letters.

ASCII cannot represent Chinese characters. One encoding designed for simplified Chinese is GB2312. The file files/shanghai_communique.chinese1 holds a Chinese translation in GB2312. Open that file in VS Code. If the editor assumes UTF-8, you may see garbled characters or an encoding warning. Python can read it once you name the encoding:

>>> f = open('files/shanghai_communique.chinese1', encoding='gb2312')
>>> text_gb2312 = f.read()
>>> print(text_gb2312[:1000])

Both the path and the encoding changed from the English version.

How the Encodings Work

ASCII pairs each English letter with one byte. The letter A is 65, which is 0x41 in hexadecimal, and you can check both directions in Python:

>>> 0x41
65
>>> b'\x41'.decode('ASCII')
'A'

The b'\x41' is a bytes object, raw numbers rather than text, which is why we call .decode to turn it into the letter A. GB2312 works the same way but spends two bytes per Chinese character (a Hanzi). For example, the Hanzi 友 (“friend”) is the two bytes \xd3 and \xd1:

>>> b'\xd3\xd1'.decode('gb2312')
'友'

Two bytes can distinguish tens of thousands of characters, which is plenty for the roughly 3000 Hanzi in everyday modern use.

When One Encoding Is Not Enough

GB2312 was designed for the simplified characters used in mainland China, and it does not cover the traditional characters used in Taiwan. Taiwanese programmers built their own system, Big5, and the two are not compatible. That incompatibility causes two problems.

First problem: the same bytes are illegal in the other encoding. The file files/shanghai_communique.chinese2 holds the same communiqué encoded in Big5. Reading these files with the wrong encoding can raise a UnicodeDecodeError when a byte sequence is not valid in the selected encoding.

One idea is to try one encoding and fall back to the other if it fails. The following function does that with try/except, which we will study properly next week:

def load_chinese_file(filename):
    # open in binary mode ('b'): reading raw bytes, no encoding yet
    f = open(filename, 'br')
    bs = f.read()
    try:
        text = bs.decode('gb2312')
        print('gb2312')
    except UnicodeDecodeError:
        text = bs.decode('big5')
        print('big5')
    return text

Save this as chinese.py in the project folder (not inside files). If you are at a >>> prompt, run exit() first, then start Python with the file preloaded:

$ python3 -i chinese.py

The function loads either kind of file, prints the encoding it detected, and returns the text:

>>> text1 = load_chinese_file('files/shanghai_communique.chinese1')
gb2312
>>> text2 = load_chinese_file('files/shanghai_communique.chinese2')
big5

Those lines print the encoding, not the contents; the contents were returned into text1 and text2, so print(text1) shows the document.

Second problem: the same bytes are legal in both encodings but mean different things. The two bytes for 友 (“friend”) in GB2312 are the two bytes for 衭 (“the front of a shirt”) in Big5:

>>> b'\xd3\xd1'.decode('gb2312')
'友'
>>> b'\xd3\xd1'.decode('big5')
'衭'

The bytes b'I love \xd3\xd1' therefore read as “I love friends” or “I love shirts” depending on the encoding, and no error warns you. This is why you cannot reliably guess an encoding from the bytes alone. An editor’s automatic guess can be wrong.

Unicode

Since the 1980s, dozens of competing Chinese encodings appeared, which made it hard for Chinese speakers to exchange files at all. The Unicode Consortium was founded in 1991 to give every character in every language a single number that works everywhere. Its three standard encodings, UTF-8, UTF-16, and UTF-32, can all represent Chinese, and since around 2000 UTF-8 has become the default nearly everywhere:

Line chart of encodings used by web pages over time: the UTF-8 line rises from near zero around 2001 to dominate by the early 2010s as older encodings decline.

A small piece of history: seven emoji in the current standard were added at North Korea’s request. The DPRK first proposed that ☕ be called HOT TEA, but an American suggested a more general name so it could also mean coffee, the North Koreans agreed, and it shipped as HOT BEVERAGE. The repo’s dprk-unicode.md has more about this technical cooperation.

Decode the Intercepted Message

For the rest of the lab you are an analyst at the US State Department.

A person in a suit walking across the seal of the Central Intelligence Agency inlaid in a lobby floor.

A US government employee is leaking classified plans for nuclear submarines to the Brazilian government, and a field agent has intercepted a message about where the leaker will next meet their Brazilian contact. The message is in part2/secret_message.txt, encoded using a character encoding you must identify:

>>> f = open('part2/secret_message.txt', 'rb')
>>> f.read()
b'\xc1\x95\xa3\xcb\x95\x89\x96k@\x85\x95\x83\x96\x95\xa3\x99\x85`\x94\x85@\x95\x81@\x85\x94\x82\x81\x89\xa7\x81\x84\x81@D@\x94\x85\x89\x81`\x95\x96\x89\xa3\x85@\x84\x85@\xa3\x85\x99H\x81`\x86\x85\x89\x99\x81K%'

Your job is to decode the message so the FBI can reach the meeting. This fictional message is inspired by a real submarine espionage case.

Hint. The full list of encodings Python knows is in the codecs documentation. Brazilians write in Portuguese, so try the encodings meant for Western Europe. Search the table for “Western Europe” and try those codec names with open(..., encoding=...) or bytes.decode(...). A successful decode is not enough: look for readable Portuguese, including sensible accents and punctuation. The decoded message is in Portuguese; run it through Google Translate if you are curious, but you do not have to.

Save the Decoded Message

Create part2/secret_message.utf8 at that exact path, containing the decoded message and saved in UTF-8. A new file in VS Code uses UTF-8 automatically.
Preserve the original Portuguese text, accents, and punctuation. Do not submit an English translation or the printed b'...' representation of the bytes.

You may save the decoded text with Python using open(..., 'w', encoding='utf-8') and the file object’s write method, or paste the decoded text into a new UTF-8 file in VS Code. Changing a filename’s extension does not change its encoding. Reading the historical documents is practice; the graded deliverable is part2/secret_message.utf8. Keep part2/secret_message.txt unchanged.

Submit

Commit and sync the video and part2/secret_message.utf8 in your lab-files repository. On Gradescope, choose GitHub and submit that repository and the branch containing your work to lab-files. Submit once for the whole week. You may resubmit as you finish more steps.

Points for Each Step

Your score is the sum of the steps that pass. Each row earns its own points; unfinished steps do not erase completed work. All checks run automatically. You may resubmit as you finish more steps. A function’s check includes its published examples and additional inputs for that same function. Keep unfinished Python bodies valid with pass: a syntax error can prevent other checks that import that file from running. Fractional points are added before the final score is rounded to four decimal places.

Graded Step	Points
Decoded Portuguese message	2
Decoded file saved at the requested path	1
Decoded file uses UTF-8	1
Downloaded video is present and recognizable	1
Video file committed	1
Total	6