The Shell

Every time you ran python3 to start the REPL or python3 -m doctest lab.py to check a lab, you used a program called the shell.

The shell is a program that reads commands you type and runs other programs for you. Before graphical interfaces became common, users typed commands into terminals. Graphical interfaces work well for selecting items on a screen. Shell commands work well for applying the same operation to many files.

The condescending Unix user is an old comic stereotype:

You do not type at the shell directly; you type into a terminal, a window that shows the shell’s prompt and its output. In VS Code, open one with Terminal → New Terminal. Mac and Linux come with a shell built in; Windows users reach one through WSL or Git Bash, set up the same way as your other tools.

The prompt

The first thing the terminal shows you is a line ending in a $. That $ is the prompt. Type a command after it and press Enter; the shell runs the command, prints its output, and displays the prompt again.

The simplest command asks the shell where you are:

$ pwd
/home/alice/cs40

pwd stands for print working directory. The shell operates inside one working directory, and pwd prints its path. For the rest of these notes, a line starting with $ is one you type, and the line under it with no $ is what the shell printed back.

The filesystem tree

The filesystem is organized as a tree, the same shape as the HTML document tree from the CSS week but made of folders instead of tags.

See what is in the working directory with ls:

$ ls
hw1.py   hw2.py  lab02.py  lab04.py   webpage
hw10.py  hw3.py  lab03.py  notes.txt

ls lists the working directory: here, eight files and one folder named webpage. Commands often take options (also called flags), which start with a dash and change what the command does. Add -a to also show hidden files, the ones whose names begin with a dot:

$ ls -a
.   .gitignore  hw10.py  hw3.py    lab03.py  notes.txt
..  hw1.py      hw2.py   lab02.py  lab04.py  webpage

Three additional names appear. .gitignore is a configuration file hidden because its name starts with a dot. The docchat project will also use .env and .gitignore files. The other two, . and .., are special directory names defined below.

Add -l for a long listing with sizes, dates, and more:

$ ls -l webpage
total 8
-rw-r--r-- 1 alice alice 54 Feb  3 14:22 index.html
-rw-r--r-- 1 alice alice 20 Feb  3 14:22 style.css

Each line describes one file: its size in bytes (54), the date it last changed, and its name. The leading - marks an ordinary file; a folder would show a d there instead.

Here is that tree drawn out for our folder:

At the top is /, the root, which contains every other directory on the machine. The diagram shows the path from home to the personal directory alice and then to the current directory cs40, which contains the listed files and webpage directory.

Moving around

There are two ways to write a path through the filesystem tree. An absolute path starts at the root and spells out every folder down to the file: /home/alice/cs40/lab02.py. Because it starts at the root, it works regardless of the current working directory. A relative path is interpreted from the current working directory: from inside cs40, that same file is lab02.py, and the project page is webpage/index.html.

Move into a folder with cd:

$ cd webpage
$ pwd
/home/alice/cs40/webpage
$ ls
index.html
style.css

cd changes directory. Here it moves into webpage, and pwd confirms the new working directory. The special name .. refers to the parent directory, so this command returns to cs40:

$ cd ..

. means the current directory. Two shortcuts are useful: cd with no argument moves to your home directory, and ~ represents that directory within a path. From cs40, where do you think cd ../.. lands you? Try it, then run pwd and check.

Making and moving files

The shell also reads, copies, renames, and deletes files.

Read a short file straight to the screen with cat:

$ cat notes.txt
remember to email the professor about the extension

cat prints a file’s contents to the terminal. Its name is short for concatenate because it prints multiple files one after another.

Copy a file with cp, which takes a source and then a destination:

$ cp lab04.py lab04.py.bak
$ ls
hw1.py   hw2.py  lab02.py  lab04.py      notes.txt
hw10.py  hw3.py  lab03.py  lab04.py.bak  webpage

cp leaves the original untouched and writes a second copy under the new name, here a .bak backup of lab04.py.

Remove that copy with rm:

$ rm lab04.py.bak

rm removes a file without an undo operation or trash can. Check the target carefully, especially when using the wildcards introduced next.

Rename a file with mv, and make a folder with mkdir:

$ mv notes.txt reminders.txt
$ mkdir homework
$ ls
homework  hw10.py  hw3.py    lab03.py  reminders.txt
hw1.py    hw2.py   lab02.py  lab04.py  webpage

mv moves a file. Moving it to a new name in the same directory renames it, so notes.txt is now reminders.txt. mkdir makes a directory, so the example now has an empty homework directory. When the destination passed to mv is a directory, the command moves the file into it.

Wildcards

Wildcards let one command operate on many files.

The * wildcard matches any run of characters, so *.py means “every name ending in .py”:

$ ls *.py
hw1.py  hw10.py  hw2.py  hw3.py  lab02.py  lab03.py  lab04.py

The shell expands *.py into the list of matching names before ls ever runs, so ls simply receives all seven filenames as if you had typed them out. That expansion is done by the shell, not by ls, so *.py also works with cp, rm, and mv.

The ? wildcard matches exactly one character:

$ ls hw?.py
hw1.py  hw2.py  hw3.py

hw?.py matches hw1.py, hw2.py, and hw3.py, but not hw10.py, because ? stands for a single character and 10 is two of them.

Move all four homework files into homework with *:

$ mv hw*.py homework/
$ ls
homework  lab02.py  lab03.py  lab04.py  reminders.txt  webpage
$ ls homework
hw1.py  hw10.py  hw2.py  hw3.py

hw*.py matches all four names, and mv moves them into homework/. The command is the same whether the pattern matches four files or four thousand. Before using a wildcard with rm, run ls with the pattern and verify every match.

Redirection and pipes

Redirection controls where a command’s output goes.

By default output lands in the terminal, but > redirects it into a file instead:

$ ls *.py > files.txt
$ cat files.txt
lab02.py
lab03.py
lab04.py

A single > overwrites the file each time; use >> when you want to add to the end instead. You will use this operation to create a project’s .env file with echo "GROQ_API_KEY=..." > .env.

A pipe, |, sends one command’s output directly to another command as input, with no intermediate file:

$ cat *.py | wc -l
14

cat *.py prints all three lab files one after another, and | sends that output to wc -l, which counts lines. The result is the total number of lines across the three files: 14.

Count the files themselves the same way:

$ ls *.py | wc -l
3

ls *.py lists the three files, and wc -l counts the three lines of output.

Gluing tools together

grep searches its input for a pattern and prints every matching line. Use it to find leftover work:

$ grep TODO *.py
lab03.py:    # TODO: handle the empty string

grep found the single line marked TODO and reported its filename and text. To search a whole project, including its subdirectories, add -r for recursive and -n for line numbers:

$ grep -rn TODO .
./lab03.py:2:    # TODO: handle the empty string

. selects the current directory, and -r searches every file below it. This command reports the TODO in lab03.py on line 2.

Using the shell in later projects

Later projects use the shell for setup and execution. The docchat project begins with these commands:

$ mkdir docchat
$ cd docchat
$ git init
Initialized empty Git repository in /home/alice/docchat/.git/

git init starts tracking the directory with git, and pip3 install -r requirements.txt installs the project’s libraries. Even running your finished code is a shell command, the same python3 you have typed since week one:

$ python3 hw1.py
1

Study for the quiz on paper and practice the commands in a terminal.