Regular Expressions

Scraped pages often contain one phone number among thousands of unrelated characters. An exact string search cannot find a whole family of strings shaped like phone numbers. A regular expression (a regex) describes a text pattern so the computer can find, extract, or replace every matching string.

You wrote related patterns in the shell. When you typed ls *.txt, the * was a pattern that matched every filename ending in .txt, and topic_0?_* matched topic_00_... through topic_09_.... Those shell patterns are called globs. A regex can describe patterns in any text, not only filenames.

Regex has a well-earned reputation for looking like someone fell asleep on the keyboard.

From globs to regex

In a glob, * matches any run of characters and ? matches exactly one character, so ?.txt matches a.txt but not ab.txt. Regex uses different symbols for the same two ideas.

The translation is:

• The glob ? (any one character) becomes . in regex.
• The glob * (any run of characters) becomes .* in regex.

The shell glob *.txt therefore becomes the regex .*\.txt. The filename’s literal dot becomes \., because a bare . means any character in regex. A backslash turns off a metacharacter’s special meaning. Regex also has syntax for patterns such as a digit, three of these, and this or that. It can find a phone number or an email address, which no glob can do.

Searching for a pattern

Python’s regexes live in the built-in re module. re.search(pattern, text) looks for a pattern anywhere inside text. With no metacharacters, the pattern is an ordinary substring:

>>> import re
>>> re.search("cat", "the cat sat on the mat")
<re.Match object; span=(4, 7), match='cat'>
>>> re.search("dog", "the cat sat on the mat")
>>>

When the pattern is found, re.search returns a match object that reports where it matched. span=(4, 7) says the match runs from index 4 up to (but not including) 7, and match='cat' shows the exact text it found. When the pattern is not found, re.search returns None, which is why the second line printed nothing at all.

That match-or-None result works directly in an if, because a match object is truthy and None is falsy:

>>> if re.search("cat", "the cat sat"):
...     print("found it")
...
found it

The building blocks

A literal pattern only finds text you could have found with in or str.find. Metacharacters stand for a category of character rather than for themselves. The most useful one is \d, which matches any single digit:

>>> re.search(r"\d", "order #42 shipped")
<re.Match object; span=(7, 8), match='4'>

The pattern \d matched the 4 at index 7, the first digit, because re.search stops at the first match. The r in r"\d" marks a raw string, which tells Python to leave the backslash alone instead of reading \d as an escape sequence. Write regex patterns as raw strings to avoid conflicts between Python’s escapes and regex escapes.

One digit is rarely what we want; we usually want the whole number. For that we need a quantifier, a symbol that says how many of the previous thing to match. The quantifier + means one or more, so \d+ grabs the longest run of digits:

>>> re.search(r"\d+", "order #42 shipped")
<re.Match object; span=(7, 9), match='42'>

The match is 42, spanning indices 7 to 9, because + kept going as long as it saw digits. The main pieces are:

• Character classes stand for a category: \d a digit, \w a word character (letter, digit, or underscore), \s whitespace, and . any character at all.
• Custom classes in square brackets match any one character you list: [aeiou] matches a vowel, [a-z] any lowercase letter, and [^0-9] any character that is not a digit (a leading ^ inside the brackets negates the class).
• Quantifiers say how many of the thing before them: + is one or more, * is zero or more, ? is zero or one (optional), {3} is exactly three, and {3,5} is between three and five.
• Anchors pin the match to a position: ^ means the start of the string and $ the end, so ^\d+$ matches text that is all digits and nothing else.

A custom class can pull every vowel out of a word. The ? quantifier makes a character optional, so one pattern can match two spellings of a word:

>>> re.findall(r"[aeiou]", "regular")
['e', 'u', 'a']
>>> re.findall(r"colou?r", "color and colour")
['color', 'colour']

The u? says that a u may or may not be present, so colou?r matches both color and colour.

Extracting every match

re.search finds the first match. re.findall(pattern, text) returns a list of every matching substring. For example, a scraped contact page may contain several phone numbers:

>>> text = "call 909-621-8000 or 213-555-0199 today"
>>> re.findall(r"\d{3}-\d{3}-\d{4}", text)
['909-621-8000', '213-555-0199']

Read the pattern left to right: \d{3} is three digits, then a literal -, then \d{3}, another -, and \d{4} for the last four digits. findall returns both US phone numbers as a list.

Often we do not want the whole match, only one piece of it, like the area code. Wrap the piece you want in parentheses to make a group, and findall returns just the group:

>>> re.findall(r"(\d{3})-\d{3}-\d{4}", text)
['909', '213']

The parentheses around (\d{3}) mark the area code as the part to extract. The rest of the pattern still has to match, but findall returns only the text captured by the group.

A regex can also pull simplified email addresses from scraped text. This pattern asks for word characters, an @, more word characters, a dot, and a final run of word characters:

>>> re.findall(r"\w+@\w+\.\w+", "reach me at ada@example.com or grace@navy.mil")
['ada@example.com', 'grace@navy.mil']

The \. matches a literal dot. A bare . would match any character and accept text such as adaXexample. This simplified pattern rejects or misreads some valid email addresses, so use it only when it fits the text you have.

Cleaning text with substitution

re.sub(pattern, replacement, text) replaces every match with the replacement string and returns the new text. For example, it can redact every phone number in a document:

>>> re.sub(r"\d{3}-\d{3}-\d{4}", "[redacted]", text)
'call [redacted] or [redacted] today'

For more examples and details, see the Automate the Boring Stuff regex chapter and the Python re docs.

Why LLMs stumble

An LLM can produce a plausible regex that matches the wrong text. Test generated patterns against real examples before using them.

Determining what a pattern can and cannot match is a mathematical problem studied in formal-languages and complexity theory. Formal-languages and complexity theory are usually taught in upper-division computer science courses rather than coding tutorials.

Coding is not the same thing as computer science. Writing code lets you use regex; mathematics answers questions about why a regex behaves as it does, including which patterns can be checked efficiently.

Testing patterns

You do not need to memorize every metacharacter, and dense regex is difficult to read at a glance.

Build a pattern against real text and watch what it catches. A tester such as regex101.com highlights every match and explains each part of the pattern. Paste in the text you actually have, adjust the pattern until the highlights are exactly the strings you want, and then copy it into your re.findall call.