Regular Expressions

Scraped pages often contain one phone number among thousands of unrelated characters. An exact string search cannot find a whole family of strings shaped like phone numbers. A regular expression (a regex) describes a text pattern so the computer can find, extract, or replace every matching string.

You wrote related patterns in the shell. When you typed ls *.txt, the * was a pattern that matched every filename ending in .txt, and topic_0?_* matched topic_00_... through topic_09_.... Those shell patterns are called globs. A regex can describe patterns in any text, not only filenames.

Regex has a well-earned reputation for looking like someone fell asleep on the keyboard.

From Globs to regex

In a glob, * matches any run of characters and ? matches exactly one character, so ?.txt matches a.txt but not ab.txt. Regex uses different symbols for the same two ideas.

The translation is:

• The glob ? (any one character) becomes . in regex.
• The glob * (any run of characters) becomes .* in regex.

The shell glob *.txt therefore becomes the regex .*\.txt when matching the whole filename. The filename’s literal dot becomes \., because a bare . means any character in regex. A backslash turns off a metacharacter’s special meaning. Regex also has syntax for patterns such as a digit, three of these, and this or that. These pieces make structured text patterns easier to express than filename wildcards do.

Searching for a Pattern

Python’s regexes live in the built-in re module. re.search(pattern, text) looks for a pattern anywhere inside text. With no metacharacters, the pattern is an ordinary substring:

>>> import re
>>> re.search("cat", "the cat sat on the mat")
<re.Match object; span=(4, 7), match='cat'>
>>> re.search("dog", "the cat sat on the mat")
>>>

When the pattern is found, re.search returns a match object that reports where it matched. span=(4, 7) says the match runs from index 4 up to (but not including) 7, and match='cat' shows the exact text it found. When the pattern is not found, re.search returns None, which is why the second line printed nothing at all.

That match-or-None result works directly in an if, because a match object is truthy and None is falsy:

>>> if re.search("cat", "the cat sat"):
...     print("found it")
...
found it

Searching within text and checking an entire value are different jobs. re.fullmatch requires the whole string to fit the pattern:

>>> bool(re.search("cat", "the cat sat"))
True
>>> bool(re.fullmatch("cat", "the cat sat"))
False
>>> bool(re.fullmatch("cat", "cat"))
True

The search finds cat somewhere in the sentence; the full match rejects the surrounding words and spaces. Use a search to locate data and a full match to check the complete shape of one field.

The Building Blocks

A literal pattern only finds text you could have found with in or str.find. Metacharacters stand for a category of character rather than for themselves. The most useful one is \d, which matches any single digit:

>>> re.search(r"\d", "order #42 shipped")
<re.Match object; span=(7, 8), match='4'>

The pattern \d matched the 4 at index 7, the first digit, because re.search stops at the first match. The r in r"\d" marks a raw string, which tells Python to leave the backslash alone instead of reading \d as an escape sequence. Write regex patterns as raw strings to avoid conflicts between Python’s escapes and regex escapes.

One digit is rarely what we want; we usually want the whole number. For that we need a quantifier, a symbol that says how many of the previous thing to match. The quantifier + means one or more, so \d+ grabs the longest run of digits:

>>> re.search(r"\d+", "order #42 shipped")
<re.Match object; span=(7, 9), match='42'>

The match is 42, spanning indices 7 to 9, because + kept going as long as it saw digits. The main pieces are:

• Character classes stand for a category: \d a digit, \w a word character (letter, digit, or underscore), \s whitespace, and . any character except a newline by default.
• Custom classes in square brackets match any one character you list: [aeiou] matches a vowel, [a-z] any lowercase letter, and [^0-9] any character that is not a digit (a leading ^ inside the brackets negates the class).
• Quantifiers say how many of the thing before them: + is one or more, * is zero or more, ? is zero or one (optional), {3} is exactly three, and {3,5} is between three and five.
• Anchors pin the match to a position: ^ means the start of the string and $ the end (or just before a final newline). Use re.fullmatch when every character must belong to the match.

A custom class can pull every vowel out of a word. The ? quantifier makes a character optional, so one pattern can match two spellings of a word:

>>> re.findall(r"[aeiou]", "regular")
['e', 'u', 'a']
>>> re.findall(r"colou?r", "color and colour")
['color', 'colour']

The u? says that a u may or may not be present, so colou?r matches both color and colour. The quantifier applies only to the piece immediately before it: in ab+, the b repeats, while (ab)+ repeats the pair. Python’s \d and \w include Unicode characters; use [0-9] when you specifically need the ten ASCII digits.

Extracting Every Match

re.search finds the first match. re.findall(pattern, text) returns a list of every matching substring. For example, a scraped contact page may contain several phone numbers:

>>> text = "call 909-621-8000 or 213-555-0199 today"
>>> re.findall(r"\d{3}-\d{3}-\d{4}", text)
['909-621-8000', '213-555-0199']

Read the pattern left to right: \d{3} is three digits, then a literal -, then \d{3}, another -, and \d{4} for the last four digits. findall returns both US phone numbers as a list.

Often we do not want the whole match, only one piece of it, like the area code. Wrap the piece you want in parentheses to make a group, and findall returns just the group:

>>> re.findall(r"(\d{3})-\d{3}-\d{4}", text)
['909', '213']

The parentheses around (\d{3}) mark the area code as the part to extract. The rest of the pattern still has to match, but findall returns only the text captured by the group.

With two capturing groups, each result is a tuple, an ordered grouping of the two captured strings:

>>> re.findall(r"(\d{3})-(\d{3}-\d{4})", text)
[('909', '621-8000'), ('213', '555-0199')]

Adding parentheses can therefore change the shape of the returned data, even when the same phone numbers match. Check whether the next step expects whole strings, one captured field, or tuples of fields.

A regex can also pull simplified email addresses from scraped text. This pattern asks for word characters, an @, more word characters, a dot, and a final run of word characters:

>>> re.findall(r"\w+@\w+\.\w+", "reach me at ada@example.com or grace@navy.mil")
['ada@example.com', 'grace@navy.mil']

The \. matches a literal dot. A bare . would match any character and accept text such as adaXexample. This simplified pattern rejects or misreads some valid email addresses, so use it only when it fits the text you have.

Cleaning Text with Substitution

re.sub(pattern, replacement, text) replaces every match with the replacement string and returns the new text. For example, it can redact every phone number in a document:

>>> re.sub(r"\d{3}-\d{3}-\d{4}", "[redacted]", text)
'call [redacted] or [redacted] today'

For more examples and details, see the Automate the Boring Stuff regex chapter and the Python re docs.

Why LLMs Stumble

An LLM can produce a plausible regex that matches the wrong text. Test generated patterns against real examples before using them.

Determining what a pattern can and cannot match is a mathematical problem studied in formal-languages and complexity theory, usually in upper-division computer science courses rather than coding tutorials.

Writing code lets you use regex; mathematics explains why a regex behaves as it does, including which patterns can be checked efficiently.

Testing Patterns

You do not need to memorize every metacharacter, and dense regex is difficult to read at a glance.

Build a pattern against real text and watch what it catches. A tester such as regex101.com highlights every match and explains each part of the pattern. Paste in the text you actually have, adjust the pattern until the highlights are exactly the strings you want, and then copy it into your re.findall call. Include near misses as well as successes: a phone number with too few digits, different punctuation, an empty string, and extra text before or after it. Then run those examples with Python’s re module, since regex tools and languages do not all implement the same rules.