Text Cleanup at Scale with pandas String Methods and Regular Expressions

Dirty text data is one of the most common obstacles between you and a working analysis. You load a CSV of customer records and immediately see phone numbers formatted six different ways, product names with inconsistent capitalization, addresses stuffed with stray punctuation, and free-text comment fields that contain buried codes you need to extract. None of this will join cleanly to another table, aggregate meaningfully, or feed into a model until you fix it. The problem is that doing this manually — row by row, find-and-replace by find-and-replace — doesn't scale beyond a few hundred rows, and it certainly doesn't scale to the next time the data arrives with new variations of the same mess.

pandas gives you two overlapping toolsets for this: the .str accessor, which puts clean, vectorized string methods directly on a Series, and Python's re module (plus pandas' regex support), which lets you describe patterns in text rather than matching character-for-character. Together, they let you clean millions of rows as fast as you'd clean one. By the end of this lesson you'll be able to normalize inconsistent text fields, extract structured data from unstructured strings, and build reusable cleaning pipelines you can drop into any project.

What you'll learn:

• How the pandas .str accessor works and why it's faster than looping
• Core string methods for normalization: stripping, casing, replacing, and splitting
• How to read and write regular expressions for realistic text patterns
• Extracting structured data from messy strings with .str.extract() and .str.extractall()
• Filtering rows based on pattern matches with .str.contains() and .str.match()
• Chaining string operations into a reusable cleaning pipeline

---

Prerequisites

You should be comfortable loading data into a DataFrame and selecting columns. If you need a refresher, see Your First pandas DataFrame: Loading CSV and Excel Files and Exploring Data and Selecting and Filtering Data in pandas: loc, iloc, and Boolean Masks. Familiarity with Cleaning Messy Data with pandas: Missing Values, Duplicates, and Data Types is helpful because we'll be combining those techniques with string operations here.

---

The `.str` Accessor: Your Vectorized String Engine

When you have a column of text, your first instinct might be to loop over rows and call Python's built-in string methods on each value. That works, but it's slow, and it doesn't handle NaN values gracefully — you'll hit a AttributeError the moment Python tries to call .strip() on a float. The .str accessor solves both problems.

Every pandas Series with dtype=object (meaning text) exposes a .str namespace that maps Python's string methods to the entire column at once, vectorized under the hood, and NaN-safe by default.

import pandas as pd

# A realistic slice of a customer import file
data = {
    "customer_name": ["  Alice Moreau ", "BOB HENDERSON", "carol díaz", None, "  FRANK O'BRIEN"],
    "email": ["Alice.Moreau@Example.COM", "bob@example.com", "CAROL@example.COM", "frank@example.com", None],
    "phone": ["(617) 555-0142", "617.555.0199", "6175550177", None, "1-617-555-0188"],
    "account_status": ["Active", " active", "ACTIVE", "inactive", "Inactive "],
}

df = pd.DataFrame(data)
print(df.dtypes)
# customer_name    object
# email            object
# phone            object
# account_status   object

Now watch how clean the normalization is with .str:

# Strip whitespace from both ends, then title-case
df["customer_name"] = df["customer_name"].str.strip().str.title()

# Lowercase the entire email column
df["email"] = df["email"].str.lower()

# Normalize account_status: strip, lowercase
df["account_status"] = df["account_status"].str.strip().str.lower()

print(df[["customer_name", "email", "account_status"]])

Output:

   customer_name                     email account_status
0   Alice Moreau      alice.moreau@example.com         active
1  Bob Henderson           bob@example.com         active
2    Carol Díaz         carol@example.com         active
3           None             frank@example.com       inactive
4  Frank O'Brien                       None       inactive

Notice row 3: customer_name is None (the original NaN) and email returned a result — pandas propagated NaN through the chain silently. You didn't have to write a single if x is not None guard.

---

Core Normalization Methods You'll Use Every Day

Let's walk through the string methods you'll reach for most often, with realistic context for each one.

Stripping and Padding

# .strip() removes leading/trailing whitespace (also .lstrip(), .rstrip())
df["account_status"] = df["account_status"].str.strip()

# .zfill() left-pads with zeros — useful for ZIP codes stored as integers
zips = pd.Series(["2134", "90210", "704", "10001"])
zips_padded = zips.str.zfill(5)
# ['02134', '90210', '00704', '10001']

Case Normalization

s = pd.Series(["new york", "LOS ANGELES", "San Francisco", "CHICAGO"])

s.str.lower()        # all lowercase
s.str.upper()        # ALL CAPS
s.str.title()        # Title Case (first letter of every word)
s.str.capitalize()   # Only First Letter

For city and name fields, .str.title() looks attractive but has a quirk with apostrophes:

pd.Series(["o'brien"]).str.title()
# Returns: ["O'Brien"]  ← correct in this case
pd.Series(["mcdonald"]).str.title()
# Returns: ["Mcdonald"]  ← you'd want "McDonald"

For proper names with prefixes like "Mc" or "Mac," you'll eventually need regex replacement — which we'll cover shortly.

Finding and Replacing

.str.replace() is your Swiss Army knife. It accepts both literal strings and regex patterns.

# Remove all parentheses and dashes from phone numbers
df["phone_clean"] = (
    df["phone"]
    .str.replace(r"[\s\-\.\(\)]", "", regex=True)
)
# '(617) 555-0142' → '6175550142'
# '617.555.0199'   → '6175550199'
# '1-617-555-0188' → '16175550188'

Splitting Strings

# A column with pipe-delimited product codes
codes = pd.Series(["SKU-1042|RED|XL", "SKU-2891|BLUE|M", "SKU-0017|GREEN|S"])

# Split into a list — returns a Series of lists
codes.str.split("|")
# 0    [SKU-1042, RED, XL]
# 1    [SKU-2891, BLUE, M]
# 2    [SKU-0017, GREEN, S]

# Split and expand into separate columns directly
codes.str.split("|", expand=True)
#          0      1   2
# 0  SKU-1042    RED  XL
# 1  SKU-2891   BLUE   M
# 2  SKU-0017  GREEN   S

expand=True is one of the most useful flags in the entire pandas string API. Instead of getting a column of Python lists — which are awkward to work with — you get a proper DataFrame that you can rename and merge back in.

split_df = codes.str.split("|", expand=True)
split_df.columns = ["sku", "color", "size"]

---

Regular Expressions: Describing Patterns Instead of Values

At some point, simple character replacement isn't enough. You need to say "match a 10-digit sequence" or "find anything between parentheses" or "extract the dollar amount from this comment." That's when you need regular expressions.

Regex has a reputation for being cryptic, and the reputation is partially earned. But if you break it down into its building blocks, it becomes a readable language for describing text shapes.

The Building Blocks

Pattern	Meaning	Example
\d	Any digit 0–9	\d\d\d matches 617
\w	Any word character (letter, digit, underscore)	\w+ matches hello_123
\s	Any whitespace character	\s+ matches spaces, tabs
.	Any single character (except newline)	c.t matches cat, cut, cot
+	One or more of the preceding	\d+ matches 1, 42, 10001
*	Zero or more of the preceding	\d* matches `` or 123
?	Zero or one (makes something optional)	colou?r matches color or colour
{n}	Exactly n repetitions	\d{4} matches exactly 2024
{n,m}	Between n and m repetitions	\d{2,4} matches 17, 617, 6175
[abc]	Any one of these characters	[aeiou] matches any vowel
[^abc]	Any character NOT in the set	[^\d] matches non-digits
^	Start of string	^SKU matches strings starting with SKU
$	End of string	\.com$ matches strings ending in .com
(...)	Capture group — extract this part	(\d{3})-(\d{4}) captures two groups
|	OR — match either side	cat|dog matches cat or dog

Testing Patterns Before You Apply Them

Before you apply a regex to a 500,000-row DataFrame, test it on a small sample. The site regex101.com lets you paste your pattern and test strings interactively with real-time feedback. Alternatively, test in Python directly:

import re

pattern = r"\d{10}"
test_cases = ["6175550142", "(617) 555-0142", "617-555-0142", "not a phone"]

for t in test_cases:
    match = re.search(pattern, t)
    print(f"{t!r:30} → {'MATCH' if match else 'no match'}")

# '6175550142'                   → MATCH
# '(617) 555-0142'               → no match  (spaces and parens intervene)
# '617-555-0142'                 → no match  (dashes intervene)
# 'not a phone'                  → no match

This tells us the pattern \d{10} only matches after we've already stripped formatting characters — which is exactly how we'd use it in a pipeline.

---

Filtering Rows with Pattern Matching

Once you can describe patterns, you can use them to filter your DataFrame. The two methods for this are .str.contains() and .str.match().

`.str.contains()` — Does This Pattern Appear Anywhere?

This is the regex equivalent of SQL's LIKE '%pattern%'. It returns a boolean Series you can use directly as a filter mask.

# Sample support ticket data
tickets = pd.DataFrame({
    "ticket_id": range(1, 8),
    "subject": [
        "Cannot log in to account",
        "Billing question - duplicate charge",
        "Login page not loading",
        "URGENT: data export failed",
        "How do I change my password?",
        "billing error on invoice #4421",
        "Feature request: dark mode",
    ]
})

# Find all login-related tickets (case-insensitive)
login_mask = tickets["subject"].str.contains(r"log\s?in|login", case=False, na=False)
tickets[login_mask]

#    ticket_id                       subject
# 0          1          Cannot log in to account
# 2          3        Login page not loading

Notice na=False: this tells pandas to treat NaN values as False rather than propagating them as NaN in the boolean mask. Without it, you'll get NaN in your mask, and boolean indexing will raise an error.

# Find urgent tickets
urgent_mask = tickets["subject"].str.contains(r"^URGENT", case=False, na=False)

# Find billing tickets
billing_mask = tickets["subject"].str.contains(r"billing|invoice|charge", case=False, na=False)

# Combine masks
priority_tickets = tickets[urgent_mask | billing_mask]

This is exactly how you'd build a routing or triage system for incoming records.

`.str.match()` — Does the Pattern Match at the Start?

.str.match() anchors the pattern to the beginning of the string. It's equivalent to re.match() vs re.search().

# Only match tickets where subject starts with an action word
action_mask = tickets["subject"].str.match(r"(Cannot|How|URGENT)", case=False, na=False)

---

Extracting Structured Data from Messy Strings

This is where regex earns its keep. You have a column of freeform text, and buried inside it is structured information you need to pull out: order numbers, dollar amounts, dates, codes, identifiers. .str.extract() and .str.extractall() do this work.

`.str.extract()` — First Match, Named Groups

.str.extract() applies a regex with capture groups to each row and returns the captured groups as new columns. Use named groups ((?P<name>...)) to get meaningful column names automatically.

# Order notes from a legacy system
notes = pd.DataFrame({
    "note": [
        "Order #ORD-10042 shipped via UPS, tracking 1Z999AA10123456784",
        "Refund processed for order #ORD-10039, amount: $47.99",
        "Order #ORD-10051 on hold pending payment",
        "Manual adjustment: -$12.50 applied to ORD-10044",
        "Order #ORD-10047 delivered, customer confirmed",
    ]
})

# Extract the order number
order_pattern = r"(?P<order_id>ORD-\d+)"
notes[["order_id"]] = notes["note"].str.extract(order_pattern)

# Extract dollar amounts
amount_pattern = r"\$(?P<amount>[\d,]+\.?\d*)"
notes[["amount"]] = notes["note"].str.extract(amount_pattern)

print(notes[["note", "order_id", "amount"]])

Output:

                                                note  order_id  amount
0  Order #ORD-10042 shipped via UPS, tracking 1Z9...  ORD-10042    None
1  Refund processed for order #ORD-10039, amount:...  ORD-10039   47.99
2         Order #ORD-10051 on hold pending payment  ORD-10051    None
3  Manual adjustment: -$12.50 applied to ORD-10044  ORD-10044   12.50
4  Order #ORD-10047 delivered, customer confirmed  ORD-10047    None

When the pattern doesn't match, pandas fills the cell with NaN. This makes it trivial to identify which rows have the information you're looking for.

For multi-group extraction in a single pass:

# Extract both order ID and UPS tracking number in one call
complex_pattern = r"(?P<order_id>ORD-\d+).*?(?P<tracking>1Z[A-Z0-9]{16})?"
extracted = notes["note"].str.extract(complex_pattern)
print(extracted)

`.str.extractall()` — All Matches per Row

When a single row might contain multiple matches, use .str.extractall(). It returns a DataFrame with a MultiIndex where one level is the original row index and the other is the match number.

# Log file with multiple error codes per line
logs = pd.Series([
    "Process completed. Errors: E101, E204, E305",
    "No errors detected",
    "Warning: E101 occurred during step 3, then E402 at step 7",
])

error_pattern = r"(?P<error_code>E\d{3})"
all_errors = logs.str.extractall(error_pattern)
print(all_errors)

Output:

       error_code
  match           
0 0          E101
  1          E204
  2          E305
2 0          E101
  1          E402

This is invaluable for many-to-one data models: one log line has many errors, one order note has many SKUs, one comment has many hashtags. After extracting, you can group by the original index and count, or flatten with .reset_index() and join back to the source data.

# Count errors per log line
error_counts = all_errors.groupby(level=0)["error_code"].count()
print(error_counts)
# 0    3
# 2    2

---

Building a Reusable Cleaning Pipeline

Everything we've done so far is more valuable when organized into a function you can call every time a new data file arrives. If you've worked through Cleaning Messy Data with pandas: Missing Values, Duplicates, and Data Types, you already know the pattern: build modular functions, apply them in sequence, and document what each step does.

Here's a realistic pipeline for a customer data file:

import pandas as pd
import re

def normalize_phone(series: pd.Series) -> pd.Series:
    """
    Standardize phone numbers to 10-digit strings.
    Strips all non-digit characters, then removes leading country code 1.
    Returns NaN for anything that doesn't resolve to exactly 10 digits.
    """
    digits_only = series.str.replace(r"\D", "", regex=True)
    # Remove leading '1' if 11 digits
    stripped = digits_only.str.replace(r"^1(\d{10})$", r"\1", regex=True)
    # Null out anything not exactly 10 digits
    return stripped.where(stripped.str.fullmatch(r"\d{10}", na=False), other=pd.NA)


def format_phone(series: pd.Series) -> pd.Series:
    """Format clean 10-digit phone numbers as (XXX) XXX-XXXX."""
    return series.str.replace(
        r"(\d{3})(\d{3})(\d{4})",
        r"(\1) \2-\3",
        regex=True
    )


def normalize_email(series: pd.Series) -> pd.Series:
    """Lowercase and strip email addresses. Null out invalid ones."""
    cleaned = series.str.strip().str.lower()
    valid_mask = cleaned.str.fullmatch(
        r"[a-z0-9._%+\-]+@[a-z0-9.\-]+\.[a-z]{2,}",
        na=False
    )
    return cleaned.where(valid_mask, other=pd.NA)


def normalize_name(series: pd.Series) -> pd.Series:
    """Strip, title-case, and collapse internal whitespace."""
    return (
        series
        .str.strip()
        .str.title()
        .str.replace(r"\s{2,}", " ", regex=True)
    )


def clean_customer_data(df: pd.DataFrame) -> pd.DataFrame:
    """
    Apply all normalization steps to a raw customer DataFrame.
    Assumes columns: customer_name, email, phone, account_status
    """
    result = df.copy()
    result["customer_name"] = normalize_name(result["customer_name"])
    result["email"] = normalize_email(result["email"])
    result["phone_digits"] = normalize_phone(result["phone"])
    result["phone_formatted"] = format_phone(result["phone_digits"])
    result["account_status"] = result["account_status"].str.strip().str.lower()
    return result


# Apply to the DataFrame we built earlier
cleaned = clean_customer_data(df)
print(cleaned[["customer_name", "email", "phone_formatted", "account_status"]])

Output:

   customer_name                     email   phone_formatted account_status
0   Alice Moreau      alice.moreau@example.com  (617) 555-0142         active
1  Bob Henderson           bob@example.com  (617) 555-0199         active
2    Carol Díaz         carol@example.com  (617) 555-0177         active
3          None         frank@example.com             <NA>       inactive
4  Frank O'Brien                    <NA>  (617) 555-0188       inactive

This function-per-concern structure means when you get a new data file six months from now, you just call clean_customer_data(new_df). And when the phone number format changes, you edit one function.

---

Advanced Patterns for Real-World Scenarios

Let's cover a few patterns that come up constantly in actual data work.

Extracting Numbers from Mixed Text

Financial data often arrives with currency symbols, commas, and units baked in:

prices = pd.Series(["$1,249.99", "$89.00", "Free", "$12,000.00", "N/A"])

# Extract numeric value, then convert to float
numeric = (
    prices
    .str.extract(r"\$(?P<value>[\d,]+\.?\d*)", expand=False)
    .str.replace(",", "", regex=False)
    .astype(float)
)
print(numeric)
# 0     1249.99
# 1       89.00
# 2         NaN
# 3    12000.00
# 4         NaN

Parsing Delimited Tags or Categories

Many systems store multi-value fields as comma-separated or pipe-separated lists in a single column. This is a known data modeling problem and a common cleanup task that connects to data reshaping work you might do with Reshaping Data with pivot_table, melt, and stack in pandas.

products = pd.DataFrame({
    "product_id": [101, 102, 103],
    "tags": ["electronics, portable, bluetooth", "kitchen, appliance", "electronics, gaming, console, accessories"],
})

# Explode tags into individual rows for analysis
products["tags_list"] = products["tags"].str.split(r",\s*")
exploded = products.explode("tags_list").reset_index(drop=True)

# Count tag frequency
tag_counts = exploded["tags_list"].value_counts()
print(tag_counts)
# electronics    2
# accessories    1
# appliance      1
# bluetooth      1
# console        1
# gaming         1
# kitchen        1
# portable       1

From here, you can join back, pivot, or feed into groupby operations — all the aggregation work covered in Grouping and Aggregating in pandas: groupby as the PivotTable Replacement.

Conditional Replacement with `.str.replace()` and Back-References

Back-references in the replacement string let you rearrange the content you matched, not just delete it:

# Date strings stored as MM/DD/YYYY — convert to ISO format YYYY-MM-DD
dates = pd.Series(["03/15/2024", "11/02/2023", "07/04/2024", "invalid", None])

iso_dates = dates.str.replace(
    r"^(\d{2})/(\d{2})/(\d{4})$",
    r"\3-\1-\2",
    regex=True
)
print(iso_dates)
# 0    2024-03-15
# 1    2023-11-02
# 2    2024-07-04
# 3       invalid    ← no match, original preserved
# 4          None

The \1, \2, \3 refer to the first, second, and third capture groups. This is much more powerful than extracting and recombining — you can transform the structure of the value in a single operation. For dates specifically, you'd want to parse with pd.to_datetime() afterward, but this pattern is useful for any format rearrangement. If you're doing date work at scale, see Working with Dates and Time Series in pandas: Parsing, Resampling, and Rolling Windows for the full picture.

Normalizing Free-Text with Multiple Passes

Sometimes you need several passes because patterns interact. The trick is to build them as a sequence rather than trying to write one super-pattern:

def normalize_address(series: pd.Series) -> pd.Series:
    """Multi-pass normalization for US street addresses."""
    return (
        series
        # Strip leading/trailing whitespace
        .str.strip()
        # Collapse multiple spaces
        .str.replace(r" {2,}", " ", regex=True)
        # Standardize common abbreviations
        .str.replace(r"\bStreet\b", "St", regex=True, case=False)
        .str.replace(r"\bAvenue\b", "Ave", regex=True, case=False)
        .str.replace(r"\bBoulevard\b", "Blvd", regex=True, case=False)
        .str.replace(r"\bApartment\b|\bApt\.?\b", "Apt", regex=True, case=False)
        # Title case the whole thing
        .str.title()
    )

addresses = pd.Series([
    "  123 main STREET  ",
    "456 oak avenue, Apt. 7B",
    "789 SUNSET  boulevard",
    None,
])

print(normalize_address(addresses))
# 0            123 Main St
# 1    456 Oak Ave, Apt 7B
# 2      789 Sunset Blvd
# 3                  None

---

Performance Considerations

For datasets up to a few hundred thousand rows, .str methods are fast enough that you won't notice any difference between approaches. For very large DataFrames (millions of rows), a few principles help:

Compile regex patterns you use repeatedly:

import re

# If you're applying the same pattern in a loop or custom function,
# compile it once outside the loop
phone_pattern = re.compile(r"\D")

# Apply with a lambda — faster than string compilation on every call
# (Though .str.replace() compiles internally; this matters more with .apply())
df["phone"] = df["phone"].apply(lambda x: phone_pattern.sub("", x) if pd.notna(x) else x)

Prefer vectorized .str methods over .apply():

# Slower — Python loop under the hood
df["name"] = df["name"].apply(lambda x: x.strip().title() if isinstance(x, str) else x)

# Faster — vectorized C operations
df["name"] = df["name"].str.strip().str.title()

Profile before optimizing:

import time

n = 1_000_000
big_series = pd.Series(["  ALICE MOREAU  "] * n)

start = time.perf_counter()
result = big_series.str.strip().str.title()
end = time.perf_counter()
print(f"Vectorized: {end - start:.3f}s")

start = time.perf_counter()
result2 = big_series.apply(lambda x: x.strip().title() if isinstance(x, str) else x)
end = time.perf_counter()
print(f"Apply: {end - start:.3f}s")

# Typical result: vectorized is 2-4x faster at this scale

For truly massive datasets (hundreds of millions of rows), tools like Polars or Dask have more efficient string handling — but for most professional data workflows, pandas .str will be your workhorse.

---

Hands-On Exercise

You'll clean a realistic dataset of product records from a simulated e-commerce export. Create a new notebook and paste the following setup code:

import pandas as pd

raw_data = {
    "product_id": ["P-00101", "p00102", "P00103 ", " P-00104", "P00105"],
    "product_name": [
        "  wireless HEADPHONES (bluetooth)",
        "USB-C Charging Cable  ",
        "laptop stand - ADJUSTABLE",
        "Mechanical keyboard, RGB",
        "  WEBCAM 1080P hd  ",
    ],
    "price_raw": ["$49.99", "$12.00", "24.95", "$199.00 USD", "free"],
    "category_tags": [
        "electronics, audio, wireless",
        "electronics, accessories",
        "office, ergonomics, accessories",
        "electronics, input-devices, gaming",
        "electronics, video, accessories",
    ],
    "sku_code": [
        "SKU-WH-BT-001",
        "SKU-CABLE-UC-002",
        "Wrong format here",
        "SKU-KB-RGB-004",
        "SKU-CAM-HD-005",
    ],
}

df = pd.DataFrame(raw_data)

Your tasks:

1. Normalize product_id: All IDs should be uppercase, formatted as P-XXXXX (letter P, dash, exactly 5 digits). Use .str.replace() to insert the dash if missing, strip spaces, and uppercase.

2. Clean product_name: Strip whitespace, remove content in parentheses (including the parentheses), collapse extra spaces, and title-case the result.

3. Extract price: Extract the numeric value from price_raw, convert to float. Rows with non-numeric prices (like "free") should become NaN.

4. Explode category_tags: Split the comma-separated tags into individual rows, then compute the top 5 most common tags across all products.

5. Validate sku_code: Flag rows where sku_code does not match the pattern SKU-[A-Z]+-[A-Z]+-\d{3} (all uppercase letters between dashes, 3-digit suffix). Add a boolean column sku_valid.

6. Combine into a pipeline function called clean_product_data(df) that runs all steps and returns the cleaned DataFrame.

---

Common Mistakes & Troubleshooting

"My .str.replace() changed more than I expected"

You passed a regex pattern when you meant a literal string. The . character in regex matches any character, so replacing "U.S.A." as a regex will also match "USAAA". Use regex=False for literal strings, or escape with re.escape():

# Dangerous — '.' matches any character
series.str.replace("U.S.A.", "USA", regex=True)

# Safe
series.str.replace(re.escape("U.S.A."), "USA", regex=True)
# or just:
series.str.replace("U.S.A.", "USA", regex=False)

"I'm getting NaN where I don't expect it from .str.extract()"

If your pattern has a capture group and the row doesn't match, you get NaN. If your pattern has no capture group at all, you get an error. Double-check that your pattern has at least one (...) group, and test it against known-matching examples with re.search() before applying to the full column.

".str.contains() is raising a ValueError about NA values"

Add na=False (or na=True if you want to include rows with null values in the matched set):

# This will error if the column has NaN
mask = df["col"].str.contains(r"pattern")

# This treats NaN as non-matching
mask = df["col"].str.contains(r"pattern", na=False)

"My chained .str operations are slow on a large dataset"

Each .str call creates a new Series. Long chains on tens of millions of rows can accumulate memory pressure. Consider breaking the chain into intermediate assignments so Python can garbage-collect intermediates, or profile with %timeit in Jupyter to find the bottleneck.

"Title case is capitalizing words I don't want capitalized"

.str.title() capitalizes every word, including prepositions and articles ("Of", "The", "And"). For proper name formatting, you'd need a custom function or a library like titlecase. For most analytical work — normalizing city names, product categories — this is acceptable. Know when the visual precision matters.

"My regex works in regex101.com but not in pandas"

Check two things: (1) Are you using a raw string (r"...")? (2) Is the regex=True flag set on .str.replace() or .str.contains()? Also check whether your pattern relies on multiline mode (re.MULTILINE) — pandas .str methods operate on individual cell values, not multi-line documents, so ^ and $ match the start and end of the cell value, not individual lines within it.

---

Summary & Next Steps

You've built a complete toolkit for text cleanup at scale. The core pattern is: reach for .str methods first (they're readable, fast, and NaN-safe), escalate to regex when you need to match patterns rather than fixed values, and organize everything into reusable functions that form a reproducible pipeline.

The skills here combine naturally with the rest of your data analysis workflow. Text normalization is often the prerequisite for a join — you can't match customer names across two tables if one is title-cased and the other is all-caps. If you need a refresher on joining DataFrames after normalizing the key columns, see Joining DataFrames with pandas merge: SQL Joins and VLOOKUP in Python. Similarly, the tag-exploding pattern we covered leads naturally into groupby aggregations.

Specific techniques to revisit and practice:

• Building a multi-pass normalization function for one of your real datasets
• Using .str.extract() with named groups to pull structured data from a comment or notes field
• Writing a validator that uses .str.fullmatch() to audit data quality in an incoming file
• Profiling .str versus .apply() on a dataset you actually work with to see real performance differences

The best way to internalize regular expressions is to use them on patterns you care about. Take a messy text column from your own work, describe what "clean" looks like in English, then translate that description into a regex one piece at a time. Within a few sessions of this, the syntax stops feeling cryptic and starts feeling like a vocabulary you already know.