When your dataset won't fit in memory, the tools that worked on small data start to fail. This expert-level lesson teaches you chunked reading, dtype optimization, Parquet conversion, and Polars — a complete toolkit for handling files that break normal pandas workflows.
You've built a solid pandas workflow. You can load CSVs, clean columns, run groupby aggregations, and spit out formatted Excel reports. Then one Monday morning, your data team drops a 4 GB transaction log on your desk and asks you to "run your usual analysis on it." You double-click the file, fire up your Jupyter notebook, call pd.read_csv(), and wait. And wait. Then Python runs out of memory and crashes.
This isn't a Python problem. It's a scale problem — and scale problems have real, learnable solutions. The tools and techniques that work perfectly on a 50,000-row dataset start to buckle around 1 million rows and fall apart entirely by 10 million. But here's the thing: most large-dataset challenges are solved not by buying more RAM or switching to a different language, but by being smarter about how you load, represent, and process data. Chunked reading, dtype optimization, and modern dataframe libraries like Polars are the three pillars of that smarter approach.
By the end of this lesson, you'll have a genuine toolkit for working with datasets that won't fit comfortably in memory. You'll understand not just what to do but why these techniques work at the level of memory layout and CPU architecture.
What you'll learn:
This is an expert-level lesson. You should already be comfortable with:
You should also have a working Python environment with pandas installed. If you need to set one up, see Setting Up Python for Data Analysis.
Before we fix the problem, let's understand it precisely. When pandas reads a CSV file, it doesn't stream data — it slurps the entire file into RAM and builds a DataFrame. That's convenient for small files, but it has a multiplier effect you might not expect: a CSV that's 500 MB on disk typically occupies 3–8x that amount once loaded into a pandas DataFrame.
Why? Three reasons:
1. Python object overhead. When pandas reads a column with mixed types or strings, it defaults to the object dtype, which stores a Python string object per cell. A Python string object carries 50+ bytes of overhead regardless of the string's actual content. A column with 1 million short strings like "APAC" or "EMEA" can consume 50–80 MB just from object overhead, when the actual data would fit in a few megabytes.
2. Numeric type defaults. By default, pandas reads integer columns as int64 (8 bytes per value) and float columns as float64 (8 bytes per value). If your column contains values that only go from 0 to 100, you're using 8 bytes to store something that would fit in 1 byte (uint8). That's an 8x waste.
3. No lazy evaluation. Pandas evaluates eagerly. Every transformation you apply creates an intermediate copy of data in memory unless you're very deliberate about in-place operations. A chain of five operations on a 2 GB DataFrame might need 10 GB of working memory.
Together, these factors mean that a file Python can "technically" load might still kill your analysis because the working memory required during processing exceeds what your machine has available.
Key insight
Memory is the primary bottleneck for large dataset work, not CPU speed. The techniques in this lesson are all fundamentally about using memory more efficiently — reading less of it at once, storing it more compactly, and processing it in smarter patterns.
Let's see exactly what's happening before we fix it. Here's how to measure your DataFrame's actual memory usage:
import pandas as pd
df = pd.read_csv("transactions.csv")
# Get memory by column
memory_by_col = df.memory_usage(deep=True)
print(memory_by_col)
print(f"\nTotal memory: {memory_by_col.sum() / 1e6:.1f} MB")
# Alternatively, a quick summary
print(df.info(memory_usage='deep'))
The deep=True argument is critical. Without it, pandas reports shallow memory — the size of the array pointers, not the actual objects they point to. On object columns, shallow reporting can undercount by a factor of 10 or more. Always use deep=True when you're diagnosing memory issues.
The single highest-leverage intervention for most large DataFrames is dtype optimization. This doesn't require any architectural changes to your code — you're still using pd.read_csv() — but you give pandas explicit instructions about how to represent each column, instead of letting it guess.
Here's a practical mental model of numeric dtypes by memory cost:
| dtype | bytes/value | range |
|---|---|---|
bool |
1 | True/False |
uint8 |
1 | 0 to 255 |
int8 |
1 | -128 to 127 |
uint16 |
2 | 0 to 65,535 |
int16 |
2 | -32,768 to 32,767 |
uint32 |
4 | 0 to 4.3B |
int32 |
4 | -2.1B to 2.1B |
float32 |
4 | ~7 significant digits |
int64 |
8 | very large integers |
float64 |
8 | ~15 significant digits |
object |
50+ | Python string or mixed |
By default, pandas uses int64 for integers and float64 for floats. The rightmost column in a transaction log that holds values like 1, 2, 3 for region codes doesn't need 8 bytes. It needs 1.
For string columns with low cardinality (meaning they repeat the same values many times), the category dtype is transformative. Instead of storing the string "North America" 500,000 times, a categorical column stores it once in a lookup table and records just an integer index per row.
Let's walk through a realistic example. Suppose you have a sales transactions file with these columns: transaction_id, date, region, product_category, sales_rep_id, amount, quantity, is_returned.
import pandas as pd
import numpy as np
# First, load a sample to understand the data
sample = pd.read_csv("transactions.csv", nrows=10000)
print(sample.dtypes)
print("\nColumn cardinalities:")
for col in sample.select_dtypes(include='object').columns:
print(f" {col}: {sample[col].nunique()} unique values")
Output might look like:
transaction_id int64
date object
region object
product_category object
sales_rep_id int64
amount float64
quantity int64
is_returned object
Column cardinalities:
date: 1095 unique values
region: 6 unique values
product_category: 24 unique values
is_returned: 2 unique values
Now you have the information you need. region has 6 unique values — perfect candidate for category. product_category has 24 — same. is_returned has 2 — that's a boolean. date has 1095 — still worth parsing as datetime rather than storing as strings. quantity and sales_rep_id are integers that probably don't need 8 bytes each.
# Define dtypes explicitly
dtypes = {
'transaction_id': 'int32', # large enough for ~2B transactions
'region': 'category',
'product_category': 'category',
'sales_rep_id': 'int32',
'amount': 'float32', # 7 sig figs is enough for dollar amounts
'quantity': 'int16', # reasonable upper bound ~32K
'is_returned': 'bool',
}
# Load with explicit dtypes and parse dates in one shot
df = pd.read_csv(
"transactions.csv",
dtype=dtypes,
parse_dates=['date'],
)
print(df.info(memory_usage='deep'))
Let's benchmark this properly. On a 5-million-row version of this file:
import time
# Naive load
t0 = time.time()
df_naive = pd.read_csv("transactions_5m.csv")
t1 = time.time()
naive_mem = df_naive.memory_usage(deep=True).sum() / 1e6
print(f"Naive load: {t1-t0:.1f}s, {naive_mem:.0f} MB")
# Optimized load
t0 = time.time()
df_opt = pd.read_csv("transactions_5m.csv", dtype=dtypes, parse_dates=['date'])
t1 = time.time()
opt_mem = df_opt.memory_usage(deep=True).sum() / 1e6
print(f"Optimized load: {t1-t0:.1f}s, {opt_mem:.0f} MB")
print(f"Memory reduction: {(1 - opt_mem/naive_mem)*100:.0f}%")
Typical results: naive load uses 1,800–2,200 MB; optimized load uses 350–450 MB. That's a 75–80% reduction without losing any data fidelity. The load time often improves as well because pandas spends less time allocating memory.
Warning
Be careful with float32 for financial calculations. If you're summing millions of small floating point values, rounding errors accumulate faster with 32-bit precision. Use float32 for reading and analysis, but convert to float64 before performing cumulative sums or other precision-sensitive aggregations.
Manually auditing every column gets tedious for wide files. Here's a utility function that inspects a sample and suggests optimal dtypes:
def suggest_dtypes(filepath: str, sample_rows: int = 50000) -> dict:
"""
Read a sample of a CSV and suggest memory-efficient dtypes.
Returns a dict you can pass directly to pd.read_csv(dtype=...).
"""
sample = pd.read_csv(filepath, nrows=sample_rows)
suggestions = {}
for col in sample.columns:
col_data = sample[col]
if col_data.dtype == 'object':
n_unique = col_data.nunique()
n_total = len(col_data.dropna())
cardinality_ratio = n_unique / n_total if n_total > 0 else 1
if n_unique == 2 and set(col_data.dropna().unique()) <= {'True', 'False', '0', '1', 'Y', 'N'}:
suggestions[col] = 'bool'
elif cardinality_ratio < 0.5: # Less than 50% unique values
suggestions[col] = 'category'
# else: leave as object (high cardinality text)
elif col_data.dtype in ['int64', 'int32']:
col_min = col_data.min()
col_max = col_data.max()
if col_min >= 0:
if col_max <= 255:
suggestions[col] = 'uint8'
elif col_max <= 65535:
suggestions[col] = 'uint16'
elif col_max <= 4294967295:
suggestions[col] = 'uint32'
else:
if col_min >= -128 and col_max <= 127:
suggestions[col] = 'int8'
elif col_min >= -32768 and col_max <= 32767:
suggestions[col] = 'int16'
elif col_min >= -2147483648 and col_max <= 2147483647:
suggestions[col] = 'int32'
elif col_data.dtype == 'float64':
suggestions[col] = 'float32'
return suggestions
# Usage
dtype_suggestions = suggest_dtypes("transactions.csv")
print("Suggested dtypes:")
for col, dtype in dtype_suggestions.items():
print(f" {col}: {dtype}")
Tip
Always validate dtype suggestions on your actual full file before using them in production. A sample might not contain the full range of values. A column that shows max value of 200 in 50K rows might hit 70,000 in the full dataset, which would overflow uint8. Add range checks or catch OverflowError exceptions.
Even with perfect dtype optimization, some files are simply too large to load entirely. A 50 GB log file isn't going to fit in 16 GB of RAM no matter how efficiently you represent the data. For these cases, you need chunked reading — reading the file in manageable pieces and processing each piece before loading the next.
pd.read_csv() accepts a chunksize parameter that transforms it from a function that returns a DataFrame into a function that returns an iterator of DataFrames:
chunk_iterator = pd.read_csv("large_transactions.csv", chunksize=100_000)
for chunk in chunk_iterator:
# chunk is a regular DataFrame with 100,000 rows
# process it here
print(f"Processing chunk with shape: {chunk.shape}")
Each chunk is a complete, normal pandas DataFrame — you can filter, aggregate, join, and transform it exactly as you would any other DataFrame. The trick is designing your processing logic to work incrementally, accumulating results as you go rather than needing access to all data simultaneously.
The most common use case: you have a huge file but only need a subset of rows. Process each chunk and keep only the relevant rows:
import pandas as pd
target_region = "EMEA"
filtered_chunks = []
for chunk in pd.read_csv("transactions.csv", chunksize=200_000, dtype=dtypes, parse_dates=['date']):
mask = chunk['region'] == target_region
if mask.any():
filtered_chunks.append(chunk[mask])
# Combine all the filtered pieces
df_emea = pd.concat(filtered_chunks, ignore_index=True)
print(f"EMEA transactions: {len(df_emea):,}")
This pattern works beautifully when the result set is much smaller than the input. Even if the input is 50 GB, if you're filtering to 5% of rows, the result fits in memory comfortably.
Warning
Don't concatenate within the loop — that's the "concat in a loop" anti-pattern. Each pd.concat() call creates a new DataFrame object, so doing it N times creates N intermediate copies. Collect chunks in a list, then call pd.concat() once at the end. This is the difference between O(n²) and O(n) memory behavior.
For summary statistics, you often don't need to keep any raw data at all. You just need to accumulate intermediate aggregation values:
import pandas as pd
# Accumulate aggregation results across chunks
chunk_summaries = []
for chunk in pd.read_csv(
"transactions.csv",
chunksize=200_000,
dtype=dtypes,
parse_dates=['date']
):
# Aggregate this chunk
chunk_agg = (
chunk
.groupby(['region', 'product_category'], observed=True)
.agg(
total_amount=('amount', 'sum'),
transaction_count=('transaction_id', 'count'),
avg_quantity=('quantity', 'mean'),
)
.reset_index()
)
chunk_summaries.append(chunk_agg)
# Combine chunk summaries and re-aggregate
# This works because sum of sums = total sum
combined = pd.concat(chunk_summaries, ignore_index=True)
final_summary = (
combined
.groupby(['region', 'product_category'], observed=True)
.agg(
total_amount=('total_amount', 'sum'),
transaction_count=('transaction_count', 'sum'),
# Weighted average: we need to recompute this properly
)
.reset_index()
)
print(final_summary)
This pattern is powerful because the memory footprint of chunk_summaries is tiny — you're storing aggregated rows, not raw data. A list of 50 chunk summaries, each with ~150 rows (6 regions × 24 categories + some empty combos), occupies almost nothing.
Key insight
The critical insight for incremental aggregation is: not all statistics compose cleanly. Sums of sums equal the total sum. Counts of counts equal the total count. But the mean of means does NOT equal the overall mean if chunks have different sizes. For means, either accumulate the sum and count separately and divide at the end, or keep raw data. For medians, percentiles, and other order statistics, chunked exact computation is genuinely difficult — you'll need approximate methods or a different tool.
Here's the correct pattern for computing overall means through chunked aggregation:
# Accumulate sum and count separately, then compute mean at the end
running_sum = {}
running_count = {}
for chunk in pd.read_csv("transactions.csv", chunksize=200_000, dtype=dtypes):
for region in chunk['region'].cat.categories:
region_data = chunk[chunk['region'] == region]['amount']
if region not in running_sum:
running_sum[region] = 0
running_count[region] = 0
running_sum[region] += region_data.sum()
running_count[region] += len(region_data)
# Compute true mean
true_means = {region: running_sum[region] / running_count[region]
for region in running_sum}
print(true_means)
Sometimes the output is also large. Rather than accumulating results in memory, write each processed chunk directly to an output file:
import pandas as pd
import os
output_path = "transactions_emea_enriched.csv"
write_header = True # only write column names on first chunk
for i, chunk in enumerate(pd.read_csv(
"transactions.csv",
chunksize=200_000,
dtype=dtypes,
parse_dates=['date']
)):
# Filter
chunk = chunk[chunk['region'] == 'EMEA'].copy()
# Transform
chunk['amount_eur'] = chunk['amount'] * 0.92
chunk['year_month'] = chunk['date'].dt.to_period('M').astype(str)
# Write to output (append mode after first chunk)
chunk.to_csv(
output_path,
mode='w' if write_header else 'a',
header=write_header,
index=False,
)
write_header = False
if i % 10 == 0:
size_mb = os.path.getsize(output_path) / 1e6
print(f"Chunk {i}: output file is {size_mb:.1f} MB")
print("Done. Processing complete.")
This is a true streaming pipeline: constant memory usage regardless of input file size. The only memory you ever hold is one chunk at a time.
There's no universally correct chunk size. The right value depends on:
A useful starting heuristic: aim for chunks that occupy 200–500 MB of RAM each. You can estimate this from your sample analysis earlier — if 50,000 rows uses 50 MB, then 500,000 rows per chunk would use 500 MB. That leaves room for intermediate computations while staying well within typical memory limits.
For the date and time series operations we often do during data cleaning, be aware that parse_dates adds some memory overhead — parsed datetime objects are stored as datetime64[ns], which is 8 bytes per value, about the same as int64.
Before we get to Polars, there's a middle step that dramatically improves your situation if you work with the same large file repeatedly: stop using CSV.
CSV is a terrible format for large data. It stores everything as text, has no type information, can't be partially read without scanning from the start, and offers no compression. Parquet is the opposite on every dimension.
import pandas as pd
# One-time conversion: read CSV, write Parquet
df = pd.read_csv("transactions.csv", dtype=dtypes, parse_dates=['date'])
df.to_parquet("transactions.parquet", engine='pyarrow', compression='snappy')
# Subsequent reads are dramatically faster and smaller
df_fast = pd.read_parquet("transactions.parquet")
Benchmarks on a 5-million-row transaction file:
| Format | File size | Read time |
|---|---|---|
| CSV (uncompressed) | 680 MB | 18.3s |
| CSV (gzip) | 145 MB | 22.1s |
| Parquet (snappy) | 89 MB | 1.4s |
| Parquet (zstd) | 71 MB | 1.6s |
Parquet reads 12x faster because:
Tip
If you're reading a large file more than once, always invest the time to convert it to Parquet on first read. The one-time conversion cost pays for itself on the second read. For datasets you're loading repeatedly in automated pipelines, the difference compounds dramatically over time.
Parquet also supports column pruning — you can read only the columns you need:
# Only read what you actually use
df = pd.read_parquet(
"transactions.parquet",
columns=['date', 'region', 'amount', 'product_category']
)
This is much faster than reading a 40-column file and then dropping 36 columns.
We've been optimizing how we use pandas. Now let's talk about when it makes sense to use something different entirely.
Polars is a DataFrame library written in Rust, with Python bindings, built from scratch for performance. It's not a replacement for pandas in all cases — but for specific workloads, it's dramatically faster.
Three architectural differences explain most of Polars' speed advantage:
1. True parallelism. Pandas is single-threaded for almost all operations. Polars automatically parallelizes operations across all CPU cores. On a modern 8-core laptop, operations that don't require inter-row state can run 6–8x faster just from parallelism.
2. Lazy evaluation. Polars has a lazy execution mode where you describe a query plan and Polars optimizes it before running anything. It can push filters down to the scan layer (only reading matching rows from disk), reorder operations, eliminate redundant work, and predicate-push down into Parquet files.
3. Memory efficiency. Polars uses Arrow-format memory layout natively, which is cache-friendly and avoids Python object overhead entirely. There's no object dtype in Polars — strings are stored as Arrow arrays.
Install it:
pip install polars
The API is similar to pandas but not identical. If you're coming from pandas, the main mental shift is:
pl.col("column_name") expressions instead of bracket notation in most contexts.lazy()) for most workimport polars as pl
# Reading a CSV — immediately faster than pandas for large files
df = pl.read_csv("transactions.csv")
# Basic operations feel familiar
print(df.shape)
print(df.dtypes)
print(df.head())
# Filtering
df_emea = df.filter(pl.col("region") == "EMEA")
# Aggregation
summary = (
df
.group_by(["region", "product_category"])
.agg([
pl.col("amount").sum().alias("total_amount"),
pl.col("transaction_id").count().alias("n_transactions"),
pl.col("amount").mean().alias("avg_amount"),
])
.sort("total_amount", descending=True)
)
print(summary)
The lazy API is where Polars truly shines for large datasets:
import polars as pl
# Build a lazy query — nothing executes yet
query = (
pl.scan_csv("transactions.csv") # lazy scan, reads nothing yet
.filter(pl.col("region").is_in(["EMEA", "APAC"]))
.filter(pl.col("amount") > 100)
.with_columns([
pl.col("amount").cast(pl.Float32).alias("amount"),
pl.col("date").str.to_date().alias("date"),
])
.group_by(["region", "product_category"])
.agg([
pl.col("amount").sum().alias("total_amount"),
pl.col("transaction_id").len().alias("n_transactions"),
])
)
# Show the query plan — Polars explains what it will do
print(query.explain(optimized=True))
# Execute the query
result = query.collect()
print(result)
The explain() output shows you the optimized execution plan. You'll see that Polars pushes the filter operations all the way down to the CSV scan — it avoids reading rows that don't match the filter from the start, rather than reading everything and then filtering.
For Parquet files, this is even more powerful:
# Polars can read only the relevant row groups from Parquet
query = (
pl.scan_parquet("transactions.parquet")
.filter(pl.col("region") == "EMEA")
.filter(
pl.col("date").is_between(
pl.lit("2023-01-01").str.to_date(),
pl.lit("2023-12-31").str.to_date(),
)
)
.select(["date", "region", "product_category", "amount"])
.collect()
)
When the Parquet file has appropriate row group ordering (e.g., sorted by date), Polars can skip entire row groups that don't match the date filter without reading them at all.
Let's compare on a concrete task: reading a 10-million-row CSV, filtering to two regions, grouping by region and product category, and computing sum and count of amounts.
import pandas as pd
import polars as pl
import time
# pandas
t0 = time.time()
df_pd = pd.read_csv("transactions_10m.csv")
df_pd = df_pd[df_pd['region'].isin(['EMEA', 'APAC'])]
result_pd = df_pd.groupby(['region', 'product_category'])['amount'].agg(['sum', 'count'])
t1 = time.time()
print(f"pandas: {t1 - t0:.2f}s")
# Polars lazy
t0 = time.time()
result_pl = (
pl.scan_csv("transactions_10m.csv")
.filter(pl.col("region").is_in(["EMEA", "APAC"]))
.group_by(["region", "product_category"])
.agg([
pl.col("amount").sum().alias("total_amount"),
pl.col("amount").len().alias("count"),
])
.collect()
)
t1 = time.time()
print(f"Polars: {t1 - t0:.2f}s")
Typical results on a modern laptop with 8 cores:
That's a 6–8x speedup — and it gets more pronounced as file size grows because Polars scales better with core count.
Note
Polars benchmarks vary significantly with hardware. On machines with more cores, Polars advantages compound further. On single-core virtual machines (common in some cloud batch jobs), the gap narrows. Always benchmark on your actual target environment.
Polars isn't always the right choice. Use pandas when:
.str accessor has more methods, more mature regex support, and better integration with Python string libraries. See Text Cleanup at Scale with pandas for the full picture.Use Polars when:
You don't have to go all-in on Polars. It's perfectly reasonable to use Polars for the heavy lifting (reading and initial aggregation) and then convert to pandas for downstream steps:
import polars as pl
import pandas as pd
# Heavy initial processing in Polars
monthly_summary_pl = (
pl.scan_parquet("transactions.parquet")
.filter(pl.col("year") == 2023)
.group_by(["region", pl.col("date").dt.month().alias("month")])
.agg(pl.col("amount").sum().alias("monthly_revenue"))
.sort(["region", "month"])
.collect()
)
# Convert to pandas for visualization or Excel output
df_pd = monthly_summary_pl.to_pandas()
# Continue with pandas ecosystem tools
df_pd.plot(...) # matplotlib
# or
df_pd.to_excel(...) # openpyxl
This hybrid approach captures most of Polars' speed advantages while keeping access to the full pandas ecosystem.
Let's build a complete, realistic pipeline that combines all three techniques: dtype optimization, Parquet conversion, and Polars for analysis.
Scenario: You receive a monthly drop of 8 GB CSV files from a payment processor. You need to produce a summary report by region and product category, flag unusual transaction amounts, and output a formatted summary.
import polars as pl
import pandas as pd
import pathlib
from datetime import datetime
def process_monthly_transactions(
csv_path: str,
output_dir: str = "output",
force_reconvert: bool = False
) -> pd.DataFrame:
"""
Full pipeline for monthly transaction processing.
1. Converts CSV to Parquet on first run (cached for future runs)
2. Uses Polars lazy API for efficient aggregation
3. Returns a pandas DataFrame for downstream reporting
"""
csv_path = pathlib.Path(csv_path)
parquet_path = pathlib.Path(output_dir) / csv_path.with_suffix('.parquet').name
pathlib.Path(output_dir).mkdir(exist_ok=True)
# Step 1: Convert to Parquet if not already done
if not parquet_path.exists() or force_reconvert:
print(f"Converting {csv_path.name} to Parquet...")
# Use pandas for initial type-safe conversion in chunks
dtypes = {
'transaction_id': 'int32',
'region': 'category',
'product_category': 'category',
'sales_rep_id': 'int32',
'amount': 'float32',
'quantity': 'int16',
'is_returned': 'bool',
}
# Chunked conversion: read CSV in chunks, write to Parquet
# (Polars can also do this directly, shown below as alternative)
chunks = []
for chunk in pd.read_csv(csv_path, chunksize=300_000, dtype=dtypes, parse_dates=['date']):
chunks.append(chunk)
df_full = pd.concat(chunks, ignore_index=True)
df_full.to_parquet(parquet_path, engine='pyarrow', compression='snappy', index=False)
parquet_size_mb = parquet_path.stat().st_size / 1e6
print(f"Parquet written: {parquet_size_mb:.0f} MB")
del df_full, chunks # explicitly free memory
else:
print(f"Using cached Parquet: {parquet_path}")
# Step 2: Analysis with Polars lazy API
print("Running analysis...")
# Main summary aggregation
summary = (
pl.scan_parquet(str(parquet_path))
.filter(pl.col("is_returned") == False)
.group_by(["region", "product_category"])
.agg([
pl.col("amount").sum().alias("total_revenue"),
pl.col("amount").mean().alias("avg_transaction"),
pl.col("amount").std().alias("std_transaction"),
pl.col("transaction_id").len().alias("n_transactions"),
pl.col("quantity").sum().alias("total_units"),
])
.with_columns([
# Flag categories where std dev > 2x the mean (high variance)
(pl.col("std_transaction") > pl.col("avg_transaction") * 2)
.alias("high_variance"),
# Revenue share within each region (computed after groupby)
])
.sort(["region", "total_revenue"], descending=[False, True])
.collect()
)
# Add revenue share per region (requires a window operation)
summary = summary.with_columns(
(pl.col("total_revenue") / pl.col("total_revenue").sum().over("region"))
.alias("revenue_share_in_region")
)
print(f"Analysis complete. Summary has {len(summary)} rows.")
# Step 3: Convert to pandas for reporting
return summary.to_pandas()
# Run the pipeline
if __name__ == "__main__":
result = process_monthly_transactions(
csv_path="data/transactions_2024_01.csv",
output_dir="data/parquet_cache"
)
# Downstream: use pandas for Excel reporting
print(result.head(20))
print(f"\nTotal revenue: ${result['total_revenue'].sum():,.0f}")
This pipeline handles a 8 GB CSV file gracefully:
Work through this exercise to cement the techniques from this lesson.
Setup: Download the NYC Yellow Taxi trip data for any one month (the Parquet files are ~50 MB; the CSV equivalents are ~500 MB). If you prefer to work with a CSV, use the CSV version.
Task: Build a pipeline that answers these questions about the taxi data:
Requirements:
Expected outcomes:
This happens when cardinality is too high. If a column has 1 million unique values out of 1.1 million rows, converting to category actually adds overhead (the lookup table is almost as large as the data itself). Use category only when a column repeats the same values many times across many rows. A rough rule: nunique() / len() should be below 0.5, and ideally below 0.1.
Almost always a windowed/ranking operation problem. Operations like percentile ranks, cumulative sums, or anything requiring knowledge of the full dataset cannot be computed correctly chunk by chunk. Either filter to a manageable size first, or use an approximation algorithm (like reservoir sampling for medians or HyperLogLog for cardinality estimates).
Check three things: (1) NaN handling — Polars and pandas treat null values differently in some operations. (2) Sort stability — group_by in Polars doesn't guarantee row order; use .sort() explicitly. (3) Integer overflow — Polars is strict about integer types; if you're summing a uint16 column where the sum exceeds 65,535, you'll get an overflow error. Cast to a larger type first: pl.col("quantity").cast(pl.Int64).sum().
You have a reference leak — something is holding a reference to each chunk after you're done with it. Common causes: appending entire chunks to a list when you only need aggregated results, or an exception handler that's keeping locals alive. Profile with tracemalloc or just add explicit del chunk at the end of your loop body.
This usually means a column has mixed types that pandas' CSV reader inferred inconsistently — e.g., a column that's mostly integers but has some rows with "N/A" strings, causing pandas to sometimes infer float64, sometimes object. Fix by either specifying the dtype explicitly in read_csv(), or using na_values=['N/A', 'none', ''] to ensure consistent null handling before writing to Parquet.
CSV is inherently slow to parse because every cell requires type inference and string parsing. If you're processing the same file more than once, invest in converting it to Parquet first. The chunked Parquet read is significantly faster because types are embedded and compression makes I/O faster. Alternatively, use pl.scan_csv() which is also noticeably faster than pandas' CSV parser.
Large dataset handling in Python isn't about one silver bullet — it's about a layered approach where each technique addresses a specific bottleneck:
Layer 1 — Efficient dtypes tackle the memory waste of pandas' default type choices. Categories for low-cardinality strings, downcasted integers and floats for numeric data. This often delivers 60–80% memory reduction with no code restructuring.
Layer 2 — Chunked reading breaks the requirement that all data fit in memory simultaneously. Design your processing logic to be incremental: accumulate aggregations, filter early, and write results to disk rather than building large in-memory collections.
Layer 3 — Parquet format eliminates the CSV tax on repeated reads. Once you've processed a file once and know its schema, converting to Parquet cuts read times by 10–15x and file sizes by 50–80%.
Layer 4 — Polars brings query optimization, automatic parallelism, and Arrow-native memory to the table. For CPU-bound aggregation and filtering at scale, it's 5–10x faster than optimized pandas with minimal code changes. Use it selectively where performance matters, and convert back to pandas for ecosystem-dependent outputs.
These techniques compound. A pipeline that reads a Parquet file with Polars lazy scanning, using predicate pushdown and parallel execution, can handle hundreds of gigabytes on a laptop that would crash with a naive pandas approach on a 2 GB CSV.
Where to go from here:
The jump from "it works on small data" to "it works on production data" is largely a jump in your understanding of memory, I/O, and computation patterns. You now have the mental models and practical tools to make that jump.