Structuring a Reusable Data Analysis Project: Functions, Modules, Notebooks, and Scripts

Introduction

Here's a scene that plays out constantly in data teams: someone builds an analysis in a Jupyter notebook over two days. It works beautifully. Three weeks later, a stakeholder wants the same analysis run against a different region's data. The original analyst spends four hours untangling hardcoded paths, copy-pasted cell blocks, and df2_final_FINAL_v3 variables before giving up and rebuilding it from scratch.

This is the organizational debt problem, and it's not unique to beginners. Even experienced analysts fall into the trap of treating notebooks as a finished product rather than a surface for exploration. The notebook runs top-to-bottom once, produces a result, and then becomes a liability the moment anyone needs to reproduce, extend, or adapt it.

By the end of this lesson, you'll know how to structure a Python data analysis project so that it's genuinely reusable — not just for others, but for yourself in three weeks. You'll understand when to write a function, when to pull it into a module, when to keep something in a notebook, and when to graduate it to a standalone script. You'll also understand the tradeoffs at each layer so you can make architectural decisions appropriate to the scale and longevity of each project.

What you'll learn:

• How to identify code that belongs in a function vs. code that belongs inline
• How to build a project module structure that notebooks and scripts can both import
• How to design functions and modules with testability and reusability in mind
• The difference between a notebook as an exploration tool vs. a notebook as a report surface
• How to convert a mature notebook analysis into a parameterized, schedulable script
• Common structural mistakes that create technical debt and how to avoid them

---

Prerequisites

You should be comfortable writing Python functions and working with pandas DataFrames. You should have a working Python environment with Jupyter available — if you need to set that up first, see Setting Up Python for Data Analysis: Python, VS Code, Jupyter, and Virtual Environments. You should also be familiar with basic DataFrame operations from Your First pandas DataFrame: Loading CSV and Excel Files and Exploring Data.

---

The Anatomy of a Badly Structured Project

Before we build something right, let's be precise about what "wrong" looks like, because most bad project structures aren't obviously wrong — they look reasonable until they fall apart.

Here's a representative directory that almost every data analyst has created at some point:

sales_analysis/
├── analysis.ipynb
├── analysis_v2.ipynb
├── analysis_FINAL.ipynb
├── analysis_FINAL_with_q4.ipynb
├── data.csv
├── data_cleaned.csv
├── data_cleaned_v2.csv
├── output.xlsx
└── output_final.xlsx

Inside analysis_FINAL_with_q4.ipynb, you'll find:

# Cell 1
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data_cleaned_v2.csv')

# Cell 2 — this one has to run before cell 7 or things break
df['date'] = pd.to_datetime(df['date'])
df['revenue'] = df['revenue'].fillna(0)
df['region'] = df['region'].str.upper().str.strip()

# Cell 3 — don't run this, it's the old version
# df = df[df['year'] == 2022]

# Cell 7 (cells 4-6 were deleted)
df_west = df[df['region'] == 'WEST']
df_west_monthly = df_west.groupby(pd.Grouper(key='date', freq='ME'))['revenue'].sum().reset_index()

# ... 40 more cells

What makes this hard to reuse isn't laziness — it's that the code was written for a single execution, not for adaptation. Every assumption is baked in: the filename, the region, the date column name, the aggregation period. When requirements change by even one degree, you're rewriting, not parameterizing.

The path from here to a well-structured project isn't a rewrite — it's a series of deliberate extractions. Let's build the structure from scratch so you understand the reasoning behind each layer.

---

Designing the Project Structure

A reusable data project needs three distinct zones: data (inputs and outputs), code (functions and modules), and surfaces (notebooks and scripts). Each zone has a different purpose and a different lifecycle.

Here's the target structure we'll build toward throughout this lesson:

sales_analysis/
├── data/
│   ├── raw/
│   │   └── sales_2024.csv
│   └── processed/
│       └── sales_clean.parquet
├── output/
│   ├── monthly_summary.xlsx
│   └── charts/
│       └── revenue_by_region.png
├── src/
│   ├── __init__.py
│   ├── load.py
│   ├── clean.py
│   ├── transform.py
│   └── report.py
├── notebooks/
│   ├── 01_exploration.ipynb
│   └── 02_reporting.ipynb
├── scripts/
│   └── run_monthly_report.py
├── tests/
│   ├── test_clean.py
│   └── test_transform.py
└── requirements.txt

This isn't the only valid structure — it's a structure where every decision has a reason. Let's walk through those reasons.

data/raw/ is immutable. Raw files should never be overwritten by your code. If your cleaning step modifies data/raw/sales_2024.csv in place, you lose your ground truth. Treat raw data the way you'd treat a database backup: read-only.

data/processed/ is reproducible. Processed files are outputs of your cleaning pipeline. If you delete them, you should be able to regenerate them by running your code. This means they're safe to commit to .gitignore — you don't need to version them because they're derived artifacts.

src/ is your library. This is where logic lives. Not results, not visuals — logic. Functions here should be importable from both notebooks and scripts without modification.

notebooks/ are numbered and purposeful. The number prefix forces a clear sequence. 01_exploration is where you experiment freely. 02_reporting is where you use your src functions to produce the final output. Notebooks should import from src, not define the logic themselves.

scripts/ are for automation. A script is a notebook that runs non-interactively. It doesn't have cells or markdown — just code that executes top-to-bottom and exits cleanly.

---

Writing Functions That Are Actually Reusable

The single most common mistake in data analysis functions is hardcoding assumptions that belong to the caller. Let's build the cleaning module for our sales project in a way that avoids this.

Starting Wrong: The Hardcoded Function

def clean_sales_data():
    df = pd.read_csv('data/raw/sales_2024.csv')
    df['date'] = pd.to_datetime(df['date'])
    df['revenue'] = df['revenue'].fillna(0)
    df['region'] = df['region'].str.upper().str.strip()
    df = df[df['revenue'] > 0]
    return df

This function looks clean, but it's a trap. It can only ever load one specific file. It assumes revenue and region and date are the column names. It silently drops rows with zero revenue, which is a business decision buried in infrastructure code. And it can't be tested without that specific CSV existing on disk.

Building Right: The Parameterized Function

# src/clean.py

import pandas as pd
import logging

logger = logging.getLogger(__name__)


def parse_dates(df: pd.DataFrame, date_col: str = 'date') -> pd.DataFrame:
    """
    Parse a date column to datetime. Coerces errors to NaT rather than raising.
    
    Returns a copy of the DataFrame with the column converted.
    """
    df = df.copy()
    original_nulls = df[date_col].isna().sum()
    df[date_col] = pd.to_datetime(df[date_col], errors='coerce')
    new_nulls = df[date_col].isna().sum()
    coerced = new_nulls - original_nulls
    if coerced > 0:
        logger.warning(f"Coerced {coerced} unparseable values in '{date_col}' to NaT")
    return df


def normalize_text_columns(df: pd.DataFrame, cols: list[str]) -> pd.DataFrame:
    """
    Strip whitespace and uppercase a list of text columns.
    Skips columns that don't exist in the DataFrame with a warning.
    """
    df = df.copy()
    for col in cols:
        if col not in df.columns:
            logger.warning(f"Column '{col}' not found — skipping normalization")
            continue
        df[col] = df[col].str.strip().str.upper()
    return df


def fill_numeric_nulls(
    df: pd.DataFrame,
    cols: list[str],
    fill_value: float = 0.0
) -> pd.DataFrame:
    """Fill nulls in numeric columns with a specified value."""
    df = df.copy()
    for col in cols:
        null_count = df[col].isna().sum()
        if null_count > 0:
            logger.info(f"Filling {null_count} nulls in '{col}' with {fill_value}")
        df[col] = df[col].fillna(fill_value)
    return df


def remove_zero_revenue(
    df: pd.DataFrame,
    revenue_col: str = 'revenue',
    keep_zeros: bool = False
) -> pd.DataFrame:
    """
    Optionally filter out rows where revenue is zero or negative.
    
    The keep_zeros flag makes this decision explicit at call time
    rather than buried in the function body.
    """
    if keep_zeros:
        return df
    before = len(df)
    df = df[df[revenue_col] > 0].copy()
    dropped = before - len(df)
    if dropped > 0:
        logger.info(f"Removed {dropped} rows with non-positive revenue")
    return df


def clean_sales(
    df: pd.DataFrame,
    date_col: str = 'date',
    revenue_col: str = 'revenue',
    text_cols: list[str] | None = None,
    keep_zeros: bool = False
) -> pd.DataFrame:
    """
    Master cleaning pipeline for sales data. Composes individual cleaning
    steps. Each step returns a new DataFrame, preserving the original.
    """
    if text_cols is None:
        text_cols = ['region', 'product_category']
    
    df = parse_dates(df, date_col=date_col)
    df = normalize_text_columns(df, cols=text_cols)
    df = fill_numeric_nulls(df, cols=[revenue_col])
    df = remove_zero_revenue(df, revenue_col=revenue_col, keep_zeros=keep_zeros)
    return df

Notice what changed. Each function takes a DataFrame and returns a DataFrame. None of them load files — that's a separate concern handled by the load.py module. Business decisions like keep_zeros are explicit parameters, not silent assumptions. Every function uses df.copy() to avoid mutating the caller's data, which prevents a class of bugs that are genuinely difficult to debug.

The master clean_sales function composes the individual steps. This composition pattern matters: it gives you a single callable for the common case while keeping each step independently testable. You can test parse_dates with a toy DataFrame in two lines without touching a CSV file.

---

Building the Module Layer

Now that we have clean functions, we need to organize them into modules that can be imported without friction. This is where src/__init__.py and the module import path come in.

Making `src` Importable

Create src/__init__.py — it can be empty, or it can expose a clean public API:

# src/__init__.py

from .load import load_raw_sales
from .clean import clean_sales
from .transform import monthly_revenue_by_region, top_performers
from .report import build_summary_workbook

This __init__.py gives you a flat import surface:

# From a notebook or script
from src import load_raw_sales, clean_sales, monthly_revenue_by_region

Instead of having to know which submodule everything lives in, callers get a clean interface. This matters when your src directory grows to 10+ modules — you don't want callers depending on internal organization that might change.

The Load Module

# src/load.py

import pandas as pd
from pathlib import Path
import logging

logger = logging.getLogger(__name__)

RAW_DATA_DIR = Path(__file__).parent.parent / 'data' / 'raw'
PROCESSED_DATA_DIR = Path(__file__).parent.parent / 'data' / 'processed'


def load_raw_sales(filename: str, data_dir: Path | None = None) -> pd.DataFrame:
    """
    Load a raw sales CSV from the data/raw directory.
    
    Args:
        filename: The CSV filename (e.g., 'sales_2024.csv')
        data_dir: Override the default raw data directory.
                  Useful for testing with a temp directory.
    
    Returns:
        Raw DataFrame with no transformations applied.
    """
    if data_dir is None:
        data_dir = RAW_DATA_DIR
    
    filepath = Path(data_dir) / filename
    
    if not filepath.exists():
        raise FileNotFoundError(
            f"Raw data file not found: {filepath}\n"
            f"Expected location: {filepath.resolve()}"
        )
    
    logger.info(f"Loading raw data from {filepath}")
    df = pd.read_csv(filepath)
    logger.info(f"Loaded {len(df):,} rows, {len(df.columns)} columns")
    return df


def save_processed(df: pd.DataFrame, filename: str, data_dir: Path | None = None) -> Path:
    """
    Save a processed DataFrame as Parquet for fast subsequent loads.
    
    Returns the path where the file was saved.
    """
    if data_dir is None:
        data_dir = PROCESSED_DATA_DIR
    
    data_dir = Path(data_dir)
    data_dir.mkdir(parents=True, exist_ok=True)
    
    filepath = data_dir / filename
    df.to_parquet(filepath, index=False)
    logger.info(f"Saved processed data to {filepath} ({len(df):,} rows)")
    return filepath

Using Path(__file__).parent.parent to locate the data directory is important. It means your modules work regardless of where the calling code runs from — whether that's a notebook in notebooks/, a script in scripts/, or a test in tests/. Relative paths like '../../data/raw' are fragile; anchoring paths to the module file's location is robust.

The Transform Module

This is where your analytical logic lives. For grouping and aggregating operations, functions in transform.py should accept DataFrames and return DataFrames — no side effects, no file I/O.

# src/transform.py

import pandas as pd
from typing import Literal


def monthly_revenue_by_region(
    df: pd.DataFrame,
    date_col: str = 'date',
    revenue_col: str = 'revenue',
    region_col: str = 'region',
    freq: str = 'ME'
) -> pd.DataFrame:
    """
    Aggregate revenue by month and region.
    
    Args:
        freq: Pandas offset alias — 'ME' (month end), 'QE' (quarter end), etc.
    
    Returns:
        DataFrame with columns: period, region, revenue, transaction_count
    """
    df = df.copy()
    df['period'] = df[date_col].dt.to_period(freq)
    
    summary = (
        df.groupby(['period', region_col], observed=True)
        .agg(
            revenue=(revenue_col, 'sum'),
            transaction_count=(revenue_col, 'count')
        )
        .reset_index()
        .rename(columns={region_col: 'region'})
        .sort_values(['period', 'region'])
    )
    return summary


def top_performers(
    df: pd.DataFrame,
    group_col: str,
    value_col: str = 'revenue',
    n: int = 10,
    method: Literal['sum', 'mean', 'median'] = 'sum'
) -> pd.DataFrame:
    """
    Return the top N groups by an aggregated value.
    
    Args:
        group_col: Column to group by (e.g., 'salesperson', 'product')
        n: Number of top performers to return
        method: Aggregation method to rank by
    
    Returns:
        DataFrame sorted descending by the aggregated value, limited to n rows.
    """
    agg_func = {'sum': 'sum', 'mean': 'mean', 'median': 'median'}[method]
    
    result = (
        df.groupby(group_col, observed=True)[value_col]
        .agg(agg_func)
        .reset_index()
        .rename(columns={value_col: f'{method}_{value_col}'})
        .sort_values(f'{method}_{value_col}', ascending=False)
        .head(n)
        .reset_index(drop=True)
    )
    return result


def compute_period_over_period_growth(
    df: pd.DataFrame,
    period_col: str = 'period',
    value_col: str = 'revenue',
    group_col: str | None = None
) -> pd.DataFrame:
    """
    Add a period-over-period growth percentage column.
    
    If group_col is provided, computes growth within each group separately.
    """
    df = df.sort_values([group_col, period_col] if group_col else [period_col]).copy()
    
    if group_col:
        df['prior_period_value'] = df.groupby(group_col, observed=True)[value_col].shift(1)
    else:
        df['prior_period_value'] = df[value_col].shift(1)
    
    df['pct_change'] = (
        (df[value_col] - df['prior_period_value']) / df['prior_period_value'] * 100
    ).round(2)
    
    df = df.drop(columns=['prior_period_value'])
    return df

---

Using Notebooks Correctly: Exploration vs. Reporting

With a proper src layer in place, notebooks change their role dramatically. They stop being the place where logic lives and become the place where you use logic.

The Exploration Notebook

# notebooks/01_exploration.ipynb

# Cell 1: Setup
import sys
sys.path.insert(0, '..')  # Make src importable from the notebooks/ subdirectory

import pandas as pd
import matplotlib.pyplot as plt
from src import load_raw_sales, clean_sales

# Cell 2: Load and inspect
df_raw = load_raw_sales('sales_2024.csv')
df_raw.head()

# Cell 3: What does messy data look like?
print(f"Shape: {df_raw.shape}")
print(f"\nNull counts:\n{df_raw.isna().sum()}")
print(f"\nUnique regions: {df_raw['region'].unique()}")

# Cell 4: Apply cleaning and verify
df = clean_sales(df_raw, keep_zeros=False)

# Verify the cleaning worked as expected
assert df['date'].isna().sum() == 0, "Unexpected nulls in date after cleaning"
assert (df['revenue'] <= 0).sum() == 0, "Expected zero-revenue rows to be removed"
print(f"Cleaned shape: {df.shape}")

# Cell 5: Quick exploration — what are we working with?
from src import monthly_revenue_by_region
monthly = monthly_revenue_by_region(df)
monthly.tail(12)

The exploration notebook is allowed to be messy. You can have dead cells, experimental code, commented-out attempts. This is the scratchpad. But notice what's already different: the actual logic — loading, cleaning, aggregating — is one import and one function call. If cleaning needs to change, you change src/clean.py and the notebook automatically benefits.

The sys.path.insert(0, '..') call is inelegant but practical for getting notebooks inside a subdirectory to find the src package. A cleaner alternative is to install your package in editable mode (pip install -e . with a minimal setup.py or pyproject.toml), which we'll touch on in the troubleshooting section.

The Reporting Notebook

The reporting notebook has a different contract: it should run cleanly top-to-bottom with no manual intervention. Every cell should be deterministic.

# notebooks/02_reporting.ipynb

# Cell 1: Configuration — all parameters at the top, visible
REPORT_YEAR = 2024
REGION_FILTER = None  # Set to 'WEST' to filter, None for all regions
TOP_N_PRODUCTS = 15
OUTPUT_PATH = '../output/monthly_summary.xlsx'

# Cell 2: Imports and setup
import sys
sys.path.insert(0, '..')

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from src import load_raw_sales, clean_sales, monthly_revenue_by_region, top_performers
from src import build_summary_workbook

# Cell 3: Load and clean
df_raw = load_raw_sales('sales_2024.csv')
df = clean_sales(df_raw)

# Apply optional filters
if REGION_FILTER:
    df = df[df['region'] == REGION_FILTER].copy()

print(f"Working with {len(df):,} transactions")

# Cell 4: Generate aggregations
monthly = monthly_revenue_by_region(df)
top_products = top_performers(df, group_col='product_category', n=TOP_N_PRODUCTS)

# Cell 5: Visualize
fig, axes = plt.subplots(1, 2, figsize=(16, 6))

# Revenue trend
monthly_totals = monthly.groupby('period')['revenue'].sum().reset_index()
axes[0].plot(monthly_totals['period'].astype(str), monthly_totals['revenue'] / 1e6)
axes[0].set_title('Monthly Revenue (Millions)')
axes[0].tick_params(axis='x', rotation=45)

# Top products
axes[1].barh(top_products['product_category'], top_products['sum_revenue'] / 1e6)
axes[1].set_title(f'Top {TOP_N_PRODUCTS} Product Categories by Revenue')
axes[1].set_xlabel('Revenue (Millions)')

plt.tight_layout()
plt.savefig('../output/charts/revenue_overview.png', dpi=150, bbox_inches='tight')
plt.show()

# Cell 6: Export
build_summary_workbook(monthly, top_products, output_path=OUTPUT_PATH)
print(f"Report saved to {OUTPUT_PATH}")

The key discipline here is putting all parameters in Cell 1. When someone needs to adapt this report for a different year or region, they change one cell at the top — they don't hunt through 20 cells for hardcoded values. This turns a notebook into something that functions like a simple configuration-driven tool.

---

The Report Module: Encapsulating Output Logic

Output formatting is messy code that belongs in src/report.py, not in notebooks. For anything involving formatted Excel workbooks, encapsulating the openpyxl calls in a function keeps notebooks readable.

# src/report.py

import pandas as pd
from pathlib import Path
import logging

logger = logging.getLogger(__name__)


def build_summary_workbook(
    monthly_summary: pd.DataFrame,
    top_performers_df: pd.DataFrame,
    output_path: str | Path,
    include_charts: bool = False
) -> Path:
    """
    Write a formatted Excel workbook with a monthly summary sheet
    and a top performers sheet.
    
    Args:
        monthly_summary: Output of monthly_revenue_by_region()
        top_performers_df: Output of top_performers()
        output_path: Where to write the .xlsx file
        include_charts: Whether to embed an openpyxl chart (experimental)
    
    Returns:
        Path to the written file
    """
    output_path = Path(output_path)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    
    with pd.ExcelWriter(output_path, engine='openpyxl') as writer:
        # Monthly summary sheet
        monthly_export = monthly_summary.copy()
        monthly_export['period'] = monthly_export['period'].astype(str)
        monthly_export['revenue'] = monthly_export['revenue'].round(2)
        monthly_export.to_excel(writer, sheet_name='Monthly by Region', index=False)
        
        # Top performers sheet
        top_performers_df.to_excel(writer, sheet_name='Top Performers', index=False)
        
        # Apply basic formatting
        workbook = writer.book
        for sheet_name in writer.sheets:
            worksheet = writer.sheets[sheet_name]
            for column_cells in worksheet.columns:
                max_length = max(
                    len(str(cell.value or '')) for cell in column_cells
                )
                worksheet.column_dimensions[
                    column_cells[0].column_letter
                ].width = min(max_length + 4, 50)
    
    logger.info(f"Workbook saved to {output_path}")
    return output_path

---

Graduating to a Script: The Automation Layer

When an analysis needs to run on a schedule — nightly, weekly, on data arrival — you need a script. A script is not a simplified notebook; it's a different artifact with different requirements.

Scripts must:

• Accept parameters from the command line or environment variables
• Log their activity so you can diagnose failures without being present
• Exit with a non-zero status code on failure (so schedulers like cron or Airflow know something went wrong)
• Be idempotent where possible — running twice should produce the same result, not double the output

# scripts/run_monthly_report.py

"""
Monthly Sales Report Generator

Usage:
    python run_monthly_report.py --year 2024 --region WEST
    python run_monthly_report.py --year 2024  # all regions
    
Environment variables:
    SALES_DATA_DIR: Override the default raw data directory
    REPORT_OUTPUT_DIR: Override the default output directory
"""

import argparse
import logging
import os
import sys
from pathlib import Path
from datetime import datetime

# Add the project root to sys.path so src is importable
project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))

from src import load_raw_sales, clean_sales, monthly_revenue_by_region, top_performers
from src import build_summary_workbook

# Configure logging — scripts log to both console and a file
log_dir = project_root / 'logs'
log_dir.mkdir(exist_ok=True)
log_filename = log_dir / f"monthly_report_{datetime.now():%Y%m%d_%H%M%S}.log"

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s | %(levelname)s | %(name)s | %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler(log_filename)
    ]
)
logger = logging.getLogger(__name__)


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description='Generate monthly sales report')
    parser.add_argument(
        '--year',
        type=int,
        required=True,
        help='Fiscal year to report on (e.g., 2024)'
    )
    parser.add_argument(
        '--region',
        type=str,
        default=None,
        help='Filter to a specific region (e.g., WEST). Omit for all regions.'
    )
    parser.add_argument(
        '--top-n',
        type=int,
        default=10,
        dest='top_n',
        help='Number of top products to include (default: 10)'
    )
    parser.add_argument(
        '--output-dir',
        type=Path,
        default=None,
        dest='output_dir',
        help='Override default output directory'
    )
    return parser.parse_args()


def main() -> int:
    """
    Returns 0 on success, 1 on failure.
    This return value is used by the OS and schedulers to detect failures.
    """
    args = parse_args()
    logger.info(f"Starting monthly report | year={args.year} | region={args.region or 'ALL'}")
    
    try:
        # Resolve paths — environment variables allow Docker/container overrides
        data_dir = Path(os.environ.get('SALES_DATA_DIR', project_root / 'data' / 'raw'))
        output_dir = args.output_dir or Path(
            os.environ.get('REPORT_OUTPUT_DIR', project_root / 'output')
        )
        
        filename = f'sales_{args.year}.csv'
        
        # Load
        logger.info(f"Loading {filename} from {data_dir}")
        df_raw = load_raw_sales(filename, data_dir=data_dir)
        
        # Clean
        df = clean_sales(df_raw)
        
        # Filter
        if args.region:
            before = len(df)
            df = df[df['region'] == args.region.upper()].copy()
            logger.info(f"Filtered to region '{args.region}': {before:,} → {len(df):,} rows")
            if len(df) == 0:
                logger.error(f"No data found for region '{args.region}' in {args.year}")
                return 1
        
        # Transform
        monthly = monthly_revenue_by_region(df)
        top = top_performers(df, group_col='product_category', n=args.top_n)
        
        # Output
        region_tag = f'_{args.region}' if args.region else ''
        output_filename = f'monthly_report_{args.year}{region_tag}.xlsx'
        output_path = output_dir / output_filename
        
        build_summary_workbook(monthly, top, output_path=output_path)
        logger.info(f"Report complete: {output_path}")
        
        return 0
    
    except FileNotFoundError as e:
        logger.error(f"Data file not found: {e}")
        return 1
    except Exception as e:
        logger.exception(f"Unexpected error in report generation: {e}")
        return 1


if __name__ == '__main__':
    sys.exit(main())

The sys.exit(main()) pattern at the bottom is important. Schedulers check the process exit code. If main() returns 1, the scheduler knows the job failed and can alert, retry, or stop downstream dependencies. If your script ends with an unhandled exception instead of a clean return 1, some schedulers won't detect the failure correctly.

---

Writing Testable Functions

The architecture we've built has a hidden benefit: every function in src/ can be tested with a tiny synthetic DataFrame, no files required. This is the payoff for keeping I/O out of transformation functions.

# tests/test_clean.py

import pandas as pd
import pytest
from src.clean import (
    parse_dates,
    normalize_text_columns,
    fill_numeric_nulls,
    remove_zero_revenue,
    clean_sales
)


@pytest.fixture
def sample_df():
    """Minimal DataFrame with known properties for testing."""
    return pd.DataFrame({
        'date': ['2024-01-15', '2024-02-20', 'not-a-date', None],
        'revenue': [1500.0, None, 300.0, 0.0],
        'region': ['  west  ', 'EAST', ' North ', 'EAST'],
        'product_category': ['Software', 'hardware', None, 'Software']
    })


def test_parse_dates_converts_valid_strings(sample_df):
    result = parse_dates(sample_df, date_col='date')
    assert result['date'].dtype == 'datetime64[ns]'
    assert pd.notna(result.loc[0, 'date'])
    assert pd.notna(result.loc[1, 'date'])


def test_parse_dates_coerces_bad_values_to_nat(sample_df):
    result = parse_dates(sample_df, date_col='date')
    # 'not-a-date' and None should both become NaT
    assert pd.isna(result.loc[2, 'date'])
    assert pd.isna(result.loc[3, 'date'])


def test_parse_dates_does_not_mutate_input(sample_df):
    original_value = sample_df.loc[0, 'date']
    parse_dates(sample_df, date_col='date')
    # Original should still be a string
    assert sample_df.loc[0, 'date'] == original_value


def test_normalize_text_strips_whitespace(sample_df):
    result = normalize_text_columns(sample_df, cols=['region'])
    assert result.loc[0, 'region'] == 'WEST'
    assert result.loc[2, 'region'] == 'NORTH'


def test_normalize_text_skips_missing_columns(sample_df):
    # Should not raise; should warn and continue
    result = normalize_text_columns(sample_df, cols=['region', 'nonexistent_col'])
    assert 'region' in result.columns


def test_fill_numeric_nulls_fills_revenue(sample_df):
    result = fill_numeric_nulls(sample_df, cols=['revenue'], fill_value=0.0)
    assert result['revenue'].isna().sum() == 0
    assert result.loc[1, 'revenue'] == 0.0


def test_remove_zero_revenue_drops_zeros(sample_df):
    df_filled = fill_numeric_nulls(sample_df, cols=['revenue'])
    result = remove_zero_revenue(df_filled)
    assert (result['revenue'] <= 0).sum() == 0
    assert len(result) == 2  # Only the two positive revenue rows


def test_remove_zero_revenue_keep_zeros_flag(sample_df):
    df_filled = fill_numeric_nulls(sample_df, cols=['revenue'])
    result = remove_zero_revenue(df_filled, keep_zeros=True)
    # Should keep all rows, including zero revenue
    assert len(result) == len(df_filled)


def test_clean_sales_full_pipeline(sample_df):
    result = clean_sales(sample_df)
    assert result['date'].dtype == 'datetime64[ns]'
    assert result['region'].str.contains('  ').sum() == 0  # No leading/trailing spaces
    assert result['revenue'].isna().sum() == 0
    assert (result['revenue'] <= 0).sum() == 0

Run these with pytest tests/ from the project root. Each test is a specification: it documents what the function is supposed to do and will catch regressions if someone changes clean_sales in a way that breaks prior behavior.

---

Solving the Import Path Problem

The sys.path.insert(0, '..') hack works but has a smell. When notebooks are nested two levels deep, or when you have multiple entry points, managing the path manually becomes fragile. The clean solution is treating your project as an installable package.

Create a minimal pyproject.toml (or setup.py):

# pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.backends.legacy:build"

[project]
name = "sales-analysis"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "pandas>=2.0",
    "openpyxl>=3.1",
    "pyarrow>=14.0",
]

[tool.setuptools.packages.find]
where = ["."]
include = ["src*"]

Then install it in editable mode:

pip install -e .

Now every notebook and script can simply write:

from src import load_raw_sales, clean_sales

With no sys.path manipulation needed. Editable mode means changes to src/ are immediately reflected without reinstalling. This is the pattern used by data science teams working in shared Conda environments or containerized pipelines.

---

Hands-On Exercise

You have a messy notebook that someone handed you for refactoring. Below is the full notebook content as a single block:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/home/alice/projects/customer_churn/data/customers_oct_2024.csv')
df['signup_date'] = pd.to_datetime(df['signup_date'])
df['last_activity_date'] = pd.to_datetime(df['last_activity_date'])
df['monthly_spend'] = df['monthly_spend'].fillna(df['monthly_spend'].median())
df['plan'] = df['plan'].str.upper().str.strip()
df['days_since_active'] = (pd.Timestamp('2024-10-31') - df['last_activity_date']).dt.days
df['is_churned'] = df['days_since_active'] > 90

churned = df[df['is_churned']]
active = df[~df['is_churned']]
print(f"Churned: {len(churned)}, Active: {len(active)}")

churn_by_plan = df.groupby('plan')['is_churned'].mean().reset_index()
churn_by_plan.columns = ['plan', 'churn_rate']
churn_by_plan['churn_rate'] = (churn_by_plan['churn_rate'] * 100).round(1)

plt.bar(churn_by_plan['plan'], churn_by_plan['churn_rate'])
plt.title('Churn Rate by Plan')
plt.ylabel('Churn Rate (%)')
plt.savefig('/home/alice/projects/customer_churn/output/churn_by_plan.png')
plt.show()

top_spend = df.groupby('plan')['monthly_spend'].mean().reset_index().sort_values('monthly_spend', ascending=False)
print(top_spend)

Your tasks:

1. Create the project structure for customer_churn/ with data/raw/, data/processed/, src/, notebooks/, scripts/, and tests/ directories.

2. Build src/load.py with a load_customers() function that uses path anchoring relative to the module file, not an absolute path.

3. Build src/clean.py with individual functions for: parsing date columns, filling nulls with the median (parameterized — callers choose which columns), and computing days_since_active with a configurable reference date rather than a hardcoded 2024-10-31.

4. Build src/transform.py with label_churned(df, days_threshold=90) and churn_rate_by_group(df, group_col) that work with any group column, not just 'plan'.

5. Rewrite the original notebook as notebooks/02_reporting.ipynb that imports from src, has all parameters (the churn threshold, the reference date, the filename) in the first cell, and runs cleanly top-to-bottom.

6. Write tests/test_clean.py with at least three tests that use synthetic DataFrames — no file I/O.

7. Write scripts/run_churn_report.py that accepts --reference-date, --churn-days, and --output-dir as command-line arguments and exits with code 1 if the data file isn't found.

This exercise takes about two hours to complete fully. The goal isn't just getting it to run — it's making every decision explicit: where does each piece of logic belong, and why?

---

Common Mistakes & Troubleshooting

Mistake 1: Putting Logic in Notebooks and Importing from There

Some analysts try to import functions they defined in a notebook. This doesn't work reliably — notebooks are not importable Python modules. If a function is worth reusing, it belongs in src/. Full stop.

Mistake 2: `sys.path` That Works on Your Machine and Nowhere Else

If you hardcode sys.path.insert(0, '/home/yourname/projects/analysis/src'), your project is broken for everyone else. Always use Path(__file__).parent anchoring or install in editable mode.

Mistake 3: Mutation Without Copying

# This is a trap
def add_month_column(df):
    df['month'] = df['date'].dt.month  # Mutates the caller's DataFrame!
    return df

Always copy at the top of any function that adds or modifies columns: df = df.copy(). The symptom of missing copies is that a variable you thought was unmodified has changed, which produces errors that only appear when notebook cells run in a particular order.

Mistake 4: Module-Level Code That Runs on Import

# src/transform.py — DON'T DO THIS
import pandas as pd

# This runs when anyone imports from this module!
df = pd.read_csv('data/raw/sales.csv')

Module-level code (outside functions) executes the moment you import the module. Keep all data loading and side effects inside functions. Module-level code should be limited to imports, constants, and logger instantiation.

Mistake 5: The `__init__.py` That Hides Errors

If your src/__init__.py imports everything eagerly:

from .load import load_raw_sales
from .clean import clean_sales

...and clean.py has a syntax error, every import from src will fail with a confusing error message. During active development, import specific submodules directly (from src.clean import clean_sales) so error messages point you to the right file.

Troubleshooting: "ModuleNotFoundError: No module named 'src'"

This is the most common error when first setting up this structure. Check in order:

1. Are you running from the project root? (cd to the directory containing src/)
2. Does src/__init__.py exist?
3. Did you add the path correctly? (sys.path.insert(0, str(Path(__file__).parent.parent)))
4. Did you install in editable mode? (pip install -e .)

Troubleshooting: Import Works in Script But Not in Notebook

Notebooks set their working directory to the notebook's own directory, not the project root. This is why sys.path.insert(0, '..') in a notebook inside notebooks/ works — it goes up one level to the project root. The script uses Path(__file__).parent.parent instead. Both anchor to the same place, but via different mechanisms appropriate to their context.

---

Summary & Next Steps

We've built a project structure from first principles — not because it follows a convention, but because each layer serves a specific purpose. The src/ module layer holds logic that's importable, testable, and independent of where it runs. Notebooks are surfaces for exploration and reporting, not homes for business logic. Scripts are the automation layer that turns a working analysis into a scheduled, observable, failure-aware process.

The key disciplines to carry forward:

• Functions get a DataFrame, return a DataFrame. No file I/O, no global state, always copy before mutating.
• Notebooks import from src/, they don't define logic. The moment you paste the same cleaning code into a second notebook, it belongs in a function.
• Scripts exit with a status code. Return 0 on success, 1 on failure. Use structured logging. Accept parameters from the command line.
• Tests use synthetic data. If testing a function requires a specific file to exist, the function's design is the problem.

For deeper work on the analytical functions that would fill out this structure, you'll want to review Grouping and Aggregating in pandas: groupby as the PivotTable Replacement and Working with Dates and Time Series in pandas: Parsing, Resampling, and Rolling Windows. For the output layer, Automating Excel Reports with pandas and openpyxl: Formatted Workbooks Without Manual Work covers the full formatting toolkit your report.py module can draw on.

The structure in this lesson scales from a one-analyst project to a team pipeline. Start with it on your next analysis — even a small one. The habit of separating logic from surface is worth building on projects where it feels like overkill, so it's automatic on the ones where it really matters.