Most data pipeline failures are invisible until a downstream consumer notices stale data. This lesson shows you how to implement unified observability across ingestion, transformation, and orchestration using OpenTelemetry — with working code for distributed traces, pipeline-specific metrics, and correlated structured logs.

It's 2:30 AM and your on-call engineer gets paged. A downstream dashboard is showing stale data — revenue figures that stopped updating six hours ago. The Airflow UI shows all tasks green. The database is healthy. The API source returned 200s all day. But somewhere between the API call and the warehouse, 400,000 rows evaporated.
Without proper observability, the next two hours look like this: grepping through fragmented log files, manually querying row counts at each stage, checking dead letter queues, and squinting at CloudWatch metrics that don't align with your pipeline's logical boundaries. You're not debugging a system — you're doing archaeology.
This is exactly the problem that unified pipeline observability solves, and OpenTelemetry (OTel) is the industry's most capable framework for building it. By the end of this lesson, you'll have a complete mental model and working implementation for instrumenting a multi-stage data pipeline with distributed traces, structured metrics, and correlated logs — all using OpenTelemetry's vendor-neutral SDK. When something breaks at 2:30 AM, you'll open a single trace view and see exactly which span failed, how long each stage took, what data volumes moved through each layer, and why.
What you'll learn:
This lesson assumes you're comfortable with Python data pipelines, have worked with at least one orchestration tool (Airflow, Prefect, or similar), and understand the basics of distributed systems. You should know what a DAG is, how HTTP APIs work, and have passing familiarity with Docker for running local backends. If you need a refresher on what a data pipeline is and how its core layers fit together, that foundation will help you understand why each layer needs distinct instrumentation strategies.
You'll need: Python 3.9+, the opentelemetry-sdk packages, and optionally Docker for running Jaeger or Prometheus locally.
Before we touch any code, let's be precise about why OpenTelemetry matters for data pipelines specifically — not just microservices.
Data pipelines are distributed systems with a distinctive shape. Unlike a web request that completes in milliseconds, a pipeline run spans seconds to hours, crosses multiple compute boundaries (a Kubernetes pod running your ingestion script, a Spark cluster doing transformation, an Airflow worker managing orchestration), and processes data in bulk operations where individual record errors matter as much as overall task health.
Traditional application monitoring tools were built for request/response systems. They assume short-lived operations, homogeneous compute, and binary pass/fail semantics. Pipeline operators end up stitching together: Airflow's built-in task logs (unstructured), CloudWatch metrics (infrastructure-level), custom print() statements (please no), and bespoke alerting scripts.
OpenTelemetry gives you a single SDK that emits the three signals — traces (causal, timing-aware chains of operations), metrics (numeric measurements over time), and logs (timestamped events with structured context) — and a wire protocol (OTLP) that ships all three to any backend you choose. For pipeline engineers, this means:
trace_id and span_id as the active trace, so you can pivot between "show me all logs for this specific pipeline run" and "show me the full trace waterfall"Key insight
OpenTelemetry's value isn't just vendor neutrality (though that matters). It's the correlation model — the ability to link a log line, a metric spike, and a trace span back to the same causal event. That correlation is what turns reactive debugging into proactive understanding.
The OpenTelemetry specification defines a clean separation: the API (interfaces and no-op implementations, safe to include in library code), the SDK (the actual implementation, configured by your application), and exporters (the bridge to backends). This layering matters when you're building shared pipeline framework code — your library instruments with the API, and operators decide where signals go by configuring the SDK.
A trace is a directed acyclic graph of spans, where each span represents a unit of work with a start time, end time, name, and set of key-value attributes. The critical property is causality: spans carry a parent-child relationship and a shared trace_id, so you can reconstruct "this span was triggered by that span, which was triggered by this root operation."
For a pipeline that ingests from an API, transforms in Pandas, and loads to Snowflake, a trace might look like:
pipeline_run [trace_id: abc123] (45.2s)
├── ingest_from_salesforce_api (12.1s)
│ ├── fetch_page[1] (1.4s)
│ ├── fetch_page[2] (1.3s)
│ └── ... (8 more pages)
├── transform_opportunity_records (28.4s)
│ ├── normalize_currency_fields (3.2s)
│ ├── deduplicate_on_external_id (18.1s) ← this is where your bottleneck lives
│ └── validate_schema (7.1s)
└── load_to_snowflake (4.7s)
├── stage_to_s3 (2.1s)
└── copy_into_opportunities (2.6s)
When deduplication takes 18 seconds instead of its usual 2, you don't need to guess — the trace tells you.
Metrics are numeric measurements aggregated over time. OTel supports several instrument types:
The key design decision for pipeline metrics is cardinality. Each unique combination of label values creates a new time series in your metrics backend. pipeline_name + source_system + environment is fine. Adding record_id or batch_uuid as a label will blow up your Prometheus storage and your cloud bill. We'll return to this in the mistakes section.
Structured logs in the OTel model carry a trace_id and span_id on every entry when emitted within an active span context. This is the bridge between your existing logging infrastructure and your new trace data. You don't have to replace your logger — you bridge it.
Let's build from the ground up. First, install the core packages:
pip install \
opentelemetry-sdk \
opentelemetry-api \
opentelemetry-exporter-otlp-proto-grpc \
opentelemetry-exporter-prometheus \
opentelemetry-instrumentation-logging
For local development, run Jaeger all-in-one via Docker (it accepts OTLP and renders trace waterfalls):
docker run -d --name jaeger \
-p 16686:16686 \
-p 4317:4317 \
jaegertracing/all-in-one:latest
Now, the SDK bootstrap. This is where most tutorials cut corners and you pay for it later. Create a dedicated observability.py module in your pipeline package:
# pipeline/observability.py
import logging
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
def configure_observability(
service_name: str,
service_version: str,
otlp_endpoint: str = "localhost:4317",
environment: str = "development"
) -> tuple:
"""
Bootstrap OTel SDK for a pipeline service.
Returns (tracer, meter) for use in pipeline code.
Call once at process startup, before any pipeline work begins.
"""
resource = Resource.create({
SERVICE_NAME: service_name,
SERVICE_VERSION: service_version,
"deployment.environment": environment,
"pipeline.framework": "custom-python",
})
# --- Traces ---
tracer_provider = TracerProvider(resource=resource)
span_exporter = OTLPSpanExporter(
endpoint=otlp_endpoint,
insecure=True # use TLS in production
)
tracer_provider.add_span_processor(
BatchSpanProcessor(
span_exporter,
max_export_batch_size=512,
export_timeout_millis=30_000,
)
)
trace.set_tracer_provider(tracer_provider)
# --- Metrics ---
metric_exporter = OTLPMetricExporter(
endpoint=otlp_endpoint,
insecure=True
)
metric_reader = PeriodicExportingMetricReader(
metric_exporter,
export_interval_millis=15_000 # 15s scrape interval
)
meter_provider = MeterProvider(
resource=resource,
metric_readers=[metric_reader]
)
metrics.set_meter_provider(meter_provider)
# --- Logs ---
logger_provider = LoggerProvider(resource=resource)
log_exporter = OTLPLogExporter(endpoint=otlp_endpoint, insecure=True)
logger_provider.add_log_record_processor(
BatchLogRecordProcessor(log_exporter)
)
set_logger_provider(logger_provider)
# Bridge Python's standard logging to OTel
otel_handler = LoggingHandler(
level=logging.DEBUG,
logger_provider=logger_provider
)
logging.getLogger().addHandler(otel_handler)
logging.getLogger().setLevel(logging.INFO)
tracer = trace.get_tracer(service_name, service_version)
meter = metrics.get_meter(service_name, service_version)
return tracer, meter
Warning
Call configure_observability() exactly once per process, at startup — not inside functions or class constructors that might be called multiple times. Multiple calls to set_tracer_provider() will silently replace the previous provider, dropping any spans that were in flight in the BatchSpanProcessor queue.
The Resource object is not cosmetic. Every span, metric, and log record inherits these attributes, and they're how your backend knows this telemetry came from the salesforce-ingestion service running version 2.4.1 in production. Without a proper resource, you're debugging blind in multi-service deployments.
The ingestion layer — where you pull from APIs, databases, or file systems — has a characteristic shape: a loop over pages or chunks, with network I/O on each iteration. This is exactly where distributed tracing shines, because you can see not just the aggregate cost but the per-page timing.
Here's a realistic Salesforce API ingestion function, fully instrumented. For more context on the API patterns we're wrapping here, see the deep dive on advanced API integration for data pipelines.
# pipeline/ingestion/salesforce.py
import logging
import time
import requests
from opentelemetry import trace
from opentelemetry.trace import SpanKind, StatusCode
from opentelemetry.semconv.trace import SpanAttributes
logger = logging.getLogger(__name__)
class SalesforceIngester:
def __init__(self, tracer, meter, base_url: str, auth_token: str):
self.tracer = tracer
self.base_url = base_url
self.auth_token = auth_token
# Define metrics instruments once at construction time
self.records_ingested = meter.create_counter(
name="pipeline.records.ingested",
description="Total records successfully ingested",
unit="records",
)
self.api_request_duration = meter.create_histogram(
name="pipeline.api.request_duration",
description="Duration of individual API page requests",
unit="ms",
)
self.api_errors = meter.create_counter(
name="pipeline.api.errors",
description="Total API request errors by type",
unit="errors",
)
self.batch_size = meter.create_histogram(
name="pipeline.batch.size",
description="Number of records per API page",
unit="records",
)
def ingest_opportunities(
self,
modified_since: str,
parent_span_context=None
) -> list[dict]:
"""
Paginate through Salesforce Opportunity records modified after
modified_since (ISO 8601). Returns all records as a list.
"""
with self.tracer.start_as_current_span(
"ingest_opportunities",
kind=SpanKind.CLIENT,
context=parent_span_context,
) as span:
span.set_attribute("source.system", "salesforce")
span.set_attribute("source.object", "Opportunity")
span.set_attribute("ingestion.modified_since", modified_since)
span.set_attribute(SpanAttributes.HTTP_URL, self.base_url)
all_records = []
page = 1
next_url = (
f"{self.base_url}/services/data/v58.0/query"
f"?q=SELECT+Id,Name,Amount,StageName,CloseDate"
f"+FROM+Opportunity+WHERE+LastModifiedDate+>+{modified_since}"
)
try:
while next_url:
page_records, next_url = self._fetch_page(
next_url, page
)
all_records.extend(page_records)
page += 1
span.set_attribute("ingestion.total_records", len(all_records))
span.set_attribute("ingestion.total_pages", page - 1)
span.set_status(StatusCode.OK)
logger.info(
"Salesforce ingestion complete",
extra={
"records": len(all_records),
"pages": page - 1,
"source": "salesforce",
}
)
return all_records
except Exception as exc:
span.set_status(StatusCode.ERROR, str(exc))
span.record_exception(exc)
raise
def _fetch_page(self, url: str, page_num: int) -> tuple[list, str | None]:
with self.tracer.start_as_current_span(
f"fetch_page",
kind=SpanKind.CLIENT,
) as span:
span.set_attribute("http.page_number", page_num)
span.set_attribute(SpanAttributes.HTTP_URL, url)
span.set_attribute(SpanAttributes.HTTP_METHOD, "GET")
start_ms = time.monotonic() * 1000
labels = {"source": "salesforce", "object": "Opportunity"}
try:
response = requests.get(
url,
headers={"Authorization": f"Bearer {self.auth_token}"},
timeout=30,
)
response.raise_for_status()
duration_ms = time.monotonic() * 1000 - start_ms
data = response.json()
records = data.get("records", [])
# Record metrics
self.api_request_duration.record(duration_ms, labels)
self.records_ingested.add(len(records), labels)
self.batch_size.record(len(records), labels)
# Annotate span with response details
span.set_attribute(SpanAttributes.HTTP_STATUS_CODE, response.status_code)
span.set_attribute("response.record_count", len(records))
span.set_attribute("response.duration_ms", round(duration_ms, 2))
span.set_status(StatusCode.OK)
logger.debug(
"Fetched API page",
extra={"page": page_num, "records": len(records), "duration_ms": duration_ms}
)
next_url = data.get("nextRecordsUrl")
if next_url:
next_url = f"{self.base_url}{next_url}"
return records, next_url
except requests.HTTPError as exc:
duration_ms = time.monotonic() * 1000 - start_ms
error_labels = {**labels, "error_type": type(exc).__name__}
self.api_errors.add(1, error_labels)
self.api_request_duration.record(duration_ms, labels)
span.set_attribute(SpanAttributes.HTTP_STATUS_CODE, exc.response.status_code)
span.set_status(StatusCode.ERROR, str(exc))
span.record_exception(exc)
raise
A few design choices worth explaining here. We use SpanKind.CLIENT for outbound API calls — this tells trace backends to render these spans as external calls rather than internal processing steps, which affects how latency is attributed in service maps. We call span.record_exception(exc) before re-raising, which writes the exception type, message, and stack trace as a span event (not just a status update). And we set span attributes before the potentially-failing operation where possible, so if we crash mid-flight, we still have context.
Tip
Use OpenTelemetry Semantic Conventions (opentelemetry.semconv.trace.SpanAttributes) for well-known attributes like http.url, http.method, and db.statement. Backends like Jaeger, Grafana Tempo, and Honeycomb understand these standard keys and render them with special UI treatment — URL parsing, HTTP status color coding, etc.
Transformation is where pipelines do the most interesting work and also where the most subtle failures occur — data type coercions that silently drop records, deduplication logic that's slightly wrong, currency conversions that use the wrong exchange rate. Tracing alone isn't enough here; you need metrics that track data quality signals alongside performance signals.
# pipeline/transformation/opportunities.py
import logging
import pandas as pd
from opentelemetry import trace
from opentelemetry.trace import StatusCode
logger = logging.getLogger(__name__)
class OpportunityTransformer:
def __init__(self, tracer, meter):
self.tracer = tracer
# Data quality metrics
self.records_dropped = meter.create_counter(
name="pipeline.transform.records_dropped",
description="Records dropped during transformation by reason",
unit="records",
)
self.transform_duration = meter.create_histogram(
name="pipeline.transform.duration",
description="Duration of transformation steps",
unit="ms",
)
self.null_field_rate = meter.create_histogram(
name="pipeline.transform.null_field_rate",
description="Fraction of records with null in a field",
unit="ratio",
)
def transform(self, raw_records: list[dict]) -> pd.DataFrame:
with self.tracer.start_as_current_span("transform_opportunities") as span:
span.set_attribute("transform.input_records", len(raw_records))
df = pd.DataFrame(raw_records)
try:
df = self._normalize_currency(df)
df = self._deduplicate(df)
df = self._validate_required_fields(df)
df = self._coerce_types(df)
span.set_attribute("transform.output_records", len(df))
span.set_attribute(
"transform.drop_rate",
round(1 - len(df) / max(len(raw_records), 1), 4)
)
span.set_status(StatusCode.OK)
return df
except Exception as exc:
span.set_status(StatusCode.ERROR, str(exc))
span.record_exception(exc)
raise
def _normalize_currency(self, df: pd.DataFrame) -> pd.DataFrame:
import time
with self.tracer.start_as_current_span("normalize_currency") as span:
start = time.monotonic() * 1000
# Convert all amounts to USD using a lookup table
# (in reality, this might call an FX rates API or lookup table)
non_usd = df[df["CurrencyIsoCode"] != "USD"]
span.set_attribute("currency.non_usd_records", len(non_usd))
df["Amount_USD"] = df.apply(
lambda row: self._convert_to_usd(
row["Amount"], row.get("CurrencyIsoCode", "USD")
),
axis=1,
)
null_amounts = df["Amount_USD"].isna().sum()
if null_amounts > 0:
self.records_dropped.add(
null_amounts,
{"stage": "normalize_currency", "reason": "unconvertible_currency"}
)
self.null_field_rate.record(
null_amounts / len(df),
{"field": "Amount_USD", "stage": "normalize_currency"}
)
logger.warning(
"Records with unconvertible currency detected",
extra={"count": null_amounts, "field": "Amount_USD"}
)
duration = time.monotonic() * 1000 - start
self.transform_duration.record(
duration, {"stage": "normalize_currency"}
)
span.set_attribute("duration_ms", round(duration, 2))
return df
def _deduplicate(self, df: pd.DataFrame) -> pd.DataFrame:
import time
with self.tracer.start_as_current_span("deduplicate_records") as span:
start = time.monotonic() * 1000
before = len(df)
df = df.sort_values("LastModifiedDate", ascending=False)
df = df.drop_duplicates(subset=["Id"], keep="first")
after = len(df)
dropped = before - after
span.set_attribute("dedup.input_records", before)
span.set_attribute("dedup.output_records", after)
span.set_attribute("dedup.duplicates_removed", dropped)
if dropped > 0:
self.records_dropped.add(
dropped,
{"stage": "deduplicate", "reason": "duplicate_id"}
)
duration = time.monotonic() * 1000 - start
self.transform_duration.record(duration, {"stage": "deduplicate"})
return df
def _validate_required_fields(self, df: pd.DataFrame) -> pd.DataFrame:
import time
with self.tracer.start_as_current_span("validate_required_fields") as span:
start = time.monotonic() * 1000
required = ["Id", "Name", "StageName", "CloseDate", "Amount_USD"]
before = len(df)
invalid_mask = df[required].isna().any(axis=1)
invalid_records = df[invalid_mask]
df = df[~invalid_mask]
if len(invalid_records) > 0:
for field in required:
null_count = invalid_records[field].isna().sum()
if null_count > 0:
self.records_dropped.add(
null_count,
{"stage": "validate", "reason": f"null_{field}"}
)
logger.warning(
"Required field null in records",
extra={"field": field, "count": null_count}
)
span.set_attribute("validation.invalid_records", before - len(df))
span.set_attribute("validation.valid_records", len(df))
duration = time.monotonic() * 1000 - start
self.transform_duration.record(duration, {"stage": "validate"})
return df
def _convert_to_usd(self, amount: float, currency: str) -> float | None:
# Simplified — in production this would hit an FX rates service
fx_rates = {"USD": 1.0, "EUR": 1.08, "GBP": 1.27, "JPY": 0.0067}
rate = fx_rates.get(currency)
if rate is None:
return None
return amount * rate if amount is not None else None
This transformation code demonstrates something important: data quality signals as first-class observability. The records_dropped counter with a reason label gives you an instant answer to "why are my output counts lower than my input counts?" — something that's nearly impossible to answer from logs alone without searching through thousands of lines.
Key insight
The most valuable pipeline metrics aren't infrastructure metrics (CPU, memory) — they're data flow metrics: records in, records out, records dropped, and why. These metrics tell you whether your pipeline is correct, not just whether it's running.
This connects directly to the broader topic of data quality validation and monitoring in pipelines. OpenTelemetry gives you the instrumentation layer; validation logic gives you the signal worth measuring.
Airflow, Prefect, and similar orchestrators sit above your pipeline code — they're the layer that decides when to run things, handles retries, and manages dependencies. Instrumenting this layer is different from instrumenting the pipeline logic itself, because you need to propagate trace context from the orchestrator into the tasks it spawns.
This is the concept of context propagation: when Airflow starts a task, that task needs to know the trace ID of the parent DAG run so its spans appear as children in the waterfall, not as disconnected traces.
Here's how to implement this in Airflow using a custom operator base class. If you're new to scheduling and orchestrating pipelines with Airflow, this pattern will make more sense once you understand how Airflow passes context between tasks.
# pipeline/orchestration/otel_operator.py
import json
import logging
from functools import wraps
from airflow.models import BaseOperator
from opentelemetry import trace, context, propagate
from opentelemetry.trace import SpanKind, StatusCode
logger = logging.getLogger(__name__)
class OtelBaseOperator(BaseOperator):
"""
Base operator that wraps execute() in an OTel span and propagates
trace context via Airflow XCom.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.tracer = trace.get_tracer("airflow.pipeline")
def execute(self, context: dict):
dag_run_id = context["dag_run"].run_id
dag_id = context["dag"].dag_id
task_id = context["task"].task_id
# Try to restore parent trace context from XCom
# (set by the upstream task or DAG-level setup task)
parent_context = self._extract_parent_context(context)
with self.tracer.start_as_current_span(
f"{dag_id}.{task_id}",
context=parent_context,
kind=SpanKind.INTERNAL,
) as span:
span.set_attribute("airflow.dag_id", dag_id)
span.set_attribute("airflow.task_id", task_id)
span.set_attribute("airflow.run_id", dag_run_id)
span.set_attribute(
"airflow.execution_date",
str(context.get("execution_date", ""))
)
# Propagate this span's context to child tasks via XCom
carrier = {}
propagate.inject(carrier)
context["task_instance"].xcom_push(
key="otel_context",
value=json.dumps(carrier)
)
try:
result = self.execute_with_tracing(context, span)
span.set_status(StatusCode.OK)
return result
except Exception as exc:
span.set_status(StatusCode.ERROR, str(exc))
span.record_exception(exc)
logger.error(
"Task failed",
extra={
"dag_id": dag_id,
"task_id": task_id,
"run_id": dag_run_id,
"error": str(exc),
}
)
raise
def _extract_parent_context(self, airflow_context: dict):
"""
Look for OTel context in XCom from upstream tasks.
Walk upstream task IDs and use the first one found.
"""
ti = airflow_context["task_instance"]
for upstream_task_id in self.upstream_task_ids:
carrier_json = ti.xcom_pull(
task_ids=upstream_task_id,
key="otel_context"
)
if carrier_json:
carrier = json.loads(carrier_json)
return propagate.extract(carrier)
return None
def execute_with_tracing(self, context: dict, span):
"""Override this in subclasses instead of execute()."""
raise NotImplementedError
class IngestOperator(OtelBaseOperator):
def __init__(self, source: str, modified_since: str, **kwargs):
super().__init__(**kwargs)
self.source = source
self.modified_since = modified_since
def execute_with_tracing(self, context: dict, span):
from pipeline.observability import get_meter
from pipeline.ingestion.salesforce import SalesforceIngester
span.set_attribute("ingestion.source", self.source)
span.set_attribute("ingestion.modified_since", self.modified_since)
ingester = SalesforceIngester(
tracer=trace.get_tracer("salesforce.ingestion"),
meter=get_meter("salesforce.ingestion"),
base_url=...,
auth_token=...,
)
records = ingester.ingest_opportunities(self.modified_since)
span.set_attribute("ingestion.records_fetched", len(records))
# Pass records downstream via XCom (or S3 for large volumes)
return {"record_count": len(records), "source": self.source}
Warning
XCom is not a data transport mechanism — it's meant for small metadata like row counts, file paths, or status codes. Never push raw dataframes or large record sets through XCom. For the actual data, write to S3 or a staging area and pass the path. If you're unsure about the right state management approach, the article on checkpointing and state management in long-running data pipelines covers this in depth.
The XCom-based context propagation pattern above is pragmatic but has a limitation: it only works within a single Airflow cluster. If your pipeline spans multiple execution environments (Airflow for orchestration, a separate Python process for heavy transformation), you'll need to serialize the trace context into an external store (Redis, a database column) or pass it as an environment variable to child processes.
The bridge between structured logs and traces is the trace_id. When you emit a log line from within an active OTel span, the Python SDK's LoggingHandler automatically injects trace_id, span_id, and trace_flags into the log record. But you need to make sure your log formatter exposes them.
# pipeline/observability.py (additions)
import logging
from opentelemetry import trace as otel_trace
class OtelCorrelatedFormatter(logging.Formatter):
"""
Log formatter that injects OTel trace context into every record.
Compatible with structured logging systems (Loki, CloudWatch Logs Insights,
Elasticsearch).
"""
def format(self, record: logging.LogRecord) -> str:
# Pull active span context if present
span = otel_trace.get_current_span()
ctx = span.get_span_context()
if ctx and ctx.is_valid:
record.trace_id = format(ctx.trace_id, "032x")
record.span_id = format(ctx.span_id, "016x")
record.trace_flags = str(ctx.trace_flags)
else:
record.trace_id = "0" * 32
record.span_id = "0" * 16
record.trace_flags = "00"
return super().format(record)
def configure_structured_logging():
handler = logging.StreamHandler()
handler.setFormatter(
OtelCorrelatedFormatter(
fmt='{"timestamp": "%(asctime)s", "level": "%(levelname)s", '
'"logger": "%(name)s", "message": "%(message)s", '
'"trace_id": "%(trace_id)s", "span_id": "%(span_id)s"}'
)
)
root = logging.getLogger()
root.addHandler(handler)
root.setLevel(logging.INFO)
With this in place, every log line emitted during an active span will carry the trace ID. In Grafana, you can configure Loki as a log data source and Tempo as a trace data source, then set up a "derived field" that turns trace_id values in log lines into clickable links that open the corresponding trace in Tempo. The result: from a log line showing "Required field null in records", one click takes you to the full 45-second trace waterfall.
Beyond start/end timing, spans support events — timestamped annotations you can attach to a span without creating a child span. These are perfect for pipeline milestones that aren't worth a full span but are worth recording:
def load_to_snowflake(df: pd.DataFrame, table: str, tracer, meter):
rows_loaded = meter.create_counter(
"pipeline.rows.loaded",
unit="rows"
)
load_errors = meter.create_counter(
"pipeline.load.errors",
unit="errors"
)
with tracer.start_as_current_span("load_to_snowflake") as span:
span.set_attribute("target.table", table)
span.set_attribute("load.row_count", len(df))
# Stage to S3 first
s3_path = f"s3://pipeline-staging/{table}/{int(time.time())}.parquet"
df.to_parquet(s3_path)
# Record a span event at the staging checkpoint
span.add_event(
"staged_to_s3",
attributes={
"s3_path": s3_path,
"file_size_bytes": df.memory_usage(deep=True).sum(),
"row_count": len(df),
}
)
# COPY INTO Snowflake
try:
conn = get_snowflake_connection()
cursor = conn.cursor()
cursor.execute(
f"COPY INTO {table} FROM '{s3_path}' FILE_FORMAT = (TYPE = PARQUET)"
)
result = cursor.fetchone()
rows_copied = result[0] if result else 0
span.add_event(
"copy_into_complete",
attributes={"rows_copied": rows_copied, "table": table}
)
rows_loaded.add(rows_copied, {"table": table})
span.set_attribute("load.rows_actually_copied", rows_copied)
# Detect silent data loss: staged more than landed
if rows_copied < len(df):
discrepancy = len(df) - rows_copied
span.add_event(
"load_discrepancy_detected",
attributes={
"staged": len(df),
"loaded": rows_copied,
"lost": discrepancy,
}
)
logger.error(
"Load discrepancy: rows staged vs rows copied differ",
extra={"staged": len(df), "loaded": rows_copied}
)
except Exception as exc:
load_errors.add(1, {"table": table, "error_type": type(exc).__name__})
span.record_exception(exc)
raise
Span events are searchable in Jaeger and Grafana Tempo. That load_discrepancy_detected event is exactly the kind of silent failure that would take hours to find in raw logs — with span events, you can query "show me all traces that contain the event load_discrepancy_detected" and get every affected pipeline run, with full context.
This complements the broader strategies for pipeline error handling and recovery — OTel gives you the visibility to know when to trigger those recovery paths.
In production, you don't export spans directly from your pipeline process to Jaeger or Grafana. You export to an OpenTelemetry Collector — a standalone process that acts as a telemetry proxy. The collector buffers, batches, filters, and routes signals to multiple backends simultaneously.
A typical production topology:
Pipeline Process
└─ OTLP gRPC → OTel Collector
├─ Traces → Grafana Tempo (or Jaeger)
├─ Metrics → Prometheus remote write
└─ Logs → Loki
A minimal collector config (otel-collector-config.yaml):
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 10s
send_batch_size: 1024
memory_limiter:
limit_mib: 512
spike_limit_mib: 128
check_interval: 5s
# Drop high-cardinality attributes before they hit your backend
attributes/sanitize:
actions:
- key: "record.id"
action: delete
- key: "batch.uuid"
action: delete
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: false
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
loki:
endpoint: "http://loki:3100/loki/api/v1/push"
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, attributes/sanitize, batch]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
For high-throughput pipelines, exporting every span can generate gigabytes of trace data per day. You need a sampling strategy. OTel supports two approaches:
Head-based sampling (decided at span creation): Fast and simple, but you don't yet know if this trace is interesting. Use a TraceIdRatioBased sampler for a consistent percentage, or ParentBased to inherit the parent's sampling decision:
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased
# Sample 10% of traces in production
sampler = ParentBased(root=TraceIdRatioBased(0.1))
tracer_provider = TracerProvider(resource=resource, sampler=sampler)
Tail-based sampling (decided at the collector after the trace completes): More expensive but far more valuable — you can sample 100% of error traces and 1% of success traces. The OTel Collector's tailsampling processor supports this:
processors:
tail_sampling:
decision_wait: 30s
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow-pipelines-policy
type: latency
latency: {threshold_ms: 60000}
- name: low-rate-baseline
type: probabilistic
probabilistic: {sampling_percentage: 5}
Tip
For pipeline observability, tail-based sampling is almost always the right choice. You want 100% of failed runs and slow runs. Healthy, fast runs are less interesting — a 5% sample is plenty for baseline performance tracking.
Build an end-to-end instrumented mini-pipeline that processes public GitHub events data. Here's the structure:
Setup: Run Jaeger locally (docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest)
Task 1: Instrumented ingestion
Write a function ingest_github_events(org: str, repo: str) that:
ingest_github_events with org and repo as attributeshttps://api.github.com/repos/{org}/{repo}/events (up to 3 pages)github.events.fetched counter metric with org and repo labelsTask 2: Instrumented transformation
Write a function transform_events(raw_events: list) that:
transform_eventsextract_actor_metrics, groups events by actor login and counts event typesfilter_bot_actors, removes actors whose logins end in [bot]transform.bot_records_dropped counter when bots are filteredtransformation_summary with input count, output count, and bot countTask 3: Correlated logging
Ensure all log lines emitted during the pipeline carry trace_id and span_id. Verify by running the pipeline and checking that the log output contains valid hex trace IDs.
Verification: Open Jaeger at http://localhost:16686, search for your service name, and find the full trace. Confirm you can see the page-level child spans and the transformation spans as siblings under a common root.
# WRONG — creates a new counter on every function call
def process_batch(records):
counter = meter.create_counter("records_processed") # recreated each call!
counter.add(len(records))
# RIGHT — create instruments once, at class or module initialization
class BatchProcessor:
def __init__(self, meter):
self.records_processed = meter.create_counter("records_processed")
def process_batch(self, records):
self.records_processed.add(len(records))
Creating the same instrument multiple times with the same name is technically idempotent in OTel (it returns the same underlying instrument), but it wastes allocations and obscures intent.
# WRONG — user_id has millions of values, creates millions of time series
meter.create_counter("records_processed").add(1, {"user_id": user_id})
# RIGHT — use bounded categorical labels
meter.create_counter("records_processed").add(1, {"pipeline": "salesforce-opportunities"})
This will silently destroy your Prometheus instance. See the backpressure and bottleneck diagnosis article for how unbounded cardinality can manifest as backend degradation rather than pipeline-side errors.
When you submit work to a thread pool or subprocess, the active span context does not automatically propagate:
# WRONG — the thread has no span context
with tracer.start_as_current_span("process_partition") as span:
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [executor.submit(process_chunk, chunk) for chunk in chunks]
# process_chunk() has no active span, its logs won't be correlated
# RIGHT — capture and pass the context explicitly
from opentelemetry import context as otel_context
with tracer.start_as_current_span("process_partition") as span:
ctx = otel_context.get_current()
def process_chunk_with_context(chunk):
token = otel_context.attach(ctx)
try:
return process_chunk(chunk)
finally:
otel_context.detach(token)
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [executor.submit(process_chunk_with_context, chunk) for chunk in chunks]
A span that wraps an entire 45-minute pipeline run is nearly useless — you can't tell where time was spent. A span for every individual record in a 10-million-row batch will generate so much trace data it crashes your collector. The right granularity is logical stages (fetch, transform, load) and iteration boundaries (per-page, per-partition), not individual records.
A common rule of thumb: if a span would exist for less than 1 millisecond or more than 10 minutes, reconsider whether it should be a span, a metric, or a span event instead.
The BatchSpanProcessor exports asynchronously. If your pipeline process exits before the flush completes, you lose the last batch of spans:
# In your pipeline entry point
import signal
import atexit
def shutdown_otel():
trace.get_tracer_provider().shutdown()
metrics.get_meter_provider().shutdown()
atexit.register(shutdown_otel)
This matters especially in containerized environments where SIGTERM can arrive at any time. The shutdown() call flushes all pending spans/metrics before the provider is destroyed.
Warning
If you're running pipelines in AWS Lambda or Google Cloud Run (short-lived containers), the BatchSpanProcessor with its default async export may not finish before the container freezes. Use SimpleSpanProcessor for serverless environments, or configure the exporter with a very short export_timeout_millis.
You now have a complete implementation pattern for OpenTelemetry-based observability across all three layers of a data pipeline. Let's consolidate what we've built:
Traces give you causal, time-ordered chains of operations — from the moment an orchestrator kicks off a DAG run to when rows land in the warehouse. Span attributes make each stage self-documenting. Span events capture milestones and anomalies (like silent data loss) without creating span overhead for every record.
Metrics give you aggregate health signals over time — records ingested, transformation drop rates, load latency distributions. The critical discipline is keeping label cardinality bounded: use categorical labels (pipeline name, stage, error type), never entity IDs.
Logs become dramatically more useful when they carry trace_id and span_id. With a correlated formatter, every log line is pivotable to its full trace context. This turns "search through 10,000 log lines" into "click to open the trace that contains this log."
The OpenTelemetry Collector is your production-grade buffer and router — it decouples your pipeline code from your observability backend, enables tail-based sampling, and lets you evolve your backend choices without changing application code.
Where to go from here:
The 2:30 AM page doesn't go away. But with OpenTelemetry instrumented across your pipeline, you'll spend the next five minutes understanding exactly what happened — not the next two hours guessing.
Data Pipeline Fundamentals