Wicked Smart Data
LearnInsightsAboutContact
Sign InLet's Build
LearnInsightsAboutContact
Sign InLet's Build
Wicked Smart Data

Intelligence, automation, and expert execution — plus an elite library of free knowledge. We turn complexity into competitive advantage.

Start a conversation

Platform

  • Learning Paths
  • Insights
  • RSS Feed

Company

  • About
  • Contact
  • Work With Us

Legal

  • Privacy Policy
  • Terms of Service

© 2026 Wicked Smart Data. All rights reserved.

Intelligence · Automation · Advantage

All Insights
Data Engineering

Implementing Pipeline Observability with OpenTelemetry: Tracing Spans, Metrics, and Logs Across Ingestion, Transformation, and Orchestration Layers

Most data pipeline failures are invisible until a downstream consumer notices stale data. This lesson shows you how to implement unified observability across ingestion, transformation, and orchestration using OpenTelemetry — with working code for distributed traces, pipeline-specific metrics, and correlated structured logs.

🔥 Expert26 min readNov 19, 2026Updated Nov 19, 2026
Implementing Pipeline Observability with OpenTelemetry: Tracing Spans, Metrics, and Logs Across Ingestion, Transformation, and Orchestration Layers
On this page
  • Introduction
  • Prerequisites
  • Why OpenTelemetry? The Case for a Unified Standard
  • Understanding the Three Pillars in a Pipeline Context
  • Traces and Spans
  • Metrics
  • Logs
  • Setting Up the OpenTelemetry SDK
  • Instrumenting the Ingestion Layer
  • Instrumenting the Transformation Layer
  • Instrumenting the Orchestration Layer
  • Correlating Logs with Traces
  • Pipeline-Level Span Events and Checkpoints
  • Production Deployment: Collector, Sampling, and Backends
  • Sampling Strategies
  • Hands-On Exercise
  • Common Mistakes & Troubleshooting
  • Mistake 1: Creating instruments inside functions
  • Mistake 2: High-cardinality label values
  • Mistake 3: Losing trace context across thread or process boundaries
  • Mistake 4: Spans that are too coarse or too fine
  • Mistake 5: Not flushing on shutdown
  • Summary & Next Steps

Implementing Pipeline Observability with OpenTelemetry: Tracing Spans, Metrics, and Logs Across Ingestion, Transformation, and Orchestration Layers

Introduction

It's 2:30 AM and your on-call engineer gets paged. A downstream dashboard is showing stale data — revenue figures that stopped updating six hours ago. The Airflow UI shows all tasks green. The database is healthy. The API source returned 200s all day. But somewhere between the API call and the warehouse, 400,000 rows evaporated.

Without proper observability, the next two hours look like this: grepping through fragmented log files, manually querying row counts at each stage, checking dead letter queues, and squinting at CloudWatch metrics that don't align with your pipeline's logical boundaries. You're not debugging a system — you're doing archaeology.

This is exactly the problem that unified pipeline observability solves, and OpenTelemetry (OTel) is the industry's most capable framework for building it. By the end of this lesson, you'll have a complete mental model and working implementation for instrumenting a multi-stage data pipeline with distributed traces, structured metrics, and correlated logs — all using OpenTelemetry's vendor-neutral SDK. When something breaks at 2:30 AM, you'll open a single trace view and see exactly which span failed, how long each stage took, what data volumes moved through each layer, and why.

What you'll learn:

  • The three pillars of observability (traces, metrics, logs) and how they complement each other in a pipeline context
  • How to instrument ingestion, transformation, and orchestration code with OpenTelemetry spans and attributes
  • How to emit pipeline-specific metrics (record counts, lag, error rates) as OTel gauges and histograms
  • How to correlate structured logs with active trace and span IDs for unified debugging
  • How to export signals to backends like Jaeger, Prometheus, and Loki, and how to wire them together in production
  • Architecture-level trade-offs between instrumentation overhead, cardinality, and operational cost

Prerequisites

This lesson assumes you're comfortable with Python data pipelines, have worked with at least one orchestration tool (Airflow, Prefect, or similar), and understand the basics of distributed systems. You should know what a DAG is, how HTTP APIs work, and have passing familiarity with Docker for running local backends. If you need a refresher on what a data pipeline is and how its core layers fit together, that foundation will help you understand why each layer needs distinct instrumentation strategies.

You'll need: Python 3.9+, the opentelemetry-sdk packages, and optionally Docker for running Jaeger or Prometheus locally.


Why OpenTelemetry? The Case for a Unified Standard

Before we touch any code, let's be precise about why OpenTelemetry matters for data pipelines specifically — not just microservices.

Data pipelines are distributed systems with a distinctive shape. Unlike a web request that completes in milliseconds, a pipeline run spans seconds to hours, crosses multiple compute boundaries (a Kubernetes pod running your ingestion script, a Spark cluster doing transformation, an Airflow worker managing orchestration), and processes data in bulk operations where individual record errors matter as much as overall task health.

Traditional application monitoring tools were built for request/response systems. They assume short-lived operations, homogeneous compute, and binary pass/fail semantics. Pipeline operators end up stitching together: Airflow's built-in task logs (unstructured), CloudWatch metrics (infrastructure-level), custom print() statements (please no), and bespoke alerting scripts.

OpenTelemetry gives you a single SDK that emits the three signals — traces (causal, timing-aware chains of operations), metrics (numeric measurements over time), and logs (timestamped events with structured context) — and a wire protocol (OTLP) that ships all three to any backend you choose. For pipeline engineers, this means:

  • A trace that starts when an Airflow DAG run begins and ends when rows land in the warehouse, with every intermediate operation as a child span
  • Metrics that capture rows ingested, transformation error rate, and load latency with pipeline-specific labels
  • Structured log entries that carry the same trace_id and span_id as the active trace, so you can pivot between "show me all logs for this specific pipeline run" and "show me the full trace waterfall"

Key insight

OpenTelemetry's value isn't just vendor neutrality (though that matters). It's the correlation model — the ability to link a log line, a metric spike, and a trace span back to the same causal event. That correlation is what turns reactive debugging into proactive understanding.

The OpenTelemetry specification defines a clean separation: the API (interfaces and no-op implementations, safe to include in library code), the SDK (the actual implementation, configured by your application), and exporters (the bridge to backends). This layering matters when you're building shared pipeline framework code — your library instruments with the API, and operators decide where signals go by configuring the SDK.


Understanding the Three Pillars in a Pipeline Context

Traces and Spans

A trace is a directed acyclic graph of spans, where each span represents a unit of work with a start time, end time, name, and set of key-value attributes. The critical property is causality: spans carry a parent-child relationship and a shared trace_id, so you can reconstruct "this span was triggered by that span, which was triggered by this root operation."

For a pipeline that ingests from an API, transforms in Pandas, and loads to Snowflake, a trace might look like:

pipeline_run [trace_id: abc123] (45.2s)
├── ingest_from_salesforce_api (12.1s)
│   ├── fetch_page[1] (1.4s)
│   ├── fetch_page[2] (1.3s)
│   └── ... (8 more pages)
├── transform_opportunity_records (28.4s)
│   ├── normalize_currency_fields (3.2s)
│   ├── deduplicate_on_external_id (18.1s)  ← this is where your bottleneck lives
│   └── validate_schema (7.1s)
└── load_to_snowflake (4.7s)
    ├── stage_to_s3 (2.1s)
    └── copy_into_opportunities (2.6s)

When deduplication takes 18 seconds instead of its usual 2, you don't need to guess — the trace tells you.

Metrics

Metrics are numeric measurements aggregated over time. OTel supports several instrument types:

  • Counter: monotonically increasing (total rows ingested, total API calls made)
  • Gauge: point-in-time value (current queue depth, number of records in flight)
  • Histogram: distribution of values (record processing latency, batch size distribution)
  • UpDownCounter: can increase or decrease (active pipeline workers)

The key design decision for pipeline metrics is cardinality. Each unique combination of label values creates a new time series in your metrics backend. pipeline_name + source_system + environment is fine. Adding record_id or batch_uuid as a label will blow up your Prometheus storage and your cloud bill. We'll return to this in the mistakes section.

Logs

Structured logs in the OTel model carry a trace_id and span_id on every entry when emitted within an active span context. This is the bridge between your existing logging infrastructure and your new trace data. You don't have to replace your logger — you bridge it.


Setting Up the OpenTelemetry SDK

Let's build from the ground up. First, install the core packages:

pip install \
  opentelemetry-sdk \
  opentelemetry-api \
  opentelemetry-exporter-otlp-proto-grpc \
  opentelemetry-exporter-prometheus \
  opentelemetry-instrumentation-logging

For local development, run Jaeger all-in-one via Docker (it accepts OTLP and renders trace waterfalls):

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

Now, the SDK bootstrap. This is where most tutorials cut corners and you pay for it later. Create a dedicated observability.py module in your pipeline package:

# pipeline/observability.py

import logging
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

def configure_observability(
    service_name: str,
    service_version: str,
    otlp_endpoint: str = "localhost:4317",
    environment: str = "development"
) -> tuple:
    """
    Bootstrap OTel SDK for a pipeline service.
    Returns (tracer, meter) for use in pipeline code.
    Call once at process startup, before any pipeline work begins.
    """
    resource = Resource.create({
        SERVICE_NAME: service_name,
        SERVICE_VERSION: service_version,
        "deployment.environment": environment,
        "pipeline.framework": "custom-python",
    })

    # --- Traces ---
    tracer_provider = TracerProvider(resource=resource)
    span_exporter = OTLPSpanExporter(
        endpoint=otlp_endpoint,
        insecure=True  # use TLS in production
    )
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            span_exporter,
            max_export_batch_size=512,
            export_timeout_millis=30_000,
        )
    )
    trace.set_tracer_provider(tracer_provider)

    # --- Metrics ---
    metric_exporter = OTLPMetricExporter(
        endpoint=otlp_endpoint,
        insecure=True
    )
    metric_reader = PeriodicExportingMetricReader(
        metric_exporter,
        export_interval_millis=15_000  # 15s scrape interval
    )
    meter_provider = MeterProvider(
        resource=resource,
        metric_readers=[metric_reader]
    )
    metrics.set_meter_provider(meter_provider)

    # --- Logs ---
    logger_provider = LoggerProvider(resource=resource)
    log_exporter = OTLPLogExporter(endpoint=otlp_endpoint, insecure=True)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(log_exporter)
    )
    set_logger_provider(logger_provider)

    # Bridge Python's standard logging to OTel
    otel_handler = LoggingHandler(
        level=logging.DEBUG,
        logger_provider=logger_provider
    )
    logging.getLogger().addHandler(otel_handler)
    logging.getLogger().setLevel(logging.INFO)

    tracer = trace.get_tracer(service_name, service_version)
    meter = metrics.get_meter(service_name, service_version)

    return tracer, meter

Warning

Call configure_observability() exactly once per process, at startup — not inside functions or class constructors that might be called multiple times. Multiple calls to set_tracer_provider() will silently replace the previous provider, dropping any spans that were in flight in the BatchSpanProcessor queue.

The Resource object is not cosmetic. Every span, metric, and log record inherits these attributes, and they're how your backend knows this telemetry came from the salesforce-ingestion service running version 2.4.1 in production. Without a proper resource, you're debugging blind in multi-service deployments.


Instrumenting the Ingestion Layer

The ingestion layer — where you pull from APIs, databases, or file systems — has a characteristic shape: a loop over pages or chunks, with network I/O on each iteration. This is exactly where distributed tracing shines, because you can see not just the aggregate cost but the per-page timing.

Here's a realistic Salesforce API ingestion function, fully instrumented. For more context on the API patterns we're wrapping here, see the deep dive on advanced API integration for data pipelines.

# pipeline/ingestion/salesforce.py

import logging
import time
import requests
from opentelemetry import trace
from opentelemetry.trace import SpanKind, StatusCode
from opentelemetry.semconv.trace import SpanAttributes

logger = logging.getLogger(__name__)

class SalesforceIngester:
    def __init__(self, tracer, meter, base_url: str, auth_token: str):
        self.tracer = tracer
        self.base_url = base_url
        self.auth_token = auth_token

        # Define metrics instruments once at construction time
        self.records_ingested = meter.create_counter(
            name="pipeline.records.ingested",
            description="Total records successfully ingested",
            unit="records",
        )
        self.api_request_duration = meter.create_histogram(
            name="pipeline.api.request_duration",
            description="Duration of individual API page requests",
            unit="ms",
        )
        self.api_errors = meter.create_counter(
            name="pipeline.api.errors",
            description="Total API request errors by type",
            unit="errors",
        )
        self.batch_size = meter.create_histogram(
            name="pipeline.batch.size",
            description="Number of records per API page",
            unit="records",
        )

    def ingest_opportunities(
        self,
        modified_since: str,
        parent_span_context=None
    ) -> list[dict]:
        """
        Paginate through Salesforce Opportunity records modified after
        modified_since (ISO 8601). Returns all records as a list.
        """
        with self.tracer.start_as_current_span(
            "ingest_opportunities",
            kind=SpanKind.CLIENT,
            context=parent_span_context,
        ) as span:
            span.set_attribute("source.system", "salesforce")
            span.set_attribute("source.object", "Opportunity")
            span.set_attribute("ingestion.modified_since", modified_since)
            span.set_attribute(SpanAttributes.HTTP_URL, self.base_url)

            all_records = []
            page = 1
            next_url = (
                f"{self.base_url}/services/data/v58.0/query"
                f"?q=SELECT+Id,Name,Amount,StageName,CloseDate"
                f"+FROM+Opportunity+WHERE+LastModifiedDate+>+{modified_since}"
            )

            try:
                while next_url:
                    page_records, next_url = self._fetch_page(
                        next_url, page
                    )
                    all_records.extend(page_records)
                    page += 1

                span.set_attribute("ingestion.total_records", len(all_records))
                span.set_attribute("ingestion.total_pages", page - 1)
                span.set_status(StatusCode.OK)
                logger.info(
                    "Salesforce ingestion complete",
                    extra={
                        "records": len(all_records),
                        "pages": page - 1,
                        "source": "salesforce",
                    }
                )
                return all_records

            except Exception as exc:
                span.set_status(StatusCode.ERROR, str(exc))
                span.record_exception(exc)
                raise

    def _fetch_page(self, url: str, page_num: int) -> tuple[list, str | None]:
        with self.tracer.start_as_current_span(
            f"fetch_page",
            kind=SpanKind.CLIENT,
        ) as span:
            span.set_attribute("http.page_number", page_num)
            span.set_attribute(SpanAttributes.HTTP_URL, url)
            span.set_attribute(SpanAttributes.HTTP_METHOD, "GET")

            start_ms = time.monotonic() * 1000
            labels = {"source": "salesforce", "object": "Opportunity"}

            try:
                response = requests.get(
                    url,
                    headers={"Authorization": f"Bearer {self.auth_token}"},
                    timeout=30,
                )
                response.raise_for_status()

                duration_ms = time.monotonic() * 1000 - start_ms
                data = response.json()
                records = data.get("records", [])

                # Record metrics
                self.api_request_duration.record(duration_ms, labels)
                self.records_ingested.add(len(records), labels)
                self.batch_size.record(len(records), labels)

                # Annotate span with response details
                span.set_attribute(SpanAttributes.HTTP_STATUS_CODE, response.status_code)
                span.set_attribute("response.record_count", len(records))
                span.set_attribute("response.duration_ms", round(duration_ms, 2))
                span.set_status(StatusCode.OK)

                logger.debug(
                    "Fetched API page",
                    extra={"page": page_num, "records": len(records), "duration_ms": duration_ms}
                )

                next_url = data.get("nextRecordsUrl")
                if next_url:
                    next_url = f"{self.base_url}{next_url}"

                return records, next_url

            except requests.HTTPError as exc:
                duration_ms = time.monotonic() * 1000 - start_ms
                error_labels = {**labels, "error_type": type(exc).__name__}
                self.api_errors.add(1, error_labels)
                self.api_request_duration.record(duration_ms, labels)

                span.set_attribute(SpanAttributes.HTTP_STATUS_CODE, exc.response.status_code)
                span.set_status(StatusCode.ERROR, str(exc))
                span.record_exception(exc)
                raise

A few design choices worth explaining here. We use SpanKind.CLIENT for outbound API calls — this tells trace backends to render these spans as external calls rather than internal processing steps, which affects how latency is attributed in service maps. We call span.record_exception(exc) before re-raising, which writes the exception type, message, and stack trace as a span event (not just a status update). And we set span attributes before the potentially-failing operation where possible, so if we crash mid-flight, we still have context.

Tip

Use OpenTelemetry Semantic Conventions (opentelemetry.semconv.trace.SpanAttributes) for well-known attributes like http.url, http.method, and db.statement. Backends like Jaeger, Grafana Tempo, and Honeycomb understand these standard keys and render them with special UI treatment — URL parsing, HTTP status color coding, etc.


Instrumenting the Transformation Layer

Transformation is where pipelines do the most interesting work and also where the most subtle failures occur — data type coercions that silently drop records, deduplication logic that's slightly wrong, currency conversions that use the wrong exchange rate. Tracing alone isn't enough here; you need metrics that track data quality signals alongside performance signals.

# pipeline/transformation/opportunities.py

import logging
import pandas as pd
from opentelemetry import trace
from opentelemetry.trace import StatusCode

logger = logging.getLogger(__name__)

class OpportunityTransformer:
    def __init__(self, tracer, meter):
        self.tracer = tracer

        # Data quality metrics
        self.records_dropped = meter.create_counter(
            name="pipeline.transform.records_dropped",
            description="Records dropped during transformation by reason",
            unit="records",
        )
        self.transform_duration = meter.create_histogram(
            name="pipeline.transform.duration",
            description="Duration of transformation steps",
            unit="ms",
        )
        self.null_field_rate = meter.create_histogram(
            name="pipeline.transform.null_field_rate",
            description="Fraction of records with null in a field",
            unit="ratio",
        )

    def transform(self, raw_records: list[dict]) -> pd.DataFrame:
        with self.tracer.start_as_current_span("transform_opportunities") as span:
            span.set_attribute("transform.input_records", len(raw_records))
            df = pd.DataFrame(raw_records)

            try:
                df = self._normalize_currency(df)
                df = self._deduplicate(df)
                df = self._validate_required_fields(df)
                df = self._coerce_types(df)

                span.set_attribute("transform.output_records", len(df))
                span.set_attribute(
                    "transform.drop_rate",
                    round(1 - len(df) / max(len(raw_records), 1), 4)
                )
                span.set_status(StatusCode.OK)
                return df

            except Exception as exc:
                span.set_status(StatusCode.ERROR, str(exc))
                span.record_exception(exc)
                raise

    def _normalize_currency(self, df: pd.DataFrame) -> pd.DataFrame:
        import time
        with self.tracer.start_as_current_span("normalize_currency") as span:
            start = time.monotonic() * 1000

            # Convert all amounts to USD using a lookup table
            # (in reality, this might call an FX rates API or lookup table)
            non_usd = df[df["CurrencyIsoCode"] != "USD"]
            span.set_attribute("currency.non_usd_records", len(non_usd))

            df["Amount_USD"] = df.apply(
                lambda row: self._convert_to_usd(
                    row["Amount"], row.get("CurrencyIsoCode", "USD")
                ),
                axis=1,
            )

            null_amounts = df["Amount_USD"].isna().sum()
            if null_amounts > 0:
                self.records_dropped.add(
                    null_amounts,
                    {"stage": "normalize_currency", "reason": "unconvertible_currency"}
                )
                self.null_field_rate.record(
                    null_amounts / len(df),
                    {"field": "Amount_USD", "stage": "normalize_currency"}
                )
                logger.warning(
                    "Records with unconvertible currency detected",
                    extra={"count": null_amounts, "field": "Amount_USD"}
                )

            duration = time.monotonic() * 1000 - start
            self.transform_duration.record(
                duration, {"stage": "normalize_currency"}
            )
            span.set_attribute("duration_ms", round(duration, 2))
            return df

    def _deduplicate(self, df: pd.DataFrame) -> pd.DataFrame:
        import time
        with self.tracer.start_as_current_span("deduplicate_records") as span:
            start = time.monotonic() * 1000
            before = len(df)

            df = df.sort_values("LastModifiedDate", ascending=False)
            df = df.drop_duplicates(subset=["Id"], keep="first")

            after = len(df)
            dropped = before - after

            span.set_attribute("dedup.input_records", before)
            span.set_attribute("dedup.output_records", after)
            span.set_attribute("dedup.duplicates_removed", dropped)

            if dropped > 0:
                self.records_dropped.add(
                    dropped,
                    {"stage": "deduplicate", "reason": "duplicate_id"}
                )

            duration = time.monotonic() * 1000 - start
            self.transform_duration.record(duration, {"stage": "deduplicate"})
            return df

    def _validate_required_fields(self, df: pd.DataFrame) -> pd.DataFrame:
        import time
        with self.tracer.start_as_current_span("validate_required_fields") as span:
            start = time.monotonic() * 1000

            required = ["Id", "Name", "StageName", "CloseDate", "Amount_USD"]
            before = len(df)

            invalid_mask = df[required].isna().any(axis=1)
            invalid_records = df[invalid_mask]
            df = df[~invalid_mask]

            if len(invalid_records) > 0:
                for field in required:
                    null_count = invalid_records[field].isna().sum()
                    if null_count > 0:
                        self.records_dropped.add(
                            null_count,
                            {"stage": "validate", "reason": f"null_{field}"}
                        )
                        logger.warning(
                            "Required field null in records",
                            extra={"field": field, "count": null_count}
                        )

            span.set_attribute("validation.invalid_records", before - len(df))
            span.set_attribute("validation.valid_records", len(df))
            duration = time.monotonic() * 1000 - start
            self.transform_duration.record(duration, {"stage": "validate"})
            return df

    def _convert_to_usd(self, amount: float, currency: str) -> float | None:
        # Simplified — in production this would hit an FX rates service
        fx_rates = {"USD": 1.0, "EUR": 1.08, "GBP": 1.27, "JPY": 0.0067}
        rate = fx_rates.get(currency)
        if rate is None:
            return None
        return amount * rate if amount is not None else None

This transformation code demonstrates something important: data quality signals as first-class observability. The records_dropped counter with a reason label gives you an instant answer to "why are my output counts lower than my input counts?" — something that's nearly impossible to answer from logs alone without searching through thousands of lines.

Key insight

The most valuable pipeline metrics aren't infrastructure metrics (CPU, memory) — they're data flow metrics: records in, records out, records dropped, and why. These metrics tell you whether your pipeline is correct, not just whether it's running.

This connects directly to the broader topic of data quality validation and monitoring in pipelines. OpenTelemetry gives you the instrumentation layer; validation logic gives you the signal worth measuring.


Instrumenting the Orchestration Layer

Airflow, Prefect, and similar orchestrators sit above your pipeline code — they're the layer that decides when to run things, handles retries, and manages dependencies. Instrumenting this layer is different from instrumenting the pipeline logic itself, because you need to propagate trace context from the orchestrator into the tasks it spawns.

This is the concept of context propagation: when Airflow starts a task, that task needs to know the trace ID of the parent DAG run so its spans appear as children in the waterfall, not as disconnected traces.

Here's how to implement this in Airflow using a custom operator base class. If you're new to scheduling and orchestrating pipelines with Airflow, this pattern will make more sense once you understand how Airflow passes context between tasks.

# pipeline/orchestration/otel_operator.py

import json
import logging
from functools import wraps
from airflow.models import BaseOperator
from opentelemetry import trace, context, propagate
from opentelemetry.trace import SpanKind, StatusCode

logger = logging.getLogger(__name__)

class OtelBaseOperator(BaseOperator):
    """
    Base operator that wraps execute() in an OTel span and propagates
    trace context via Airflow XCom.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.tracer = trace.get_tracer("airflow.pipeline")

    def execute(self, context: dict):
        dag_run_id = context["dag_run"].run_id
        dag_id = context["dag"].dag_id
        task_id = context["task"].task_id

        # Try to restore parent trace context from XCom
        # (set by the upstream task or DAG-level setup task)
        parent_context = self._extract_parent_context(context)

        with self.tracer.start_as_current_span(
            f"{dag_id}.{task_id}",
            context=parent_context,
            kind=SpanKind.INTERNAL,
        ) as span:
            span.set_attribute("airflow.dag_id", dag_id)
            span.set_attribute("airflow.task_id", task_id)
            span.set_attribute("airflow.run_id", dag_run_id)
            span.set_attribute(
                "airflow.execution_date",
                str(context.get("execution_date", ""))
            )

            # Propagate this span's context to child tasks via XCom
            carrier = {}
            propagate.inject(carrier)
            context["task_instance"].xcom_push(
                key="otel_context",
                value=json.dumps(carrier)
            )

            try:
                result = self.execute_with_tracing(context, span)
                span.set_status(StatusCode.OK)
                return result
            except Exception as exc:
                span.set_status(StatusCode.ERROR, str(exc))
                span.record_exception(exc)
                logger.error(
                    "Task failed",
                    extra={
                        "dag_id": dag_id,
                        "task_id": task_id,
                        "run_id": dag_run_id,
                        "error": str(exc),
                    }
                )
                raise

    def _extract_parent_context(self, airflow_context: dict):
        """
        Look for OTel context in XCom from upstream tasks.
        Walk upstream task IDs and use the first one found.
        """
        ti = airflow_context["task_instance"]
        for upstream_task_id in self.upstream_task_ids:
            carrier_json = ti.xcom_pull(
                task_ids=upstream_task_id,
                key="otel_context"
            )
            if carrier_json:
                carrier = json.loads(carrier_json)
                return propagate.extract(carrier)
        return None

    def execute_with_tracing(self, context: dict, span):
        """Override this in subclasses instead of execute()."""
        raise NotImplementedError


class IngestOperator(OtelBaseOperator):
    def __init__(self, source: str, modified_since: str, **kwargs):
        super().__init__(**kwargs)
        self.source = source
        self.modified_since = modified_since

    def execute_with_tracing(self, context: dict, span):
        from pipeline.observability import get_meter
        from pipeline.ingestion.salesforce import SalesforceIngester

        span.set_attribute("ingestion.source", self.source)
        span.set_attribute("ingestion.modified_since", self.modified_since)

        ingester = SalesforceIngester(
            tracer=trace.get_tracer("salesforce.ingestion"),
            meter=get_meter("salesforce.ingestion"),
            base_url=...,
            auth_token=...,
        )
        records = ingester.ingest_opportunities(self.modified_since)
        span.set_attribute("ingestion.records_fetched", len(records))

        # Pass records downstream via XCom (or S3 for large volumes)
        return {"record_count": len(records), "source": self.source}

Warning

XCom is not a data transport mechanism — it's meant for small metadata like row counts, file paths, or status codes. Never push raw dataframes or large record sets through XCom. For the actual data, write to S3 or a staging area and pass the path. If you're unsure about the right state management approach, the article on checkpointing and state management in long-running data pipelines covers this in depth.

The XCom-based context propagation pattern above is pragmatic but has a limitation: it only works within a single Airflow cluster. If your pipeline spans multiple execution environments (Airflow for orchestration, a separate Python process for heavy transformation), you'll need to serialize the trace context into an external store (Redis, a database column) or pass it as an environment variable to child processes.


Correlating Logs with Traces

The bridge between structured logs and traces is the trace_id. When you emit a log line from within an active OTel span, the Python SDK's LoggingHandler automatically injects trace_id, span_id, and trace_flags into the log record. But you need to make sure your log formatter exposes them.

# pipeline/observability.py (additions)

import logging
from opentelemetry import trace as otel_trace

class OtelCorrelatedFormatter(logging.Formatter):
    """
    Log formatter that injects OTel trace context into every record.
    Compatible with structured logging systems (Loki, CloudWatch Logs Insights,
    Elasticsearch).
    """

    def format(self, record: logging.LogRecord) -> str:
        # Pull active span context if present
        span = otel_trace.get_current_span()
        ctx = span.get_span_context()

        if ctx and ctx.is_valid:
            record.trace_id = format(ctx.trace_id, "032x")
            record.span_id = format(ctx.span_id, "016x")
            record.trace_flags = str(ctx.trace_flags)
        else:
            record.trace_id = "0" * 32
            record.span_id = "0" * 16
            record.trace_flags = "00"

        return super().format(record)


def configure_structured_logging():
    handler = logging.StreamHandler()
    handler.setFormatter(
        OtelCorrelatedFormatter(
            fmt='{"timestamp": "%(asctime)s", "level": "%(levelname)s", '
                '"logger": "%(name)s", "message": "%(message)s", '
                '"trace_id": "%(trace_id)s", "span_id": "%(span_id)s"}'
        )
    )
    root = logging.getLogger()
    root.addHandler(handler)
    root.setLevel(logging.INFO)

With this in place, every log line emitted during an active span will carry the trace ID. In Grafana, you can configure Loki as a log data source and Tempo as a trace data source, then set up a "derived field" that turns trace_id values in log lines into clickable links that open the corresponding trace in Tempo. The result: from a log line showing "Required field null in records", one click takes you to the full 45-second trace waterfall.


Pipeline-Level Span Events and Checkpoints

Beyond start/end timing, spans support events — timestamped annotations you can attach to a span without creating a child span. These are perfect for pipeline milestones that aren't worth a full span but are worth recording:

def load_to_snowflake(df: pd.DataFrame, table: str, tracer, meter):
    rows_loaded = meter.create_counter(
        "pipeline.rows.loaded",
        unit="rows"
    )
    load_errors = meter.create_counter(
        "pipeline.load.errors",
        unit="errors"
    )

    with tracer.start_as_current_span("load_to_snowflake") as span:
        span.set_attribute("target.table", table)
        span.set_attribute("load.row_count", len(df))

        # Stage to S3 first
        s3_path = f"s3://pipeline-staging/{table}/{int(time.time())}.parquet"
        df.to_parquet(s3_path)

        # Record a span event at the staging checkpoint
        span.add_event(
            "staged_to_s3",
            attributes={
                "s3_path": s3_path,
                "file_size_bytes": df.memory_usage(deep=True).sum(),
                "row_count": len(df),
            }
        )

        # COPY INTO Snowflake
        try:
            conn = get_snowflake_connection()
            cursor = conn.cursor()
            cursor.execute(
                f"COPY INTO {table} FROM '{s3_path}' FILE_FORMAT = (TYPE = PARQUET)"
            )
            result = cursor.fetchone()
            rows_copied = result[0] if result else 0

            span.add_event(
                "copy_into_complete",
                attributes={"rows_copied": rows_copied, "table": table}
            )
            rows_loaded.add(rows_copied, {"table": table})
            span.set_attribute("load.rows_actually_copied", rows_copied)

            # Detect silent data loss: staged more than landed
            if rows_copied < len(df):
                discrepancy = len(df) - rows_copied
                span.add_event(
                    "load_discrepancy_detected",
                    attributes={
                        "staged": len(df),
                        "loaded": rows_copied,
                        "lost": discrepancy,
                    }
                )
                logger.error(
                    "Load discrepancy: rows staged vs rows copied differ",
                    extra={"staged": len(df), "loaded": rows_copied}
                )

        except Exception as exc:
            load_errors.add(1, {"table": table, "error_type": type(exc).__name__})
            span.record_exception(exc)
            raise

Span events are searchable in Jaeger and Grafana Tempo. That load_discrepancy_detected event is exactly the kind of silent failure that would take hours to find in raw logs — with span events, you can query "show me all traces that contain the event load_discrepancy_detected" and get every affected pipeline run, with full context.

This complements the broader strategies for pipeline error handling and recovery — OTel gives you the visibility to know when to trigger those recovery paths.


Production Deployment: Collector, Sampling, and Backends

In production, you don't export spans directly from your pipeline process to Jaeger or Grafana. You export to an OpenTelemetry Collector — a standalone process that acts as a telemetry proxy. The collector buffers, batches, filters, and routes signals to multiple backends simultaneously.

A typical production topology:

Pipeline Process
  └─ OTLP gRPC → OTel Collector
                    ├─ Traces → Grafana Tempo (or Jaeger)
                    ├─ Metrics → Prometheus remote write
                    └─ Logs → Loki

A minimal collector config (otel-collector-config.yaml):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s
  # Drop high-cardinality attributes before they hit your backend
  attributes/sanitize:
    actions:
      - key: "record.id"
        action: delete
      - key: "batch.uuid"
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: false
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, attributes/sanitize, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

Sampling Strategies

For high-throughput pipelines, exporting every span can generate gigabytes of trace data per day. You need a sampling strategy. OTel supports two approaches:

Head-based sampling (decided at span creation): Fast and simple, but you don't yet know if this trace is interesting. Use a TraceIdRatioBased sampler for a consistent percentage, or ParentBased to inherit the parent's sampling decision:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased

# Sample 10% of traces in production
sampler = ParentBased(root=TraceIdRatioBased(0.1))
tracer_provider = TracerProvider(resource=resource, sampler=sampler)

Tail-based sampling (decided at the collector after the trace completes): More expensive but far more valuable — you can sample 100% of error traces and 1% of success traces. The OTel Collector's tailsampling processor supports this:

processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-pipelines-policy
        type: latency
        latency: {threshold_ms: 60000}
      - name: low-rate-baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Tip

For pipeline observability, tail-based sampling is almost always the right choice. You want 100% of failed runs and slow runs. Healthy, fast runs are less interesting — a 5% sample is plenty for baseline performance tracking.


Hands-On Exercise

Build an end-to-end instrumented mini-pipeline that processes public GitHub events data. Here's the structure:

Setup: Run Jaeger locally (docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest)

Task 1: Instrumented ingestion Write a function ingest_github_events(org: str, repo: str) that:

  • Creates a parent span ingest_github_events with org and repo as attributes
  • Paginates through https://api.github.com/repos/{org}/{repo}/events (up to 3 pages)
  • Creates a child span per page with HTTP status code and record count
  • Emits a github.events.fetched counter metric with org and repo labels

Task 2: Instrumented transformation Write a function transform_events(raw_events: list) that:

  • Creates a parent span transform_events
  • In a child span extract_actor_metrics, groups events by actor login and counts event types
  • In a child span filter_bot_actors, removes actors whose logins end in [bot]
  • Emits a transform.bot_records_dropped counter when bots are filtered
  • Attaches a span event transformation_summary with input count, output count, and bot count

Task 3: Correlated logging Ensure all log lines emitted during the pipeline carry trace_id and span_id. Verify by running the pipeline and checking that the log output contains valid hex trace IDs.

Verification: Open Jaeger at http://localhost:16686, search for your service name, and find the full trace. Confirm you can see the page-level child spans and the transformation spans as siblings under a common root.


Common Mistakes & Troubleshooting

Mistake 1: Creating instruments inside functions

# WRONG — creates a new counter on every function call
def process_batch(records):
    counter = meter.create_counter("records_processed")  # recreated each call!
    counter.add(len(records))
# RIGHT — create instruments once, at class or module initialization
class BatchProcessor:
    def __init__(self, meter):
        self.records_processed = meter.create_counter("records_processed")

    def process_batch(self, records):
        self.records_processed.add(len(records))

Creating the same instrument multiple times with the same name is technically idempotent in OTel (it returns the same underlying instrument), but it wastes allocations and obscures intent.

Mistake 2: High-cardinality label values

# WRONG — user_id has millions of values, creates millions of time series
meter.create_counter("records_processed").add(1, {"user_id": user_id})

# RIGHT — use bounded categorical labels
meter.create_counter("records_processed").add(1, {"pipeline": "salesforce-opportunities"})

This will silently destroy your Prometheus instance. See the backpressure and bottleneck diagnosis article for how unbounded cardinality can manifest as backend degradation rather than pipeline-side errors.

Mistake 3: Losing trace context across thread or process boundaries

When you submit work to a thread pool or subprocess, the active span context does not automatically propagate:

# WRONG — the thread has no span context
with tracer.start_as_current_span("process_partition") as span:
    with concurrent.futures.ThreadPoolExecutor() as executor:
        futures = [executor.submit(process_chunk, chunk) for chunk in chunks]
        # process_chunk() has no active span, its logs won't be correlated
# RIGHT — capture and pass the context explicitly
from opentelemetry import context as otel_context

with tracer.start_as_current_span("process_partition") as span:
    ctx = otel_context.get_current()

    def process_chunk_with_context(chunk):
        token = otel_context.attach(ctx)
        try:
            return process_chunk(chunk)
        finally:
            otel_context.detach(token)

    with concurrent.futures.ThreadPoolExecutor() as executor:
        futures = [executor.submit(process_chunk_with_context, chunk) for chunk in chunks]

Mistake 4: Spans that are too coarse or too fine

A span that wraps an entire 45-minute pipeline run is nearly useless — you can't tell where time was spent. A span for every individual record in a 10-million-row batch will generate so much trace data it crashes your collector. The right granularity is logical stages (fetch, transform, load) and iteration boundaries (per-page, per-partition), not individual records.

A common rule of thumb: if a span would exist for less than 1 millisecond or more than 10 minutes, reconsider whether it should be a span, a metric, or a span event instead.

Mistake 5: Not flushing on shutdown

The BatchSpanProcessor exports asynchronously. If your pipeline process exits before the flush completes, you lose the last batch of spans:

# In your pipeline entry point
import signal
import atexit

def shutdown_otel():
    trace.get_tracer_provider().shutdown()
    metrics.get_meter_provider().shutdown()

atexit.register(shutdown_otel)

This matters especially in containerized environments where SIGTERM can arrive at any time. The shutdown() call flushes all pending spans/metrics before the provider is destroyed.

Warning

If you're running pipelines in AWS Lambda or Google Cloud Run (short-lived containers), the BatchSpanProcessor with its default async export may not finish before the container freezes. Use SimpleSpanProcessor for serverless environments, or configure the exporter with a very short export_timeout_millis.


Summary & Next Steps

You now have a complete implementation pattern for OpenTelemetry-based observability across all three layers of a data pipeline. Let's consolidate what we've built:

Traces give you causal, time-ordered chains of operations — from the moment an orchestrator kicks off a DAG run to when rows land in the warehouse. Span attributes make each stage self-documenting. Span events capture milestones and anomalies (like silent data loss) without creating span overhead for every record.

Metrics give you aggregate health signals over time — records ingested, transformation drop rates, load latency distributions. The critical discipline is keeping label cardinality bounded: use categorical labels (pipeline name, stage, error type), never entity IDs.

Logs become dramatically more useful when they carry trace_id and span_id. With a correlated formatter, every log line is pivotable to its full trace context. This turns "search through 10,000 log lines" into "click to open the trace that contains this log."

The OpenTelemetry Collector is your production-grade buffer and router — it decouples your pipeline code from your observability backend, enables tail-based sampling, and lets you evolve your backend choices without changing application code.

Where to go from here:

  • Apply these patterns to streaming pipelines — the batch vs. stream processing article will help you understand where streaming observability differs (consumer lag becomes a critical metric, and span lifetimes need careful design for long-running stream processors)
  • Connect observability to SLA management — building and managing data pipeline SLAs shows you how to turn latency histograms into actionable freshness guarantees
  • Explore logging, alerting, and observability fundamentals for data pipelines for the alerting side of this story — once you have traces and metrics, you need alert rules that fire on the right signals without paging you for noise

The 2:30 AM page doesn't go away. But with OpenTelemetry instrumented across your pipeline, you'll spend the next five minutes understanding exactly what happened — not the next two hours guessing.

Work With Us

From insight to implementation

Reading is the start. When you're ready to build the data, automation, or AI systems behind it, our team turns strategy into shipped results.

Let's Build

Data Pipeline Fundamentals

Previous

Implementing Cross-Source Data Harmonization in Transformation Pipelines

Related Insights

Data EngineeringPractitioner

Implementing Airflow Dataset-Aware Scheduling and Data-Driven DAG Triggers for Dependency Management Across Multi-Team Pipelines

23 min
Data EngineeringPractitioner

Implementing Cross-Source Data Harmonization in Transformation Pipelines

22 min
Data EngineeringFoundation

Snowflake Query Performance Tuning: Reading Query Profiles, Identifying Spill, and Optimizing Clustering Keys for Analytics Workloads

20 min

On this page

  • Introduction
  • Prerequisites
  • Why OpenTelemetry? The Case for a Unified Standard
  • Understanding the Three Pillars in a Pipeline Context
  • Traces and Spans
  • Metrics
  • Logs
  • Setting Up the OpenTelemetry SDK
  • Instrumenting the Ingestion Layer
  • Instrumenting the Transformation Layer
  • Instrumenting the Orchestration Layer
  • Correlating Logs with Traces
  • Pipeline-Level Span Events and Checkpoints
  • Production Deployment: Collector, Sampling, and Backends
  • Sampling Strategies
  • Hands-On Exercise
  • Common Mistakes & Troubleshooting
  • Mistake 1: Creating instruments inside functions
  • Mistake 2: High-cardinality label values
  • Mistake 3: Losing trace context across thread or process boundaries
  • Mistake 4: Spans that are too coarse or too fine
  • Mistake 5: Not flushing on shutdown
  • Summary & Next Steps