Migrating a Monolithic dbt Project to a Multi-Project Architecture with dbt Mesh: Cross-Project References, Access Controls, and Dependency Management in Production

Introduction

Your dbt project started as a reasonable thing. A handful of models, a few sources, some tests. Then the company grew, the data team doubled, and somewhere along the way your single dbt_project.yml became a 400-model monolith that takes 45 minutes to fully rebuild and requires three people to understand how the finance models connect to the marketing attribution pipeline. Pull requests sit in review limbo because nobody wants to approve changes to fct_revenue.sql without first understanding how it affects the dozen downstream models that nobody outside the analytics engineering team can name.

This is the moment when dbt Mesh stops being an interesting conference talk and starts being a serious architectural consideration. dbt Mesh — released as part of dbt Cloud's multi-project capabilities — lets you decompose that monolith into independently deployable, governed projects that can still share data contracts across team boundaries. It's not just a "split your models into folders" story. It introduces a genuine interface layer between teams: explicit public contracts, version-controlled access controls, and cross-project references that fail loudly when upstream teams break things.

By the end of this lesson, you'll understand how to plan and execute a migration from a monolithic dbt project to a multi-project Mesh architecture. We'll cover the full journey — from deciding where to cut the project apart, to implementing public models with contracts, to wiring up cross-project ref() calls, to managing the production deployment dependencies that most tutorials skip entirely.

What you'll learn:

• How to identify the right boundaries for splitting a monolithic project using dependency analysis and team ownership patterns
• How to configure access, group, and contract in your dbt models to create explicit, enforced interfaces
• How to implement cross-project ref() calls and understand what happens at compile time and runtime
• How to manage production deployment ordering and avoid circular dependency nightmares
• Advanced patterns for handling breaking changes, version management, and rollback strategies in a Mesh environment

---

Prerequisites

This lesson assumes you're comfortable with:

• dbt Core concepts: models, sources, tests, ref(), schema.yml, dbt_project.yml
• dbt Cloud multi-environment setup (dev, staging, prod environments)
• Basic understanding of your data warehouse's cross-database or cross-schema query capabilities (Snowflake, BigQuery, Databricks, or Redshift)
• Git branching workflows and CI/CD basics
• At least one production dbt project under your belt — ideally a large one that's starting to cause you pain

If you haven't used dbt Core beyond the fundamentals, revisit the dbt Fundamentals learning path first.

---

Understanding What dbt Mesh Actually Is (and What It Isn't)

Before we touch a single config file, let's establish a precise mental model. A common misconception is that dbt Mesh is primarily a code organization feature — like a better version of splitting models into subdirectories. It isn't. Mesh is a dependency and governance system for teams, not just for models.

The core primitive is the public model with a contract. In a standard dbt project, any model can ref() any other model freely. There's no enforced boundary. In Mesh, you explicitly declare which models are part of your project's public interface, enforce their schemas with contracts, and allow other projects to reference them via cross-project ref(). Everything else is private by default.

This matters because the real pain in a monolithic dbt project isn't usually the compile time (though that's annoying). It's the invisible coupling. When a junior analyst modifies a staging model to fix a column name, they don't know that three downstream models in the finance subdirectory depend on that exact column name. Mesh makes those dependencies explicit and enforced.

Here's the hierarchy of access controls dbt Mesh introduces:

# In model YAML
models:
  - name: fct_orders
    access: public          # public | protected | private
    group: finance          # which team "owns" this model
    config:
      contract:
        enforced: true      # schema contract is mandatory for public models

• private: Only models in the same group can reference this model. Default for most of your models.
• protected: Models in the same project can reference this. The default if you don't specify anything.
• public: Any project in your Mesh can reference this model via cross-project ref().

The distinction between protected and private trips people up. protected is project-scoped (the whole project can use it, even across groups). private is group-scoped (only models with the same group declaration). Plan your group structure carefully — it drives your internal access patterns just as much as the cross-project ones.

---

Phase 1: Mapping Your Monolith Before You Cut It

Migration mistakes almost always happen because teams jump to splitting the project before they understand the dependency graph. Spend real time here. It will save you from creating a Mesh architecture with a tangled web of cross-project dependencies that's harder to manage than the original monolith.

Step 1: Generate and Analyze the DAG

Run this in your existing project to get a machine-readable picture of your dependency graph:

dbt ls --select +fct_revenue+ --output json > dag_analysis.json

For a more thorough analysis, use dbt's manifest.json after compilation:

dbt compile

The target/manifest.json contains the full dependency graph. You can parse it programmatically:

import json
from collections import defaultdict

with open("target/manifest.json") as f:
    manifest = json.load(f)

# Build a reverse dependency map
dependents = defaultdict(list)
for node_id, node in manifest["nodes"].items():
    for dep in node.get("depends_on", {}).get("nodes", []):
        dependents[dep].append(node_id)

# Find the most-referenced models (your public interface candidates)
reference_counts = {
    node_id: len(deps) 
    for node_id, deps in dependents.items()
}

sorted_by_refs = sorted(
    reference_counts.items(), 
    key=lambda x: x[1], 
    reverse=True
)

for node_id, count in sorted_by_refs[:20]:
    model_name = node_id.split(".")[-1]
    print(f"{model_name}: referenced by {count} downstream models")

This script surfaces your natural public interface candidates — the models that many other things depend on. These are your future public models.

Step 2: Identify Team Ownership Clusters

The best project boundaries follow team ownership, not data domain alone. A pure domain decomposition (marketing data in one project, finance data in another) sounds clean but often doesn't match how teams actually work. Instead, look for:

1. Who owns the models in production? If the data engineering team owns staging and core models, and separate domain teams own their marts, that's a natural cut.
2. Which models cross team boundaries most frequently? These are your future public interfaces.
3. Where do PRs cause the most cross-team friction? Where people are nervous to merge is where boundaries should be enforced.

A typical breakdown we see work well in practice:

platform_project (owned by data engineering)
├── staging models (all sources → clean, typed staging)
├── core dimension models (dim_customers, dim_products, dim_dates)
└── core fact models (fct_orders, fct_sessions)

finance_project (owned by finance analytics team)  
├── finance-specific marts
├── revenue models
└── cost attribution models

marketing_project (owned by growth/marketing team)
├── campaign performance models
├── attribution models  
└── channel reporting

product_project (owned by product analytics team)
├── funnel models
├── retention cohorts
└── feature adoption models

The platform project becomes the foundation — it publishes well-tested, contracted public models that downstream projects consume. Downstream projects are largely consumers, though they may expose some public models to each other (carefully).

Step 3: Draw the Dependency Lines

Before you write any config, explicitly map which models need to cross project boundaries. For each cross-project dependency you identify, ask:

• Does the consuming project need this at the raw data level, or can it consume a pre-aggregated mart?
• Is this dependency temporary (can we denormalize or pre-compute to eliminate it)?
• Does this create a cycle? (finance → platform AND platform → finance = instant problem)

Cross-project circular dependencies are impossible in Mesh by design — but you can create logical cycles through shared intermediate concepts. Identify these now.

---

Phase 2: Setting Up the Multi-Project Structure

With your boundaries mapped, let's build the actual project structure. We'll use a concrete example: migrating a monolith with a platform engineering team's core models and a finance team's revenue analytics.

Creating the Producer Project

The platform project produces public models that others consume. Here's how its dbt_project.yml looks:

# platform/dbt_project.yml
name: 'platform'
version: '1.0.0'

profile: 'platform'

model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]

target-path: "target"
clean-targets: ["target", "dbt_packages"]

models:
  platform:
    staging:
      +materialized: view
      +access: private          # staging models are internal only
      +group: platform_eng
    core:
      +materialized: table
      +access: protected        # core models accessible within project
      +group: platform_eng
    public_interfaces:
      +materialized: table
      +access: public           # explicitly public models
      +group: platform_eng
      +contract:
        +enforced: true

Notice that we're using folder structure to set default access levels. This is intentional — it makes access levels obvious at a glance and reduces the chance of accidentally exposing private models.

Defining Groups and Public Models

Groups are defined in a groups.yml file (convention — it can live anywhere in your model paths):

# platform/models/groups.yml
groups:
  - name: platform_eng
    owner:
      name: Platform Engineering
      email: platform-data@company.com

Now define your public models with contracts. This is where the real work happens:

# platform/models/public_interfaces/schema.yml
models:
  - name: dim_customers
    description: >
      Canonical customer dimension. Single source of truth for customer 
      attributes across all downstream projects. Do not bypass this model 
      by joining directly to raw CRM tables.
    access: public
    group: platform_eng
    config:
      contract:
        enforced: true
    columns:
      - name: customer_id
        description: Surrogate key for customer dimension
        data_type: varchar
        constraints:
          - type: not_null
          - type: primary_key
      - name: customer_key
        description: Natural key from source CRM system
        data_type: varchar
        constraints:
          - type: not_null
      - name: full_name
        data_type: varchar
      - name: email_address
        data_type: varchar
      - name: customer_segment
        data_type: varchar
      - name: acquisition_channel
        data_type: varchar
      - name: first_order_date
        data_type: date
      - name: is_active
        data_type: boolean
      - name: created_at
        data_type: timestamp_tz
      - name: updated_at
        data_type: timestamp_tz

The contract here is doing real work. When dbt compiles dim_customers, it will verify that the model's actual SQL output matches these column definitions — names, types, and constraints. If you add a new transformation that accidentally drops acquisition_channel, the contract enforcement catches it before it reaches production.

The SQL Model Behind the Contract

Your public model SQL should be defensive and explicit:

-- platform/models/public_interfaces/dim_customers.sql
{{
    config(
        materialized='table',
        on_schema_change='fail'
    )
}}

with source as (
    select * from {{ ref('stg_crm__customers') }}
),

enriched as (
    select
        {{ dbt_utils.generate_surrogate_key(['customer_key']) }} as customer_id,
        customer_key,
        coalesce(first_name || ' ' || last_name, email) as full_name,
        lower(trim(email)) as email_address,
        case 
            when customer_tier in ('enterprise', 'growth') then 'commercial'
            when customer_tier = 'starter' then 'self-serve'
            else 'unknown'
        end as customer_segment,
        acquisition_channel,
        first_order_date,
        is_active,
        created_at,
        updated_at
    from source
)

select * from enriched

Note on_schema_change='fail'. Combined with the contract, this makes your public model maximally conservative — it will refuse to run if someone changes the upstream staging model in a way that alters the output schema. This is the right behavior for a public interface.

---

Phase 3: Configuring Cross-Project References

Now we set up the consuming projects. This is where the dependencies.yml file comes in — a file specific to multi-project Mesh setups that's separate from packages.yml.

The Producer Side: dbt_project.yml Exports

The platform project doesn't need any special configuration to be consumable — the access: public on models is sufficient. But you should configure your project's metadata to help consumers find it:

# platform/dbt_project.yml (additions)
name: 'platform'
version: '1.0.0'

# This is how consumers reference your project
dispatch:
  - macro_namespace: platform
    search_order: ['platform', 'dbt']

The Consumer Side: dependencies.yml

In your consuming project (finance, marketing, etc.), create a dependencies.yml file at the project root. This is not the same as packages.yml — packages are Python-package-style installs, while dependencies are project-to-project Mesh relationships:

# finance/dependencies.yml
projects:
  - name: platform

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.1

Using Cross-Project ref()

Now in your finance project models, you can reference platform models:

-- finance/models/marts/fct_revenue.sql
{{
    config(
        materialized='table'
    )
}}

with customers as (
    -- Cross-project ref: 'platform' is the project name, 'dim_customers' is the model
    select * from {{ ref('platform', 'dim_customers') }}
),

orders as (
    select * from {{ ref('platform', 'fct_orders') }}
),

payments as (
    select * from {{ ref('stg_stripe__payments') }}
),

revenue_by_customer as (
    select
        c.customer_id,
        c.customer_segment,
        c.acquisition_channel,
        o.order_id,
        o.order_date,
        p.amount_usd,
        p.payment_method,
        p.payment_status
    from orders o
    inner join customers c 
        on o.customer_id = c.customer_id
    inner join payments p 
        on o.order_id = p.order_id
    where p.payment_status = 'succeeded'
)

select * from revenue_by_customer

The two-argument ref('platform', 'dim_customers') is the cross-project reference syntax. At compile time, dbt resolves this to the actual table location in your warehouse — the platform project's production schema where dim_customers lives.

What Happens at Compile Time

This is where many people get confused. Cross-project ref() works differently from same-project ref(). In a normal project, ref('dim_customers') resolves to a table in whatever target schema dbt is running against. Cross-project ref('platform', 'dim_customers') resolves to the platform project's production table — always, regardless of what environment the finance project is running in.

This means:

Finance project DEV run → fct_revenue reads from platform PRODUCTION dim_customers
Finance project STAGING run → fct_revenue reads from platform PRODUCTION dim_customers  
Finance project PRODUCTION run → fct_revenue reads from platform PRODUCTION dim_customers

This is intentional and actually valuable — your development and staging environments in downstream projects are always working against real, production-quality upstream data. But it does mean you need to be thoughtful about the platform project's production reliability. When platform breaks, every downstream project breaks.

Advanced consideration: dbt Cloud does support configuring cross-project references to point to non-production environments, but this requires explicit configuration and is typically only used for testing framework changes. The default behavior (always point to production) is correct for most use cases.

---

Phase 4: The Migration Execution Strategy

You have the architecture mapped. Now let's talk about executing the actual migration without taking down production.

The Strangler Fig Pattern for dbt Projects

Don't try to do this as a big bang migration. Instead, use the strangler fig pattern: gradually grow the new Mesh structure alongside the existing monolith, shift traffic to it, and then remove the old code.

Phase 4a: Create the Platform Project as a Thin Wrapper

Start by creating your platform project with models that are initially just select * wrappers around the existing monolith's materialized tables:

-- platform/models/public_interfaces/dim_customers.sql
-- TEMPORARY: wrapping existing monolith table during migration
-- TODO: replace with full model once staging models migrated

{{
    config(
        materialized='view'   -- view during migration for zero-copy overhead
    )
}}

select
    customer_id,
    customer_key,
    full_name,
    email_address,
    customer_segment,
    acquisition_channel,
    first_order_date,
    is_active,
    created_at,
    updated_at
from {{ source('monolith_prod', 'dim_customers') }}

This creates a valid, contractable public interface that consuming teams can start building against, while the monolith continues to run normally. You haven't broken anything. You're just adding a new layer.

Phase 4b: Set Up Consuming Projects with Cross-Project Refs

While the platform project is still wrapping the monolith, have consuming teams start writing their models against the platform project's public interfaces rather than the monolith directly. New models use cross-project ref(). Old models in those teams' projects stay as-is for now.

This lets you validate that cross-project ref() works in your environment, the CI/CD pipelines are configured correctly, and consuming teams understand the new access patterns — all without affecting production data.

Phase 4c: Migrate the Platform Project Internals

Now move the actual staging and core model logic into the platform project. The public interface contracts stay identical (that's the point of contracts — they protect consumers from internal changes). Internal staging models migrate without downstream teams noticing:

# From monolith, cut models over to platform project
# Week 1: staging models
git mv monolith/models/staging/crm/ platform/models/staging/crm/
git mv monolith/models/staging/stripe/ platform/models/staging/stripe/

# Week 2: core dimension models  
git mv monolith/models/core/dimensions/ platform/models/core/dimensions/

# Week 3: core fact models
git mv monolith/models/core/facts/ platform/models/core/facts/

Update the platform project's public interface models to use internal ref() instead of the monolith source:

-- platform/models/public_interfaces/dim_customers.sql
-- MIGRATION COMPLETE: now referencing internal staging model

{{
    config(
        materialized='table',
        on_schema_change='fail'
    )
}}

with source as (
    select * from {{ ref('stg_crm__customers') }}  -- internal ref now
),
...

Phase 4d: Clean Up the Monolith

Once consuming teams have fully migrated their ref() calls and production has run successfully with the new architecture for at least one full data cycle, remove the corresponding models from the monolith. Update CI checks to ensure no new cross-team dependencies get added to the monolith.

---

Phase 5: Production Deployment Orchestration

This is the part that makes senior engineers nervous and junior engineers not nervous enough. Multi-project dbt Mesh changes how you think about deployment ordering in production.

The Fundamental Problem

In a monolith, your orchestrator (Airflow, Prefect, dbt Cloud's own scheduler, or Dagster) runs one dbt job and everything is in topological order within that job. In Mesh, you have multiple independent projects. The platform project must finish before the finance project can start — but how do you enforce this?

Option 1: dbt Cloud's Built-In Job Chaining

If you're on dbt Cloud, the cleanest solution is job chaining with triggers:

In your dbt Cloud interface, navigate to your finance project's job configuration. Under "Triggers," you can configure the job to run "After another job finishes." Set it to trigger after the platform project's production job completes successfully.

This creates an explicit dependency: finance production → waits for → platform production. If platform fails, finance doesn't run. This is the correct behavior.

For multiple downstream projects depending on platform:

Platform Production Job
    ↓ (triggers on success)
├── Finance Production Job  
├── Marketing Production Job
└── Product Production Job

Option 2: Orchestrator-Level Dependencies (Airflow/Prefect/Dagster)

If you're running dbt Core or prefer external orchestration:

# Airflow DAG for Mesh orchestration
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
from datetime import timedelta

default_args = {
    'owner': 'data-engineering',
    'depends_on_past': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'dbt_mesh_production',
    default_args=default_args,
    description='Multi-project dbt Mesh production run',
    schedule_interval='0 6 * * *',  # 6 AM daily
    start_date=days_ago(1),
    catchup=False,
) as dag:
    
    platform_run = BashOperator(
        task_id='run_platform_project',
        bash_command="""
            cd /opt/dbt/platform && 
            dbt run --target prod --profiles-dir /opt/dbt/profiles &&
            dbt test --target prod --profiles-dir /opt/dbt/profiles
        """,
    )
    
    finance_run = BashOperator(
        task_id='run_finance_project',
        bash_command="""
            cd /opt/dbt/finance && 
            dbt run --target prod --profiles-dir /opt/dbt/profiles
        """,
    )
    
    marketing_run = BashOperator(
        task_id='run_marketing_project',
        bash_command="""
            cd /opt/dbt/marketing && 
            dbt run --target prod --profiles-dir /opt/dbt/profiles
        """,
    )
    
    product_run = BashOperator(
        task_id='run_product_project', 
        bash_command="""
            cd /opt/dbt/product && 
            dbt run --target prod --profiles-dir /opt/dbt/profiles
        """,
    )
    
    # Dependencies: platform must complete before any downstream project
    platform_run >> [finance_run, marketing_run, product_run]

Partial Failure Handling

Define explicit failure modes for each project's production job. A common pattern is to use dbt's --defer flag in downstream projects:

# If finance project fails, you can defer to previous run artifacts
dbt run --defer --state /path/to/last-successful-state --select state:modified+

This runs only modified models and their dependents, using the last successful run's artifacts for everything else. When combined with Mesh's cross-project refs pointing to production, this gives you an efficient partial-failure recovery path.

---

Phase 6: Contract Versioning and Breaking Changes

The hardest ongoing challenge in dbt Mesh isn't the initial setup — it's managing breaking changes to public models over time. The platform team needs to evolve dim_customers, but they can't break the finance, marketing, and product teams in the process.

dbt's Model Versioning Feature

dbt Core 1.5+ supports explicit model versioning, designed precisely for this scenario:

# platform/models/public_interfaces/schema.yml
models:
  - name: dim_customers
    latest_version: 2
    access: public
    group: platform_eng
    
    versions:
      - v: 1
        defined_in: dim_customers_v1   # SQL file: dim_customers_v1.sql
        config:
          contract:
            enforced: true
        deprecation_date: 2024-06-01   # When this version stops being supported
        columns:
          - name: customer_id
            data_type: varchar
          # ... original schema

      - v: 2
        defined_in: dim_customers      # SQL file: dim_customers.sql
        config:
          contract:
            enforced: true
        columns:
          - name: customer_id
            data_type: varchar
          - name: customer_uuid        # new column in v2
            data_type: varchar
          # ... expanded schema

Consuming projects can explicitly pin to a version:

-- finance project: still using v1 while migrating
select * from {{ ref('platform', 'dim_customers', v=1) }}

-- product project: already migrated to v2
select * from {{ ref('platform', 'dim_customers', v=2) }}

This is the correct pattern for additive changes (new columns, new metrics). For destructive changes (removing or renaming columns), you need a full version bump with a deprecation period.

Breaking Change Protocol

Establish a written protocol for breaking changes. Here's one that works in practice:

1. Platform team announces intent: File a GitHub issue or Slack message to all consuming teams at least two weeks before the planned change. Include the old and new schemas.

2. Platform team deploys v2: Both v1 and v2 are live simultaneously. v1 has a deprecation_date set. Consuming teams can begin migrating at their own pace.

3. Two-week migration window: Consuming teams update their cross-project refs to use v2. The platform team is available to answer questions but doesn't block on consuming team timelines.

4. Deprecation: At the deprecation_date, the platform team removes v1. If consuming teams haven't migrated, their production jobs fail — but this is by design. The warning was given.

Anti-pattern: Never silently change the schema of a public model in place. Even "obviously safe" changes like renaming a column to be clearer will break downstream contracts immediately. The formal protocol above feels bureaucratic until the first time a silent change causes a finance team's monthly close dashboard to return null revenue figures.

Detecting Downstream Impact Before Merging

Before any platform team PR merges that touches public interfaces, run a cross-project impact analysis:

# In platform project, check what would break
dbt ls --select dim_customers --output json | jq '.[]'

# Then manually verify with consuming project manifests
# Check finance project's compiled manifest for references to dim_customers
grep -r "platform.*dim_customers" finance/target/manifest.json

In dbt Cloud, the CI job for the platform project should include a step that validates contracts still hold. Configure it to fail the PR if the contract for any public model changes without a version bump.

---

Hands-On Exercise

You'll execute a scaled-down but realistic migration that hits all the major decision points.

Setup

Clone or create a dbt project with at least 15-20 models representing a small e-commerce data stack. Your structure should look roughly like:

monolith/
├── models/
│   ├── staging/
│   │   ├── stg_orders.sql
│   │   ├── stg_customers.sql
│   │   ├── stg_products.sql
│   │   └── stg_payments.sql
│   ├── core/
│   │   ├── dim_customers.sql
│   │   ├── dim_products.sql
│   │   └── fct_orders.sql
│   └── marts/
│       ├── finance/
│       │   ├── fct_revenue.sql
│       │   └── rpt_monthly_revenue.sql
│       └── marketing/
│           ├── fct_campaign_performance.sql
│           └── rpt_channel_attribution.sql

Exercise Steps

Step 1: Dependency Mapping (20 minutes)

Run the Python manifest analysis script from Phase 1 of this lesson. Produce a list of:

• Which models are referenced by models outside their own directory
• The top 5 most-referenced models in the project

Step 2: Create the Platform Project (30 minutes)

Create a new directory platform/ at the same level as monolith/. Set up:

• dbt_project.yml with appropriate folder-level access configurations
• groups.yml with a platform_eng group
• schema.yml for dim_customers with a full contract (all columns typed)
• The dim_customers.sql model behind the contract

Validate the contract locally: dbt compile --select dim_customers should succeed. Intentionally break it (remove a column from the SQL) and verify dbt run --select dim_customers fails with a contract violation error.

Step 3: Create the Finance Consumer Project (30 minutes)

Create a finance/ directory. Set up:

• dbt_project.yml
• dependencies.yml referencing the platform project
• Migrate fct_revenue.sql to use {{ ref('platform', 'dim_customers') }} and {{ ref('platform', 'fct_orders') }}

Attempt to compile the finance project. You'll hit the first real challenge: dbt needs the platform project's artifacts to resolve cross-project refs. In a local setup without dbt Cloud, you'll need to generate and point to the platform project's manifest.json manually using the --state flag or a local mock.

Step 4: Simulate a Breaking Change (20 minutes)

In the platform project, "accidentally" rename email_address to email in dim_customers.sql. Run dbt run --select dim_customers. Observe the contract enforcement failure.

Now do it correctly: bump to v2, set a deprecation date on v1, update the schema.yml with both versions, and update the finance project to explicitly reference v=1. Verify finance compiles successfully, then migrate it to v=2.

Reflection Questions:

• How would you automate the detection of cross-project dependency changes in your CI pipeline?
• What's the right deprecation_date policy for your team's velocity and planning cycles?
• At what number of cross-project dependencies does the versioning overhead become unsustainable, and what architectural changes would reduce that number?

---

Common Mistakes & Troubleshooting

Mistake 1: Making Too Many Models Public

The temptation is to make public anything that might be useful to another team. Resist this. Every public model is a contract you have to maintain. Start with only the models you know consuming teams already need. Add more public models when the request comes — not speculatively.

A good rule of thumb: if you can't write a one-sentence description of why this model should be a public interface, it probably shouldn't be one yet.

Mistake 2: Forgetting That Cross-Project ref() Always Hits Production

We covered this in the lesson, but it catches teams in the wild constantly. During development of a finance model, you're reading from the platform project's production dim_customers. This means:

• If platform runs at 6 AM and finance runs at 7 AM, you're always reading "yesterday's" platform data in finance production. Budget your schedule windows accordingly.
• Your finance dev environment always queries production platform data. Understand the cost implications for your warehouse.

Mistake 3: Circular Logical Dependencies

You can't have true circular project dependencies in Mesh (dbt will prevent it). But you can create a situation where:

• Platform publishes dim_customers which includes customer_segment
• Marketing uses dim_customers to build campaign attribution
• Platform wants to use marketing attribution to enrich customer_segment

This isn't a technical circular dependency (platform doesn't ref() marketing), but it's a logical one that leads to data that's always one cycle stale. Recognize this pattern early and either accept the staleness, create a separate enrichment layer, or restructure which project owns the enriched concept.

Mistake 4: Not Testing Cross-Project Refs in CI

Many teams get Mesh working in production but forget to set up CI validation in consuming projects. Your finance project's CI job must compile and run tests, which means it needs access to the platform project's production artifacts. Set this up explicitly in your CI configuration.

In dbt Cloud, this is handled by the project dependency configuration. In self-hosted setups, your CI pipeline needs to fetch the platform project's manifest.json and pass it as --state:

# CI step in finance project
aws s3 cp s3://your-dbt-artifacts/platform/manifest.json ./platform_state/manifest.json
dbt compile --state ./platform_state

Mistake 5: Ignoring the `on_schema_change` Config

Every public model should have on_schema_change: 'fail' or on_schema_change: 'sync_all_columns'. Without this, an upstream change can silently change the schema of your materialized table while dbt happily runs without error. This is particularly dangerous for incremental models that consumers depend on.

Troubleshooting: "Model not found" on Cross-Project ref()

If you see Compilation Error: Model 'platform.dim_customers' could not be found, check in order:

1. Is dim_customers configured with access: public in the platform project? Protected models can't be cross-referenced.
2. Does the finance project's dependencies.yml list the platform project by its exact name from the platform's dbt_project.yml?
3. In dbt Cloud: has the platform project's production environment been configured and run at least once to generate artifacts?
4. In dbt Core: are you pointing to the correct manifest.json from the platform project?

Troubleshooting: Contract Enforcement Failures

Contract failures come in two flavors:

Type mismatch: Your SQL returns INTEGER but the contract declares varchar. Fix the SQL to explicitly cast: customer_id::varchar as customer_id.

Column missing: Your SQL's SELECT doesn't include a column declared in the contract. Either add it to the SQL or remove it from the contract (if it was removed intentionally, that's a breaking change requiring a version bump).

---

Summary & Next Steps

You've worked through the full lifecycle of a dbt Mesh migration: from diagnosing why the monolith needs to be split, to mapping dependency boundaries that reflect actual team ownership, to configuring public models with enforced contracts, to wiring up cross-project ref() calls, to orchestrating multi-project production deployments, and finally to managing the ongoing discipline of contract versioning and breaking change protocols.

The core insight to carry forward: dbt Mesh isn't primarily a technical architecture — it's a governance system. The technical implementation (public/protected/private access, contracts, cross-project refs) exists to enforce team boundaries that you've consciously decided on. The migration will fail if you implement the technical parts without genuine alignment on who owns what and how breaking changes get communicated.

A few things to do next:

1. Run the dependency analysis script on your actual production project today, before making any architectural decisions. The output often surprises teams — the models they assumed were foundational aren't actually that referenced, and the real hotspots are sometimes unexpected.

2. Start with read-only exploration of dbt Cloud's multi-project features if you haven't already. Create a second project in your dbt Cloud account and try to configure a cross-project dependency against your existing project. The UI configuration experience gives you real intuition for what the YAML is actually doing.

3. Investigate Dagster's dbt integration if you're running dbt Core. Dagster's asset graph is native to multi-project thinking — it was designed to represent cross-system dependencies including Mesh projects, and gives you much richer visibility into cross-project dependency failures than Airflow.

4. Draft your team's breaking change protocol before you need it. A one-page document that everyone agrees on before the first Mesh project goes live prevents a lot of friction.

5. Study dbt's --defer flag deeply — it becomes significantly more powerful in Mesh environments and is your primary tool for efficient partial-failure recovery in production.

The Mesh architecture you build now will reflect the team structure you have today. As teams evolve, be willing to renegotiate project boundaries. The contracts make this process transparent and manageable. That's the whole point.