Designing domain-oriented dbt projects and making them work in Airflow - Talk

Nov 2, 2026

Uploaded by Mary Man

A talk titled “Designing domain-oriented dbt projects and making them work in Airflow” explains why large dbt monolith repositories stop scaling when teams and models grow. It contrasts slow feedback, increasing CI parsing time, blast-radius failures, blurred ownership, and merge friction in a single-project setup with a dbt Mesh approach. The talk covers domain project splitting, public interfaces with access controls, contracts and versions, and cross-project refs orchestrated by Cosmos and Airflow.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages31

Reading Time1 h 18 min

AccessFree to download

The problem

One monolith, many teams, why it stops scaling

How the monolith actually hurts

The design

dbt Mesh: split into domain projects with clean interfaces

The interface: +access: public

Governed change: contracts & versions

The seam: reference, don't copy

Make it run

Cosmos + Airflow orchestrate the pieces

AIRFLOW SUMMIT 2026 Designing domain-oriented dbt projects and making them work in Airﬂow Pankaj Koti Senior Software Engineer, Astronomer · Apache Airﬂow Committer & Cosmos maintainer THE STORY IN ONE LINE Your dbt project started at 20 models. Today it is 2,000, and every team blocks every other team. 01 The problem One monolith, many teams, why it stops scaling 02 The design dbt Mesh: split into domain projects with clean interfaces 03 Make it run Cosmos + Airﬂow orchestrate the pieces ACT 1 · THE PROBLEM One project. Every team. Every model. ● A single dbt_project.yml that ﬁnance, marketing, product and platform all commit to ● One `dbt run` builds the whole graph, staging, marts, everything, together ● It felt great at 50 models. It does not feel great at 2,000 03 / 30 ACT 1 · THE PROBLEM How the monolith actually hurts Slow feedback Parsing and CI grow with the whole graph, every change waits on everything Blast radius One broken staging model can stall marts for teams that never touched it Blurred ownership Who owns this model? The whole company can edit it, so no one is accountable Merge friction Dozens of engineers, one repo, one YAML, constant conﬂicts 04 / 30 ACT 1 · THE DESIGN The ﬁx: split by domain Data mesh, applied to dbt. Give each business domain its own dbt project, owned by the team that lives in that data. ● Platform owns shared, foundational models ● Finance, Marketing, Product own their own marts ● Bounded contexts, clear edges, clear owners monolith → 05 / 30 ACT 1 · THE DESIGN A domain project has a public interface ● Owned by one team, in its own directory or repo ● Exposes a small, deliberate set of public models, its API ● Everything else stays private, free to refactor without breaking others ● Other domains depend on the interface, not the internals platform project private (staging, intermediate) +access: public → the interface ﬁnance / marketing consume only the public models 06 / 30 ACT 1 · THE DESIGN dbt Mesh is the toolbox for this access public / protected / private, deﬁne the interface versions evolve a model while old consumers still work contracts pin column names & types so consumers don't break groups assign ownership to a team cross-project ref reference another project's model by name Cosmos leans on access and cross-project ref the most, that is the seam between projects. 07 / 30 ACT 1 · THE DESIGN The interface: +access: public ● Only public models can be referenced from another project ● private (default) and protected stay inside the domain ● This one line turns a model into a contract-bearing API surface upstream/dbt_project.yml models: upstream: staging: +materialized: view +access: public \# expose to others intermediate: +materialized: view +access: public \# everything else stays private 08 / 30 ACT 1 · THE DESIGN Governed change: contracts & versions Contracts Pin the column names and data types a public model promises. A pre-ﬂight check fails the build if the shape drifts, so a producer can't silently break a downstream consumer. Versions Ship v2 of a model while v1 still serves existing consumers. Consumers migrate on their own schedule, coordinated change instead of a ﬂag day. 09 / 30 ACT 1 · THE DESIGN The seam: reference, don't copy ● Declare the upstream project in dependencies.yml ● Then use ref('project', 'model') in your SQL ● One deﬁnition, one owner, no copy-paste of shared logic downstream/dependencies.yml projects: - name: upstream downstream/models/fct_ revenue.sql select * from {{ ref('upstream', 'int_orders_enriched') }} where status = 'completed' 10 / 30

FAQs

Why does a dbt monolith stop scaling when models reach around 2,000?

The talk highlights slow feedback as feedback cycles wait on the whole graph, growing CI parsing time for every change, and a larger blast radius where one broken staging model can stall marts. It also notes blurred ownership and merge friction due to many engineers sharing one repo and YAML.

What is the core design change proposed in dbt Mesh?

Split the monolith into domain projects, applying data-mesh ideas to dbt. Each business domain gets its own dbt project owned by the team that lives in that data, with bounded contexts and clear edges.

How do teams prevent breaking downstream consumers when a public model changes?

Use contracts and versions. Contracts pin public column names and data types and fail builds if the shape drifts, while versions let producers ship v2 while v1 continues to serve existing consumers so consumers migrate on their own schedule.

How are domain projects connected without copying shared logic?

Declare upstream projects in dependencies.yml and use ref('project','model') in SQL. The talk emphasizes one definition and one owner, using cross-project ref as the seam between projects.

Nov 2, 2026

Uploaded by Mary Man