Slim CI in dbt: state:modified+ and --defer on a 1,000-Model Project

Our CI used to rebuild the whole project on every pull request. At 300 models that was 20 minutes and tolerable. At 1,000 models it was 50 minutes, engineers stopped waiting for it, and merges started happening on “CI is probably fine.”

Slim CI fixed it: 4 to 8 minutes for a typical PR, and CI got its authority back. The idea is simple. Build only the models the PR changed, plus everything downstream of them. For every upstream model you did not touch, read prod’s copy instead of rebuilding it. Two flags do this: --select state:modified+ and --defer.

The simple idea has sharp edges — where the comparison manifest comes from, what “modified” means when macros and configs change, and what happens the day the artifact is missing. This post is the setup I run and the edge cases I have paid for.

flowchart LR
  pr[PR branch] --> compile[Compile PR manifest]
  prod[Prod job] --> artifact["Stored manifest.json"]
  compile --> diff{state comparison}
  artifact --> diff
  diff --> changed["Build modified plus children"]
  diff --> defer["Defer unchanged refs to prod schema"]

Two manifests walk into a diff. Only the difference gets built.

The two flags, precisely

--select state:modified+ asks dbt to compare the current project’s manifest against another manifest you point at with --state, and select every node that differs, plus (+) all descendants. “Differs” covers SQL body, config, database representation, and upstream macros — more on that later.

--defer changes what ref() resolves to. Normally a ref points at the target schema, which in CI is an empty scratch schema. With defer, any ref to a model that is not being built in this run resolves to its location in the state manifest — that is, prod. Your changed model reads real prod inputs without rebuilding them.

Put together:

dbt build \
  --select state:modified+ \
  --defer \
  --state ./prod-artifacts \
  --target ci

One PR touching stg_payments builds stg_payments, its children, its tests, and nothing else. The 990 untouched models cost zero compute and zero minutes.

Where the state manifest comes from

Everything rests on ./prod-artifacts/manifest.json being an honest snapshot of what is deployed. Two sources work:

1. Your prod job uploads its manifest after every successful run. S3, GCS, or CI artifact storage. This is what I do.
2. dbt Cloud’s --defer against the environment handles it for you, if you are on Cloud.

For self-hosted, the prod deploy job ends with an upload:

# .github/workflows/dbt-prod.yml (tail of the deploy job)
      - name: Run dbt
        run: dbt build --target prod

      - name: Upload manifest
        if: success()
        uses: actions/upload-artifact@v4
        with:
          name: dbt-prod-manifest
          path: target/manifest.json
          retention-days: 30

And the CI job downloads it before comparing:

# .github/workflows/dbt-ci.yml
name: dbt-slim-ci
on:
  pull_request:
    paths: ['models/**', 'macros/**', 'snapshots/**', 'seeds/**', 'dbt_project.yml', 'packages.yml']

jobs:
  slim-ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: pip install dbt-snowflake==1.9.*

      - name: Fetch prod manifest
        id: manifest
        continue-on-error: true
        uses: dawidd6/action-download-artifact@v6
        with:
          workflow: dbt-prod.yml
          name: dbt-prod-manifest
          path: prod-artifacts
          workflow_conclusion: success

      - name: Slim build
        if: steps.manifest.outcome == 'success'
        run: |
          dbt deps
          dbt build --select state:modified+ --defer \
            --state prod-artifacts --target ci

      - name: Fallback full build of changed folders
        if: steps.manifest.outcome != 'success'
        run: |
          dbt deps
          dbt build --select marts staging --target ci

Note the fallback. The first time your artifact retention expires over a holiday, CI will find no manifest. Without a fallback, some setups quietly select nothing and go green on an empty build. A green check that tested nothing is worse than a red one.

The --target ci profile writes to a per-PR schema (ci_pr_1234), which a nightly job drops. Deferred refs read prod; only the diff lands in the scratch schema.

What “modified” actually catches

state:modified is a union of sub-selectors, and knowing them explains most surprises:

• state:modified.body — the SQL changed.
• state:modified.configs — materialization, tags, meta, schema.
• state:modified.relation — the database/schema/alias changed.
• state:modified.macros — a macro this model calls, changed.
• state:modified.contract — the model’s contract changed.

The macros one is the big one. Edit a macro that 400 models call — a cents_to_dollars, a custom generate_schema_name — and all 400 are modified. That is correct; the compiled SQL of all 400 really did change. But your 4-minute CI just became a 40-minute CI for that one PR.

I handle it two ways. For genuinely global macros, accept the big run; that is the risk surface of the change and CI should reflect it. For macros under active development, split the PR: land the macro behind no call sites first, then migrate callers in batches. The macro-writing guide covers when a macro earns that blast radius at all.

Environment-dependent configs: the false-positive factory

Classic failure: your schema names include the target ({{ target.name }}_marts), so every model’s config differs between the prod manifest and the CI compile. Everything is “modified.” Slim CI silently becomes full CI and nobody notices for a month — the check is green, just slow and expensive.

Two fixes. Prefer writing configs so they do not render target-dependent values directly — push the logic into generate_schema_name, which is compared as a macro, not per model. And set the flag that makes dbt compare unrendered config values where it can:

# dbt_project.yml
flags:
  state_modified_compare_more_unrendered_values: true

Then verify. Run the comparison locally and read the list:

dbt ls --select state:modified --state prod-artifacts --target ci

If a PR that touched one model lists 200, you have a rendering leak. Chase it now; a slim CI that selects everything gives you full-CI cost with slim-CI confidence.

flowchart TD
  edit[One-line PR] --> ls["dbt ls state:modified"]
  ls --> one{Lists roughly one model?}
  one -->|yes| healthy[Comparison is healthy]
  one -->|no| leak["Rendered config leak (target names, env vars)"]
  leak --> fix["Unrendered compare flag or macro-ize the config"]

Audit the selector with dbt ls before trusting it. It lies quietly, never loudly.

Defer’s fine print

Defer reads prod. That means CI results depend on prod’s current state, which cuts both ways.

The good: your changed model is tested against real upstream data, including all the ugly rows dev schemas never have. Combined with a testing strategy that runs on modified-plus-children, a PR gets meaningful validation in minutes.

The sharp edge: if prod is mid-incident — an upstream table is stale or half-loaded — your PR can fail CI for reasons that have nothing to do with the PR. I keep --favor-state off so that anything actually built in the PR schema takes precedence over prod, and I accept the occasional rerun during incidents. It is a fair trade for testing against reality.

Also remember defer applies to tests. A relationships test between a model you built and a dimension you did not will read the prod dimension. Late-arriving keys that self-heal in prod can flake in CI; that is a warn_if threshold decision, not a defer bug.

One more interaction: incremental models in CI build from scratch (there is no existing relation in the PR schema), so CI validates full-refresh logic, not the incremental merge path. For merge-path coverage you want unit tests with fixtures, not slim CI.

Pitfalls

No fallback when the manifest is missing. Selecting on state with no state should fail loudly or build a deliberate subset. Empty green checks erode the whole system.

Comparing against a stale manifest. If the prod job has been red for a week, the stored manifest is a week old and the diff is wrong in both directions. Alert on prod-manifest age, not just prod-job status.

Per-PR schemas that never get dropped. A thousand ci_pr_* schemas later, your metadata queries crawl and your storage bill asks questions. Drop them nightly by pattern.

Seeds and snapshots surprise-selected. A CSV re-save with different line endings makes the seed “modified,” and snapshots selected in CI write to your snapshot schema unless the CI target redirects them. Exclude snapshots in CI: --exclude resource_type:snapshot.

Treating a big selection as a bug. Sometimes 400 models are modified because you changed a macro 400 models call. The selector is telling you the blast radius. Read it before you shrink it.

FAQ

Does slim CI replace a full nightly build? No. I still run a full-project job on a schedule. State comparison verifies diffs; the nightly proves the whole graph still compiles, builds, and passes together, and it regenerates the manifest that CI depends on.

state:modified or state:modified+? Always + in CI. Building a changed model without its children skips exactly the regressions you built CI to catch — a renamed column passes alone and breaks every consumer.

How does this interact with dbt Cloud? Cloud’s CI jobs do the manifest storage and defer wiring for you — “compare changes against environment.” The concepts are identical; this post is the self-hosted spelling of it.

What is result:error+ for? Retry, not CI. After a failed prod run, --select result:error+ --state ./last-run-artifacts rebuilds failures and their children. Same machinery, different job.

Is 1,000 models even sane in one project? That is a project structure question. Slim CI is one of the things that makes a big monorepo livable, but it does not fix ownership or folder sprawl.

What this means for data engineers

CI that takes 50 minutes is CI that gets skipped, and CI that gets skipped is a formality. State comparison plus defer is the highest leverage change you can make to a growing dbt project: minutes-long checks that run the changed subgraph against real prod inputs.

The engineering is in the plumbing, not the flags. Store the prod manifest on every successful deploy, alert on its age, keep a loud fallback for the day it is missing, and audit dbt ls --select state:modified whenever the selection size surprises you in either direction.

Fast CI is not a luxury. It is what makes people run it.