All stacks · Transformation
SQL transformation, testing, and analytics engineering.
dbtHow incremental dbt models actually write: unique_key as merge grain, is_incremental filters, lookback windows, and the late-arrival cases that silently drop rows.
dbtSet up a dbt project from scratch: staging models, marts, tests, and documentation — with the project structure conventions used by real teams.
dbtCutting dbt run time and warehouse spend on a large project: how to find the real critical path, when threads stop helping, and how to attribute cost to individual models.
dbtWhich Jinja patterns are worth the indirection in a production dbt project, how to test macros, and the metaprogramming that quietly makes a project unmaintainable.
dbtChoosing between check and timestamp snapshot strategies, what happens when a snapshot run is missed, and the cases where a hand-rolled SCD2 model beats dbt snapshot.
dbtHow I lay out staging, intermediate, and marts so a dbt project still makes sense after two years: naming rules, ref boundaries, and the folders that always rot.
dbtHow to get dbt CI from 50 minutes to 4 with state comparison and defer: manifest artifacts, GitHub Actions wiring, and the edge cases that bite.
dbtHow I structure dbt tests in production: generic tests on keys, unit tests on logic, dbt-expectations on shape, severity that means something.
DagsterSoftware-defined assets vs Airflow tasks, how partitions change backfills, asset checks versus dbt tests, and why warehouse credentials belong in resources.
SnowflakeHow Dynamic Tables actually refresh: TARGET_LAG, the warehouse that pays for it, streams vs DT, and why I still keep dbt for gold.
SnowflakeCREATE STREAM on a table, consume it in a Task MERGE, SHOW STREAMS, stale offsets, and the cases where Airflow still orchestrates dbt and everything outside Snowflake.
ProductivityHow to actually use AI coding assistants for pipeline work — SQL generation, dbt refactoring, debugging, and documentation — plus the failure modes to watch for.
AirbyteGreen sync, empty destination, raw JSON versus typed silver, self-host versus Cloud, and why Airbyte does not replace dbt.
FivetranMonthly active rows surprise you, warehouse writes have their own bill, a custom extractor can beat a connector, and Fivetran still does not replace dbt.
dbt turned SQL transformation into software engineering: version control, tests, documentation, and environments for the layer of your platform that used to live in undocumented stored procedures. It is the closest thing analytics engineering has to a standard.
These guides focus on what separates a demo project from a production one — incremental models that survive late-arriving data, tests that catch real regressions instead of adding noise, slim CI that builds only what changed, and project structures that stay navigable past 500 models.
No — dbt only does the T. It transforms data already loaded into your warehouse using SQL SELECT statements it materializes as tables and views. Extraction and loading belong to tools like Fivetran, Airbyte, or your own ingestion pipelines.
dbt Core is the open-source CLI and works anywhere you can run Python; you bring your own scheduler and CI. dbt Cloud adds a managed scheduler, IDE, and APIs. Most teams start with Core plus GitHub Actions and revisit Cloud when scheduling and self-service become painful.
When rebuilding it from scratch is meaningfully slow or expensive — typically large event tables. Start with plain tables; convert to incremental when build time or cost hurts, and always define how late-arriving rows get picked up (a lookback window or merge keys).
Every model should declare not_null and unique on its primary key, and relationships on important foreign keys. Beyond that, test business invariants that would silently corrupt downstream numbers. Hundreds of redundant tests slow CI and train people to ignore failures.
No — they solve different problems. dbt builds the transformation DAG inside the warehouse; Airflow orchestrates across systems (ingestion, dbt runs, exports, ML). Many production stacks run dbt as one task inside an Airflow DAG.
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.