Airflow vs Dagster vs Prefect in 2026: Which Orchestrator Should You Pick?

Every pipeline needs something to run it on schedule, retry it when it fails, and tell you when it does not recover. Three Python-native orchestrators dominate that job. Having run production workloads on all three, here is the honest comparison.

This is not a feature matrix from a vendor site. It is which object you will debug — a DAG run, an asset materialization, or a flow run — and whether your cloud already operates the control plane.

flowchart TD
  need[Need an orchestrator] --> size{Team and ecosystem?}
  size -->|industry default| af[Airflow]
  size -->|asset lineage + tests| dg[Dagster]
  size -->|small Python team| pf[Prefect]

What all three actually do

They are not warehouses. They are not Spark. They kick off work, pass small metadata, retry, and page you.

Shared requirements you should demand from any of them:

• Retries with backoff on the task, not “re-run the whole day by hand”
• Idempotent work (same logical date → same tables)
• A UI that shows why it is not running (deps, slots, crash)
• Somewhere the schedule lives in git
• Secrets that are not in the DAG file

If your “orchestrator” is a cron on an EC2 box named airflow that nobody can SSH into, you do not have Airflow. You have folklore.

A first pipeline tutorial (Airflow-shaped) lives here: Building your first data pipeline. The habits — idempotency, no giant XCom, test before schedule — transfer.

Apache Airflow: the default for a reason

The case for it. Airflow is the industry standard. Every managed cloud has a hosted version (MWAA, Cloud Composer, Astronomer), every tool ships an Airflow provider, and every data engineer you hire already knows it. The TaskFlow API cleaned up most of the old boilerplate.

The case against. Local development remains clunky (you run a scheduler + webserver to test a DAG), backfills are workable but awkward, and its scheduler-centric model treats data as an afterthought — it schedules tasks, not datasets, though data-aware scheduling has been improving.

Pick it when: you want the safest choice, managed hosting, and the largest hiring pool.

# TaskFlow — still Airflow, less XML-in-Python
from datetime import datetime
from airflow.decorators import dag, task

@dag(start_date=datetime(2026, 1, 1), schedule="@daily", catchup=False)
def orders_daily():
    @task
    def extract(ds: str) -> str:
        # write bronze for ds; return a path, not a dataframe
        return f"s3://lake/bronze/orders/dt={ds}/"

    @task
    def transform(path: str, ds: str) -> None:
        # merge silver; idempotent on ds
        ...

    transform(extract(), "{{ ds }}")

orders_daily()

Ops notes that bite

• Executor choice (Local, Celery, Kubernetes) is an architecture decision. MWAA/Composer hide some of it and constrain the rest.
• Connections and Variables in the metadata DB become tribal state. Move them to a secrets backend early.
• catchup=True on a three-year DAG is how you discover the cluster is not sized for history. Default False until you intend a backfill.
• XCom is for pointers (table name, partition, file URI). It is not a data bus. A 200 MB dataframe in XCom will make you famous.
• Providers lag or pin awkwardly on managed Airflow versions. Read the hosted version’s provider list before you promise SnowflakeOperator vN.

Data-aware scheduling (outlets / datasets) is the right direction. It is still easier to reason about than Dagster assets if your team’s mental model is “tasks at 6am.” It is still worse than Dagster if your mental model is “these five tables.”

Dagster: assets over tasks

The case for it. Dagster inverts the model — you declare the data assets (tables, files, models) and Dagster derives the execution. The payoff is a UI that shows your actual data lineage, first-class dbt integration that maps every dbt model to an asset, and dramatically better local development and testing.

The case against. Smaller ecosystem, more concepts to learn up front (assets, ops, resources, definitions), and the asset model can feel like overhead for simple “run this script at 6am” jobs.

Pick it when: your pipelines are mostly building tables/models (especially around dbt) and you value lineage and testability.

from dagster import asset, AssetExecutionContext, Definitions

@asset(deps=["stg_orders"])
def gold_orders(context: AssetExecutionContext) -> None:
    context.log.info("merge gold.orders for partition %s", context.partition_key)
    # warehouse SQL / dbt invocation — the asset *is* the table

defs = Definitions(assets=[gold_orders])

Ops notes that bite

• You will rewrite “a DAG of bash” into assets. That is the point. Budget the rewrite. A lift-and-shift of Airflow operators into Dagster ops without assets wastes the product.
• Resources (warehouse clients, S3) belong in resources, not globals. Tests become possible; so does “forgot to configure prod.”
• Partitions and freshness policies are where Dagster earns its keep. If you never use them, you bought a nicer Airflow.
• Dagster+ (or self-host) is the production control plane. “We will just run dagster-webserver on a VM” is an Airflow-shaped mistake with different YAML.
• Hiring pool is smaller. You are hiring for the model, not for “I memorized Jinja ds.”

If dbt is the center of the platform, Dagster’s mapping of models → assets is the best of the three. Airflow’s Cosmos / dbt operators are adequate. Adequate is fine when Airflow is already the law.

Prefect: the lightest touch

The case for it. The smallest gap between “Python script” and “orchestrated pipeline” — decorate functions with @flow and @task and you are done. Dynamic, parameter-heavy, event-driven workflows that fight Airflow’s static DAG model feel natural here.

The case against. The thinnest batteries-included integration catalog of the three, and more architectural decisions left to you (where things run, how they are deployed).

Pick it when: your workflows are dynamic or event-driven, or you are a small team that wants minimal ceremony.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=30)
def pull_api(day: str) -> str:
    return f"/tmp/orders-{day}.json"

@task
def load_wh(path: str, day: str) -> None:
    ...

@flow
def orders_flow(day: str) -> None:
    load_wh(pull_api(day), day)

Ops notes that bite

• Work pools / workers are the production story. “It ran on my laptop with Prefect Cloud” is not a deploy.
• Dynamic DAGs are a feature until nobody can answer “what runs on Tuesday.” Put parameters in the UI and in the runbook.
• Fewer official warehouse operators means you will write the Snowflake/BQ client. That is less magic and more tests — fine if you wanted that.
• Prefect Cloud vs OSS is a real fork in auth, scheduling, and who pages. Decide before you have 80 flows.

Prefect is where teams who hate YAML go to still get retries. That is a valid personality if the team is small and the graphs are dynamic. It is a weaker default if you need MWAA tomorrow because security already approved it.

The comparison table

	Airflow	Dagster	Prefect
Mental model	Tasks in DAGs	Data assets	Decorated functions
Local dev & testing	Weak	Excellent	Excellent
dbt integration	Adequate	Best in class	Adequate
Managed hosting	MWAA, Composer, Astronomer	Dagster+ / Cloud	Prefect Cloud
Hiring pool	Largest	Growing	Growing
Dynamic graphs	Awkward (you fight the DAG)	Possible, more structured	Natural
Data-aware / lineage	Improving (datasets)	Native	You build more of it
Provider / operator catalog	Widest	Smaller, opinionated	Thinnest
Backfills	First-class, clunky	Partitions / backfills	Parameters + reruns
Best fit	Default / compliance / hire	dbt + tables + tests	Small team / dynamic / events

Local development (the hidden decision)

Airflow: Docker Compose or a managed “local astro/dev” story. Unit-testing a DAG is possible (dag.test, pytest + dagbag). It is never as fast as pytest on a function. People skip it. Then the DAG fails on the scheduler because of a parse-time query.

Dagster: dagster dev and asset tests are the product. If your team actually writes tests, this is the orchestrator that does not fight you.

Prefect: flows are functions. Tests look like tests. You still need a worker story before prod.

If “we do not have time for local Airflow” is already the culture, switching to Dagster will not create discipline. It will only make discipline cheaper. That is still a reason to pick Dagster for a new platform.

Hosting: pick the one your cloud already runs

This rule saves more pain than any feature:

You already have	Default orchestrator
AWS, security likes IAM + VPC	MWAA (Airflow)
GCP, Composer approved	Composer (Airflow)
Azure, ADF for movement	Airflow or ADF; do not add a third without a reason
No managed Airflow, dbt-centric	Dagster+ is a clean buy
Tiny team, Python scripts, events	Prefect Cloud

“But Airflow is old” is not a reason to run Dagster on a weekend Kubernetes cluster your platform team does not want. Managed Airflow is boring. Boring is how batch ships.

Astronomer and similar are Airflow with a nicer deploy if MWAA’s version pin is the blocker. That is still Airflow — hiring and providers stay.

dbt, Spark, and “who owns the schedule”

• dbt Cloud can schedule models. Fine for a warehouse-only shop. The moment you have “dbt then a Python quality check then an ML job,” you want an orchestrator around dbt, not a second calendar.
• Databricks Workflows / Snowflake Tasks can own a closed ecosystem. Adding Airflow on top to trigger_job is normal. Reimplementing Spark inside Airflow workers is not.
• Dagster + dbt is the tightest lineage story. Use it if lineage is a requirement, not a slide.

One schedule owner per pipeline. Two calendars is how gold updates twice and BI still sees yesterday.

Switching cost

Rewriting 200 DAGs is a quarter you will not spend on data quality. Operators, connections, SLAs, and on-call muscle memory do not port. The unglamorous truth: all three run production workloads fine. The switching cost between them is far higher than the difference between them.

Migrate only when:

• The current orchestrator is unmanaged and on fire
• You are already rewriting the pipelines (warehouse move)
• A hard requirement (asset tests, event-native graphs) is blocked, not merely inconvenient

Do not migrate because a conference talk made assets look clean.

flowchart TD
  fail[Task error] --> n{Attempts left?}
  n -->|yes| wait[Backoff]
  wait --> run[Run the same logical date again]
  run --> sink{Write pattern}
  sink -->|DELETE plus INSERT or MERGE on ds| ok[Same row count]
  sink -->|INSERT only| dup[Doubled fact]
  n -->|no| page[Task failed, page]

The retry re-runs the same interval. The sink has to treat that as a rewrite, not a second append.

# Same contract on Airflow, Dagster, or Prefect.
# retries=2 only helps if this MERGE is keyed on the logical date.
from datetime import timedelta

DEFAULT_ARGS = {
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
}

def upsert_orders(conn, ds: str) -> None:
    conn.execute(
        """
        merge into warehouse.orders as t
        using (
          select order_id, amount, updated_at
          from staging.orders
          where order_date = %s
        ) as s
          on t.order_id = s.order_id
         and t.order_date = %s
        when matched then update set
          amount = s.amount,
          updated_at = s.updated_at
        when not matched then insert
          (order_id, order_date, amount, updated_at)
        values (s.order_id, %s, s.amount, s.updated_at)
        """,
        (ds, ds, ds),
    )

A retry that appends is a quality incident with a green task square. The orchestrator did its job. The sink did not.

Pitfalls

• Giant XCom / result blobs in Airflow. Write a table.
• Parse-time warehouse queries in Airflow DAG files. The scheduler will thank you by falling over.
• Dagster without assets — you paid for a new runtime to keep a task graph.
• Prefect without a worker pool — laptop-as-prod.
• Three orchestrators (ADF + Airflow + Databricks Workflows) because each team picked independently. Pick a default; allow one exception with an owner.
• Catchup / backfill against non-idempotent loads. You will double-count revenue. See Python data quality checks.
• Sensors that poll every 30s on Airflow. Slot burn, angry APIs. Prefer deferrable sensors or an event.
• “We’ll use KubernetesExecutor later” as a substitute for making tasks small and idempotent.

Decision rules

• Team already knows Airflow, or you need cloud-managed hosting today → Airflow (MWAA / Composer / Astronomer).
• Platform is dbt/table-centric and lineage / testability is a stated goal → Dagster.
• Dynamic workflows, small team, minimal ceremony, event-shaped jobs → Prefect.
• Security already approved one hosted control plane → that one, even if a blog prefers another.
• Existing Airflow that is stable → keep it. Spend the energy on idempotent tasks and quality checks.

If two rules fire, hosting and hiring win. Features you can approximate; a control plane you cannot deploy is a non-starter.

FAQ

Should I switch off Airflow because assets look cleaner? Not if Airflow is stable and hosted. Switching cost is a quarter of pipeline work. Migrate when the control plane is on fire, you are already rewriting, or a hard requirement is blocked.

Dagster without assets — is that fine? You paid for a new runtime to keep a task graph. Budget the rewrite to tables. Ops-as-Airflow-operators wastes the product.

Is Prefect “Airflow without YAML” enough for a regulated shop? Maybe for a small team. If security already approved MWAA or Composer, that is the orchestrator. A nicer decorator will not pass a review that already finished.

Why did a retry double yesterday’s revenue? The load appended. DELETE+INSERT the ds partition or MERGE on the grain. Retries are a scheduler setting. Idempotency is a write pattern.

Can dbt Cloud replace the orchestrator? For warehouse-only shops, sometimes. The moment you have “dbt then a Python check then an ML job,” you want one schedule around dbt, not a second calendar.

XCom for the pandas DataFrame? No. Write a table or a file. Pass the path. A 200 MB XCom will make you famous.

What this means for data engineers

Pick one orchestrator and make the work boring: idempotent loads, small metadata between steps, tests you can run without a cluster. Airflow is the default because of gravity. Dagster is the upgrade when tables are the product. Prefect is the light path when the graph will not sit still.

The orchestrator will not fix a bad grain. It will only run it every day, with retries, forever.