Airflow Interview Questions cheat sheet
Scheduling semantics, executors, XComs, backfills, and Airflow 3 changes that orchestration interviews focus on.
Core concepts and scheduling
What is a DAG and why must it be acyclic?- A DAG is a collection of tasks with directed dependencies describing execution order, not the work itself. Acyclicity guarantees a topological order exists, so the scheduler can always decide what runs next and termination is provable. A cycle would mean a task waiting on its own descendant, which the parser rejects at import time.
Explain the difference between logical date and run date.- The logical date (formerly execution_date) is the start of the data interval the run covers, so a daily DAG for 2026-03-01 starts after that day ends. Airflow 2 added data_interval_start and data_interval_end to make this explicit, and Airflow 3 removed the misleading execution_date alias. Interviewers ask because everyone has been bitten by an off-by-one-day partition.
What is catchup and when do you disable it?- With catchup=True the scheduler creates a run for every missed interval between start_date and now, which can spawn hundreds of runs the moment you unpause a DAG with an old start date. Set catchup=False for jobs that only care about current state, and keep it on for partitioned backfillable pipelines where every interval must be processed.
How do you prevent overlapping runs of the same DAG?- max_active_runs=1 at the DAG level serializes runs, and depends_on_past=True at the task level makes a task wait for its own previous run to succeed. The difference matters: depends_on_past will freeze the pipeline forever after one failure, so it needs an alerting and clearing plan, while max_active_runs simply queues.
What is a sensor, and what is the danger of the default mode?- A sensor polls for a condition such as a file or partition arriving. In the default poke mode it occupies a worker slot for its entire timeout, so a handful of long sensors can deadlock the pool. Use mode='reschedule' so it frees the slot between checks, or deferrable operators, which hand the wait to the triggerer and use almost no resources.
How does the scheduler decide what to run?- It parses DAG files on a loop, creates DagRuns for due intervals, and moves task instances from scheduled to queued when dependencies, pool slots, and concurrency limits allow; the executor then runs them. Throughput is bounded by parsing time, so heavy top-level code in DAG files is the most common cause of a sluggish scheduler.
Executors and deployment
Compare the LocalExecutor, CeleryExecutor, and KubernetesExecutor.- LocalExecutor runs tasks as subprocesses on the scheduler host — simple, fine for modest loads, no horizontal scaling. CeleryExecutor distributes to a pool of persistent workers through Redis or RabbitMQ, giving low task latency but fixed worker environments. KubernetesExecutor launches a pod per task for full isolation and elastic scale, at the cost of pod startup latency of several seconds.
When would you pick a hybrid or the KubernetesPodOperator?- The KubernetesPodOperator runs arbitrary containers from any executor, which is the cleanest way to let teams ship their own dependencies without rebuilding the Airflow image. Choose it over KubernetesExecutor when most tasks are light and only some need isolation, so you avoid paying pod startup on every trivial task.
What is the role of the metadata database?- It stores DAG structure, task instance state, variables, connections, and XComs, and it is the coordination point for scheduler, workers, and API server. It is also the usual scaling bottleneck: heavy XCom use and long log retention bloat it. Postgres is the recommended backend, and periodic 'airflow db clean' is real operational work.
How do you isolate dependencies between DAGs that need conflicting libraries?- Container-per-task with KubernetesPodOperator or DockerOperator, or PythonVirtualenvOperator and ExternalPythonOperator for lighter isolation within the same image. Putting every team's requirements in one image eventually produces an unresolvable dependency graph, so interviewers like hearing that you plan for isolation before it breaks.
What changed architecturally in Airflow 3?- Airflow 3, released in 2025, introduced a Task Execution API so workers no longer connect directly to the metadata database, which enables remote and multi-language execution and improves the security boundary. It also shipped a rewritten React UI, DAG versioning so a run keeps the code version it started with, and made assets a first-class scheduling concept.
Data passing and task communication
What is an XCom and what should you never put in one?- XComs pass small values between tasks through the metadata database. Never push DataFrames, file contents, or anything above a few kilobytes — you will bloat and slow the database. Pass a pointer instead: an S3 key, a table name, a partition value. Custom XCom backends can transparently store payloads in object storage when you genuinely need larger objects.
How does the TaskFlow API change how you write DAGs?- Decorating functions with @task and calling them builds dependencies and XCom wiring automatically from the Python call graph, so return values flow as arguments instead of manual xcom_pull. It is the recommended style for Python-heavy DAGs; classic operators remain right for prebuilt integrations, and the two mix freely in one DAG.
What is dynamic task mapping and how does it differ from generating tasks in a loop?- expand() creates task instances at runtime from a list produced by an upstream task, so the count can change per run and the UI groups them as mapped instances. A Python loop at parse time fixes the task count in the DAG structure and requires a re-parse to change, so it cannot depend on runtime data.
How do you branch conditionally?- BranchPythonOperator or @task.branch returns the task_id or ids to follow, and unselected downstream tasks are skipped. The gotcha is the join task: with the default all_success trigger rule it will also be skipped, so set trigger_rule='none_failed_min_one_success' on the task where branches converge.
How do you trigger work across DAGs?- TriggerDagRunOperator pushes, ExternalTaskSensor pulls and must have its execution_delta or execution_date_fn aligned or it hangs forever. In Airflow 2.4 and later, asset-based (formerly dataset) scheduling is usually better: the producer declares outlets and consumers schedule on those assets, which expresses data dependency rather than time coupling.
Reliability, retries, and backfills
What retry settings would you set as defaults, and why?- Typically retries=2 or 3 with retry_delay of a few minutes and retry_exponential_backoff=True, plus an execution_timeout so a hung task cannot run forever. The important qualifier is that retries are only safe when the task is idempotent; retrying a non-idempotent append duplicates data, which is a more damaging outcome than the original failure.
How do you run a backfill safely?- Confirm the tasks are idempotent and partition-scoped, limit concurrency with max_active_runs and pools so the backfill does not starve production runs, and run it in a window when the warehouse has capacity. Airflow 3 moved backfills to be scheduler-managed rather than a blocking CLI process, so they are visible and cancellable in the UI.
Explain trigger rules.- The default all_success runs a task only if every upstream succeeded. all_done runs regardless of outcome and suits cleanup. one_failed and all_failed drive alerting paths, none_failed_min_one_success handles branch joins, and always ignores upstream entirely. Misused trigger rules are why a cleanup task silently never ran after a failure.
What are pools and priority_weight for?- A pool caps how many task instances of a group run at once, which is how you protect a fragile source database or a limited API quota from being hammered by parallel tasks. priority_weight orders which queued tasks claim free slots first. Together they express the resource contract that concurrency settings alone cannot.
A task succeeded in Airflow but produced no data. How do you catch that?- Exit status is not data quality. Add explicit checks after the load — SQLColumnCheckOperator, a dbt test, or Great Expectations — and fail the task on zero rows or a row count outside the expected band. Then add SLAs or deferrable timeout alerts for lateness, since a task that never runs raises no failure at all.
Design, testing, and operations
Why is top-level code in a DAG file a problem?- The scheduler re-parses every DAG file on an interval (default 30 seconds for the parsing loop), so an API call, a database query, or a heavy import at module level runs constantly and delays scheduling for the whole deployment. Move that work inside task functions and keep imports light, checking parse time with 'airflow dags list-import-errors' and the DAG parse duration metric.
How do you test a DAG?- Three tiers: a static import test asserting the file parses and has no cycles, unit tests on the Python callables with Airflow objects mocked, and a dag.test() or integration run against a scratch environment. The parse test alone catches most breakages and is cheap enough to run on every commit.
How should secrets be handled?- Not in Variables as plain text and never in code. Use Connections with a secrets backend such as AWS Secrets Manager, Vault, or GCP Secret Manager so the value is fetched at runtime and never lands in the metadata database. Remember that anything printed to logs is visible in the UI, and Airflow only masks names matching its sensitive-field patterns.
What makes a DAG idempotent, and why does Airflow care so much?- Every task must produce the same result when re-run for the same logical date, which means writing to a partition derived from the interval rather than from now(), and using overwrite or MERGE instead of append. Airflow's entire retry, clear, and backfill model assumes idempotency; without it, normal operations corrupt data.
How do you monitor Airflow itself?- Watch scheduler heartbeat and DAG parse time, task queue depth and time in the queued state, pool utilization, and metadata database size and connection count. Airflow emits StatsD and OpenTelemetry metrics. The signal that catches most incidents early is queued duration rising, which means capacity is short before anything actually fails.
When is Airflow the wrong tool?- For sub-minute latency or event-per-record streaming, since the scheduler works in intervals and task startup costs seconds. Also for pure in-warehouse transformation DAGs, where dbt or a warehouse-native scheduler is simpler, and for simple cron jobs that need no dependencies. Saying this honestly reads better than claiming Airflow fits everything.
From DataLane — tutorials at/blog, practice SQL live in theplayground.