Topic
7 posts ·view all posts
AirflowHow the triggerer frees workers: why classic sensors sleep on a slot, FileSensor vs deferrable waits, and the operational habits that keep deferred tasks from sitting forever.
ToolsAn honest comparison of the three major Python orchestrators: where each one shines, where each one hurts, and a simple decision rule.
AirflowExtract from an API, transform with pandas, load to a warehouse — scheduled in Airflow with retries, idempotency, and a local Docker setup.
AirflowA four-layer testing setup for Airflow: parse checks, DAG structure tests, task unit tests with mocked hooks, and dag.test() end-to-end runs against local infrastructure.
AirflowPractical patterns for expand, expand_kwargs, and partial in Airflow: how to size a fan-out, batch it safely, and avoid the mapped tasks that overwhelm your scheduler and metadata database.
AirflowHow to move cross-team DAG dependencies from padded cron offsets to dataset-driven triggers, including conditional scheduling, naming rules, and the failure modes nobody warns you about.
AirflowThe Airflow 3 changes that break real DAGs: the Task SDK boundary, logical_date becoming optional, removed context keys, DAG versioning, and a pre-upgrade checklist.