All stacks · Languages & local engines
pandas, data quality, and pipeline engineering in Python.
PythonSchema plus SLAs as a contract owned by the producer — fail the write path when the grain or freshness breaks, instead of discovering it in a warehouse test after the fact.
PythonA practical framework for adding data quality checks to Python pipelines: what to check, where to check it, and when to fail loudly vs quarantine.
PythonWhere the memory actually goes in a pandas job, how much dtype selection and Arrow-backed strings really save, and when to stop loading and start streaming.
PythonA layered testing strategy for Python data pipelines: pure transform tests, DuckDB as a warehouse fake, property-based invariants, and what belongs in CI versus production.
PythonA production migration guide: the expression API, lazy frames and query optimization, null versus NaN semantics, and the pandas patterns with no polars equivalent.
MLOpsThe MLOps landscape explained through a data engineering lens: what feature stores actually solve, why training pipelines are just DAGs, and the skills that transfer.
DockerA Friday CI green on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes, promote the same image, and stop treating Compose as Kafka.
AI & GenAIBeyond chatbots: how data engineers use LLMs for entity extraction, data classification, documentation generation, and pipeline triage — with cost controls.
PolarsEager Polars still OOM’d a 12 GB box. Lazy scan, streaming collect, and leaving most pandas jobs alone — plus when DuckDB or Spark is the honest next step.
PrefectA flow is Python. A deployment is what actually runs. Work pools are where. Prefect will call an empty extract a success unless you raise.
AI & GenAIRetrieval-augmented generation is an ETL problem wearing an AI costume. How to build a production RAG ingestion pipeline with the skills you already have.
Python is the connective tissue of every data platform: ingestion scripts, Airflow DAGs, dbt macros, Spark jobs, and the glue between all of them. The difference between a script that works on your laptop and a pipeline that runs unattended for a year is mostly engineering discipline, not syntax.
These tutorials focus on that discipline — typed, tested, dependency-pinned pipeline code; pandas and polars used deliberately instead of by habit; and the packaging and environment practices (uv, lockfiles, containers) that make Friday deploys uneventful.
Comfortable fluency, not CS-competition depth: functions, dataclasses, comprehensions, context managers, typing, and error handling cover most pipeline code. The differentiator is engineering habits — tests, logging, idempotency — not exotic language features.
polars for new transformation-heavy code: multi-core, lazy execution, stricter API, and far better performance on GB-scale data. pandas remains fine for small data and unavoidable for its ecosystem. For SQL-shaped work, DuckDB often beats both.
Unit-test transformation functions with small in-memory frames, contract-test inputs (schemas, nullability), and run one end-to-end test against local files or DuckDB in CI. Avoid tests that need production credentials — they rot into permanently-skipped decorations.
uv is a Rust-based package and environment manager that replaces pip, virtualenv, and pip-tools with one very fast tool and a real lockfile. For new projects there is little reason not to use it; migrating existing projects is usually an hour of work.
Yes. Pipelines are long-lived and refactored under pressure; mypy or pyright catches the None-handling and schema-shape mistakes that otherwise surface at 3 a.m. Annotate function boundaries first — full coverage can come later.
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.