About this blog

Workflow orchestration: DAGs, scheduling, and operations.

Software-defined assets, asset checks, and data-aware orchestration.

Pythonic flows, deployments, and lighter-weight scheduling.

Visual flow-based ingestion, routing, and edge collection.

Distributed processing with PySpark and Spark SQL.

Stateful stream processing, watermarks, and exactly-once sinks.

Portable batch and streaming pipelines — Dataflow, Flink, Spark runners.

HDFS, YARN, Hive, and the on-prem stack still running in many banks.

Event streaming, topics, Kafka Connect, and real-time pipelines.

Multi-tenant messaging with segment storage and built-in functions.

Change data capture from Postgres, MySQL, and SQL Server into Kafka.

SQL transformation, testing, and analytics engineering.

Virtual environments, column-level lineage, and plan-based SQL transforms.

Cloud warehousing, performance, and cost control.

Lakehouse architecture, Delta Lake, Unity Catalog, and notebooks.

Serverless analytics on Google Cloud.

RA3, Spectrum, concurrency scaling, and the AWS warehouse.

Columnar OLAP for high-cardinality analytics and real-time inserts.

In-process analytics and lightweight local pipelines.

OneLake, Lakehouse, Data Factory, and Power BI in one Microsoft plane.

Federated SQL across Iceberg, Hive, and warehouses without moving data.

Open table format, catalogs, and multi-engine lakehouses.

Incremental upserts, copy-on-write vs merge-on-read, and streaming lakes.

Open-source ELT connectors into warehouses and lakes.

Managed ELT with schema drift handling and usage-based pricing.

Star schemas, SCDs, Data Vault, and warehouse design.

Contracts, Great Expectations, Soda, dbt tests, and freshness SLAs.

Glue, Kinesis, Redshift, S3, Athena, EMR, and the AWS data platform.

Data Factory, Synapse, Event Hubs, and Microsoft Fabric.

BigQuery, Dataflow, Pub/Sub, Dataproc, and Composer.

Infrastructure as code for warehouses, IAM, and data accounts.

Airflow, Spark, Flink, and Kafka operators in production clusters.

Local stacks, image discipline, and the unit of pipeline compute.

pandas, data quality, and pipeline engineering in Python.

Query patterns, window functions, and modeling.

Lazy, multi-threaded DataFrames that replace pandas on large local jobs.

The Spark and Flink JVM language still required in many platforms.

The OLTP source of truth: CDC, logical replication, and indexing.

Document sources, change streams, and how they land in the warehouse.

Wide-column serving stores and the modeling rules that keep them fast.

Caches, feature serving, and when not to treat it as a database.

Search, logs, and OpenSearch as a serving plane beside the warehouse.

Looker, Tableau, Power BI, Superset, and the metrics layer.

LLMs, RAG pipelines, vector search, and AI on your data.

Feature stores, model pipelines, and ML infrastructure.