Databricks10 min read
Delta Lake vs Apache Iceberg: Default on Databricks vs Shared Lake
Table-format choice in 2026: deletion vectors, catalogs, and which engine you are willing to lock in.
- databricks
- snowflake
- tools
All stacks · Warehouses & analytics
Databricks
Lakehouse architecture, Delta Lake, Unity Catalog, and notebooks.
Databricks
Databricks9 min read
Databricks and Delta Lake: A Practical Introduction to the Lakehouse
What the lakehouse actually is, how Delta Lake adds ACID transactions to cheap object storage, and the medallion architecture in practice.
- databricks
- delta-lake
- spark
Databricks10 min read
Declarative Pipelines in Databricks: Expectations, Streaming Tables, and When the Abstraction Fights You
What DLT genuinely buys you, how expectations and streaming tables behave in production, and the four situations where I still write plain Spark jobs instead.
- databricks
- pipelines
Databricks11 min read
Cutting a Databricks Bill: Job Clusters, Spot, Photon, and Finding the Top Offenders in System Tables
How I take 30 percent off a Databricks bill in a week: kill all-purpose clusters for jobs, price spot properly, test Photon per workload, and rank spend with system.billing.
- databricks
- cost
Databricks8 min read
Unity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastore
How I structure Unity Catalog for real teams: three-level namespace design, group-based grants, lineage that works, and a hive_metastore migration that ships.
- databricks
- governance
Databricks9 min read
Delta Lake Internals: The Transaction Log, Checkpoints, and Why Small Files Happen
How the _delta_log actually works: JSON commits, optimistic concurrency, checkpoints, deletion vectors, and the small-file problem nobody escapes.
- databricks
- delta-lake
Related reading
Kubernetes4 min read
Kubernetes Operators for Data Jobs: Unbounded Pods, Then the Cluster Dies
Airflow and Spark can spawn unbounded pods, requests and limits are the difference between busy and evicted, crash loops hide in operators, and MWAA or Databricks is sometimes the right way to not run K8s.
- kubernetes
- orchestration
Tools11 min read
Snowflake vs Databricks in 2026: Pick the Workload, Not the Logo
An honest head-to-head: SQL warehousing vs Spark lakehouse, Iceberg interoperability, Cortex vs Mosaic, and when you actually need both.
- snowflake
- databricks
- tools
About Databricks
Databricks turned Spark into a platform: Delta Lake for reliable tables on object storage, Unity Catalog for governance, and serverless SQL warehouses that compete head-on with Snowflake. The lakehouse pitch is real — one copy of data serving BI, engineering, and ML.
These articles cover the platform as it is actually operated: Delta internals and optimization, Auto Loader ingestion, Unity Catalog permission design, workflows versus external orchestrators, and the cost model behind DBUs before the first invoice surprises you.
What you'll learn here
- Delta Lake internals: transaction log, OPTIMIZE, liquid clustering, and vacuum discipline
- Unity Catalog: metastore layout, grants, lineage, and cross-workspace sharing
- Ingestion with Auto Loader and COPY INTO, batch and streaming
- Job clusters vs all-purpose clusters and the DBU cost model
- When Databricks SQL replaces a warehouse — and when it should not
Frequently asked questions
What is a lakehouse, concretely?
Warehouse-grade tables (ACID transactions, schema enforcement, time travel) implemented directly on object storage via an open format like Delta or Iceberg, queryable by multiple engines. You keep one copy of data in open files instead of loading a proprietary warehouse.
Databricks or Snowflake?
Snowflake for SQL-first analytics with minimal operations; Databricks for Spark, ML, and streaming-heavy platforms that want to own their storage. Plenty of enterprises run both. Our comparison article walks through workload-by-workload decisions rather than a single verdict.
Why is my Databricks bill high?
Usually all-purpose clusters running interactive workloads around the clock, oversized driver nodes, and jobs that never enabled autoscaling or spot instances. Job clusters with aggressive auto-termination are the single biggest saver.
Do I need OPTIMIZE and VACUUM?
Yes. Streaming and frequent small writes create small files that slow every read; OPTIMIZE compacts them (liquid clustering supersedes static ZORDER for new tables). VACUUM removes stale files but respects the retention window — never shorten it below your time-travel needs.
Is Unity Catalog optional?
Not for new deployments. It is the governance layer for everything current on the platform — fine-grained permissions, lineage, sharing — and new features increasingly assume it. Migrating from hive_metastore later is much more painful than starting on UC.
New Databricks posts, straight to your inbox
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.