Databricks Interview Questions cheat sheet
Delta Lake internals, Unity Catalog, cluster and warehouse choices, and lakehouse design questions asked at senior level.
Platform architecture
Explain the control plane and compute plane split.- The control plane, run by Databricks, holds the web UI, job scheduling, notebook metadata, and cluster manager. Compute runs in the customer's cloud account (classic compute) or in a Databricks-managed account (serverless), and data stays in your object storage. Interviewers ask because the boundary determines your network, cost, and compliance story.
What is a lakehouse and what does it change?- Open file formats in object storage with a transactional metadata layer on top, giving ACID transactions, schema enforcement, and time travel over what used to be a schema-on-read data lake. The consequence is one copy of data serving BI, streaming, and machine learning instead of a lake feeding a separate warehouse, so there is no nightly copy to keep in sync.
Compare all-purpose clusters, job clusters, and SQL warehouses.- All-purpose clusters are interactive, shared, and stay up — convenient and the biggest source of wasted spend. Job clusters spin up per job run and terminate after, which is cheaper and isolates failures. SQL warehouses are tuned for BI concurrency with Photon and result caching; serverless SQL warehouses start in seconds instead of minutes.
What is Photon and when does it not help?- A vectorized C++ execution engine that replaces parts of Spark's JVM execution for SQL and DataFrame operations, typically giving multiples on scans, joins, and aggregations. It does not accelerate Python or Scala UDFs, RDD code, or most machine learning workloads, and unsupported operators fall back to Spark — so verify in the plan rather than assuming.
What are the compute access modes and why do they matter?- Standard (formerly shared) access mode supports multiple users with Unity Catalog row and column security enforced, dedicated (formerly single user) mode is assigned to one principal, and no-isolation clusters bypass those controls. Some features such as certain Scala and RDD APIs behave differently across modes, which is a common surprise during migrations.
How do you control Databricks cost?- Aggressive auto-termination on interactive clusters, job clusters for scheduled work, autoscaling with sensible minimums, spot or preemptible instances for fault-tolerant jobs with on-demand drivers, and Photon where it actually pays. Then attribute spend with cluster tags and the system billing tables, because untagged shared clusters make cost accountability impossible.
Delta Lake
How does Delta Lake provide ACID transactions on object storage?- Every table has a _delta_log directory of ordered JSON commit files describing added and removed Parquet files, checkpointed to Parquet every 10 commits. Readers reconstruct the current file list from the log, so they never see a partial write, and writers use optimistic concurrency: they read a version, write files, and atomically claim the next log number, retrying on conflict.
What is time travel and what limits it?- You can query a past version with VERSION AS OF or TIMESTAMP AS OF as long as both the log entries and the underlying data files still exist. VACUUM removes files no longer referenced, with a default 7-day retention, and log retention defaults to 30 days — so vacuuming aggressively silently destroys your ability to time travel or roll back.
What do OPTIMIZE and Z-ORDER do?- OPTIMIZE compacts many small files into larger ones (targeting around 1 GB) to fix the small files problem. ZORDER BY co-locates related values across multiple columns using a space-filling curve so data skipping prunes more files on those predicates. Z-order helps a small number of high-cardinality filter columns; adding five columns dilutes the benefit.
What is liquid clustering and why prefer it over partitioning?- Liquid clustering replaces both Hive-style partitioning and Z-ordering with an incremental clustering scheme you can change without rewriting the table. It avoids the classic partitioning failures — skewed partition sizes and over-partitioned small directories — and it handles evolving query patterns, since CLUSTER BY columns can be altered later.
How does MERGE work and why is it sometimes slow?- MERGE matches source rows to target rows on a condition and applies update, delete, or insert clauses in one transaction. It is slow when the match condition cannot prune files, forcing a full rewrite; add a partition or clustering predicate to the ON clause. Deletion vectors, on by default for new tables, avoid rewriting whole files for small changes.
What is a concurrent modification conflict and how do you avoid it?- Two writers commit against the same version and touch overlapping files, so optimistic concurrency fails the second with a ConcurrentAppendException. Avoid it by having concurrent writers touch disjoint partitions and including that partition predicate in the operation condition, so Delta can prove no overlap. Serializing writes through one job is the simpler fallback.
Unity Catalog and governance
Explain the Unity Catalog object hierarchy.- Metastore, then catalog, then schema, then tables, views, volumes, functions, and models — the three-level catalog.schema.table namespace replacing the older two-level Hive metastore. One metastore per region is shared across workspaces, which is what enables consistent permissions and lineage rather than per-workspace copies of the same grants.
What is the difference between a managed and an external table?- Unity Catalog manages both the metadata and the storage lifecycle of a managed table, so DROP TABLE deletes the data (recoverable for 7 days through UNDROP). An external table points at a location you control and DROP leaves the files. Managed tables get automatic optimization such as predictive optimization, which is why they are now the default recommendation.
How do storage credentials and external locations work?- A storage credential wraps a cloud identity such as an IAM role or a managed identity; an external location binds that credential to a specific storage path. Grants are then made on the external location instead of handing cloud credentials to users, which is how Unity Catalog removes per-cluster instance profiles and mounted DBFS paths.
How do you implement row and column level security?- Row filters are table-valued functions attached to a table that add a predicate per querying principal, and column masks are functions applied to a column that return a redacted value based on the caller's group membership. Both require Unity Catalog and a compatible access mode; dynamic views using is_account_group_member are the older equivalent.
What does Delta Sharing solve?- An open protocol for sharing live tables across organizations without copying data or requiring the recipient to be on Databricks, since clients exist for pandas, Spark, and others. The provider issues short-lived signed URLs to the underlying files. It replaces the usual pattern of nightly extracts into a shared bucket, which always drifts and cannot be revoked.
Pipelines and workflows
What is Lakeflow Declarative Pipelines (formerly Delta Live Tables)?- A declarative framework where you define streaming tables and materialized views and the platform manages dependency order, incremental processing, retries, and infrastructure. Expectations declare data quality rules that can warn, drop rows, or fail the pipeline. The trade-off is less control over execution details in exchange for far less orchestration code.
What is Auto Loader and why not just list the directory?- Auto Loader (cloudFiles) incrementally ingests new files, tracking what it has seen in a RocksDB checkpoint and optionally using cloud notification queues instead of directory listing. Listing a bucket with millions of files gets slower every day and eventually times out; notification mode stays constant time regardless of directory size.
How does Change Data Feed work and when do you enable it?- Enabling delta.enableChangeDataFeed records row-level inserts, updates, and deletes with a change type, which you read with table_changes(). It lets a downstream job process only what changed instead of diffing full snapshots, which is the clean way to propagate updates through medallion layers. It adds write overhead, so enable it where it is consumed.
How do Databricks Jobs handle dependencies and failures?- A job is a DAG of tasks with dependency edges, per-task retries, and conditional run-if rules, and tasks can share a job cluster to avoid repeated startup. Task values pass small results between tasks. For complex cross-system orchestration teams still front it with Airflow, calling the Jobs API and letting Databricks own the intra-job graph.
How would you design a streaming ingestion pipeline here?- Auto Loader from the landing zone into a bronze streaming table with rescued data enabled, a silver streaming table applying dedupe with a watermark and quality expectations, and gold materialized views for serving. Use trigger AvailableNow if near-real-time is unnecessary, since it gives streaming semantics and checkpointing at batch cost.
Performance and operations
How do you diagnose a slow Databricks job?- Start with the Spark UI stage view for skew and spill, then check the number of files scanned against total in the query profile to confirm data skipping worked. Then look at cluster metrics for whether it was actually CPU-bound or waiting on object storage, and check whether autoscaling spent the run ramping up.
What is predictive optimization?- Databricks automatically runs OPTIMIZE, VACUUM, and statistics collection on Unity Catalog managed tables based on observed query patterns, so teams stop scheduling maintenance jobs and stop forgetting to. It is enabled by default for new accounts; the caveat is that automatic VACUUM interacts with time travel retention, so set the retention you actually need.
How do you handle schema evolution in a Delta pipeline?- mergeSchema on a write adds new columns, and overwriteSchema replaces the schema entirely on an overwrite. In Auto Loader, schemaEvolutionMode and the rescued data column capture unexpected fields rather than failing or dropping them silently. Column mapping enables renames and drops without rewriting files, which is required before those are possible at all.
What is the difference between DBFS and Unity Catalog volumes?- DBFS root and mounts are the legacy path-based access model with workspace-wide credentials and no fine-grained governance; Databricks now discourages storing production data there. Volumes are Unity Catalog objects for non-tabular files with the same grant model as tables, which is the current answer for landing zones, models, and libraries.
How do you run reliable CI/CD for Databricks?- Keep code in Git with Databricks Asset Bundles defining jobs, clusters, and permissions as YAML deployed per environment, so a dev bundle and a prod bundle come from the same source. Run unit tests on pure transformation functions locally or with Databricks Connect, and reserve a small integration suite for a scratch catalog.
From DataLane — tutorials at/blog, practice SQL live in theplayground.