Cutting a Databricks Bill: Job Clusters, Spot, Photon, and Finding the Top Offenders in System Tables

The first Databricks bill I was asked to cut was 96,000 dollars a month. Within eight days it was 61,000, and I had not made a single job run faster. Every dollar came from where the compute ran, what it ran on, and how long it stayed up after finishing.

That ratio has held across every environment since. Query optimization matters, but it is slow work with uneven returns. Compute configuration is fast work with immediate, arithmetic returns, and most teams have not done it because nobody owns the bill. The engineers optimize runtime; finance sees a number; and the gap between them is filled with all-purpose clusters running nightly ETL at four times the necessary rate.

Databricks pricing has two components multiplied together: DBUs, which is Databricks’ charge for the workload type, and the cloud instances underneath, which your cloud provider bills. Both are adjustable, and the adjustments are independent. Most of the savings I find live in the DBU rate, which is the part people assume is fixed.

Here is the order I work in, because it is the order of return per hour of effort.

Find the top offenders before changing anything

Do not tune what you have not measured. Unity Catalog exposes billing and compute metadata as queryable tables, and one query tells you where the money is.

-- Top 25 spend sources over the last 30 days, ranked by list-price DBU cost
WITH usage AS (
  SELECT
    u.usage_date,
    u.sku_name,
    u.usage_quantity                                  AS dbus,
    u.usage_metadata.job_id                           AS job_id,
    u.usage_metadata.cluster_id                       AS cluster_id,
    u.usage_metadata.warehouse_id                     AS warehouse_id
  FROM system.billing.usage u
  WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
    AND u.usage_unit = 'DBU'
),
priced AS (
  SELECT
    us.*,
    us.dbus * p.pricing.default                       AS usd
  FROM usage us
  JOIN system.billing.list_prices p
    ON us.sku_name = p.sku_name
   AND p.price_end_time IS NULL
)
SELECT
  coalesce(job_id, warehouse_id, cluster_id)          AS entity,
  CASE WHEN job_id IS NOT NULL     THEN 'job'
       WHEN warehouse_id IS NOT NULL THEN 'sql_warehouse'
       ELSE 'interactive_cluster' END                 AS entity_type,
  sku_name,
  round(sum(dbus), 1)                                 AS dbus_30d,
  round(sum(usd), 2)                                  AS usd_30d
FROM priced
GROUP BY 1, 2, 3
ORDER BY usd_30d DESC
LIMIT 25;

The distribution is always the same shape. Three to five entities account for half the bill, and there is at least one surprise — usually a SQL warehouse someone left on a 4-hour auto-stop, or a “temporary” all-purpose cluster from a migration that has been running for five months.

Two follow-up queries make this actionable. First, cost per run, which tells you whether a job is expensive because it is heavy or because it runs 400 times a day:

-- Cost per run and runtime trend for one job
SELECT
  r.period_start_time::date        AS d,
  count(*)                         AS runs,
  round(avg(datediff(second, r.period_start_time, r.period_end_time)), 0) AS avg_seconds,
  round(sum(u.usage_quantity), 1)  AS dbus
FROM system.lakeflow.job_run_timeline r
JOIN system.billing.usage u
  ON u.usage_metadata.job_run_id = r.run_id
WHERE r.job_id = '842317'
  AND r.period_start_time >= current_date() - INTERVAL 14 DAYS
GROUP BY 1
ORDER BY 1;

Second, the single highest-yield audit in Databricks: which scheduled work runs on all-purpose compute.

-- Interactive DBUs attributed to jobs: money left on the table
SELECT
  u.usage_metadata.job_id,
  u.sku_name,
  round(sum(u.usage_quantity), 1) AS dbus_30d
FROM system.billing.usage u
WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
  AND u.usage_metadata.job_id IS NOT NULL
  AND u.sku_name ILIKE '%ALL_PURPOSE%'
GROUP BY 1, 2
ORDER BY dbus_30d DESC;

Every row in that result is an immediate saving. Which brings us to the reason why.

Job compute versus all-purpose: the 2-4x you are paying for nothing

All-purpose compute is priced for interactive use: attached notebooks, shared state, a cluster that stays warm between queries. Jobs compute is priced for ephemeral, single-purpose runs. The DBU rate difference is roughly 2x on the standard tiers and larger on some premium SKUs, for identical instances doing identical work.

A nightly pipeline on an all-purpose cluster is therefore paying an interactive premium for a workload with no human attached. Moving it is a configuration change with no code impact:

{
  "name": "silver_orders_nightly",
  "job_clusters": [
    {
      "job_cluster_key": "etl",
      "new_cluster": {
        "spark_version": "15.4.x-scala2.12",
        "node_type_id": "r6gd.2xlarge",
        "driver_node_type_id": "r6gd.2xlarge",
        "num_workers": 8,
        "runtime_engine": "PHOTON",
        "aws_attributes": {
          "availability": "SPOT_WITH_FALLBACK",
          "first_on_demand": 1,
          "spot_bid_price_percent": 100
        },
        "data_security_mode": "SINGLE_USER"
      }
    }
  ],
  "tasks": [
    { "task_key": "build_silver", "job_cluster_key": "etl",
      "notebook_task": { "notebook_path": "/Repos/de/silver/orders" } }
  ]
}

Note first_on_demand: 1. That keeps the driver on-demand while workers run spot, which is the configuration that makes spot safe.

The trade-off is startup time. A job cluster spins up in two to four minutes, so a job that runs for 90 seconds spends more time booting than working. For those, use serverless jobs compute, or group several short tasks into one job that shares a single job_cluster_key so the cluster boots once.

flowchart TD
  work["Scheduled workload"] --> dur{"Runtime per run?"}
  dur -->|"under 2 min"| srv["Serverless or shared job cluster"]
  dur -->|"2 min to hours"| jc["Job compute, spot workers"]
  work --> human{"Human attached?"}
  human -->|yes| ap["All-purpose, aggressive auto-terminate"]
  human -->|no| jc
  jc --> ph{"Photon helps?"}
  ph -->|"SQL, scans, joins"| on["Photon on"]
  ph -->|"Python UDFs"| off["Photon off"]

The decision is mostly about who is waiting and what the code does. Neither is a tuning question.

Photon: measure it per workload, do not assume

Photon is a vectorized C++ execution engine that replaces parts of the JVM query path. It costs roughly 2x the DBU rate, so it only saves money if it makes the job more than twice as fast.

What I have measured across real pipelines:

• Large scans, joins, and aggregations in SQL or DataFrame API: 2.5x to 4x faster. Clear win, often 40 percent cheaper net.
• Delta MERGE-heavy CDC pipelines: 2x to 3x faster. Usually a win, worth verifying.
• Python UDF or pandas UDF pipelines: 1.0x to 1.2x. Photon cannot vectorize Python, falls back to the JVM path for those operators, and you pay double for nothing.
• Streaming with small micro-batches: roughly neutral. The per-batch overhead dominates.

So the rule is simple: Photon on for SQL and native DataFrame transformation jobs, off for anything whose hot path is Python functions. And test rather than assume, because “native DataFrame” gets violated by one udf() call in a helper module. The Spark UI shows which operators ran in Photon; if the plan is full of Row-based fallback nodes, you are paying the multiplier for JVM execution.

Rewriting a Python UDF into native expressions often beats both options, which is the same lesson as removing an unnecessary shuffle: change the work before changing the hardware.

Spot instances, safely

Spot capacity runs 60 to 80 percent below on-demand, and Spark is designed to survive node loss. The reason teams get burned is configuration, not the idea.

My standard settings:

• Driver always on-demand. Losing the driver kills the job and loses all shuffle state. first_on_demand: 1 on AWS, or an explicit on-demand driver node type on Azure and GCP.
• Bid at 100 percent of on-demand with fallback. You are not trying to get a lower price than the market; you are trying to use spare capacity at the spot discount without eviction storms.
• Fallback to on-demand so a capacity crunch delays cost savings rather than the pipeline.
• Not for tight-SLA jobs. A job that must finish by 6 a.m. for a regulatory feed runs on-demand. The savings are not worth the variance.
• Diversify instance types where the platform allows it. Spot availability is per instance type per availability zone, and one narrow choice concentrates your eviction risk.

For long shuffle-heavy jobs, node loss means recomputing that node’s shuffle output, so a heavily evicted cluster can run slower than on-demand at the same total cost. I watch for that pattern in jobs that run over an hour and shuffle terabytes.

Sizing, autoscaling, and auto-termination

Three settings, each with a specific failure mode.

Auto-termination on all-purpose clusters should be 20 minutes, not the 120-minute default. A shared cluster idling from 6 p.m. to 8 a.m. costs 14 hours of interactive-rate DBUs for zero work. This alone was 9 percent of one bill I audited.

Autoscaling is genuinely useful for variable interactive workloads and SQL warehouses. It is often counterproductive for short batch jobs: the cluster starts at minimum workers, Spark schedules tasks against what exists, scale-up takes a minute or two, and a 6-minute job spends a third of its life underprovisioned. For predictable ETL I size fixed and correct, using the partition math from the memory post rather than a guess.

Instance family is the easy structural win. Graviton-class instances on AWS deliver 20 to 40 percent better price-performance for Spark work at the same DBU count, and local-NVMe families (r6gd, i4i) matter for shuffle-heavy jobs because shuffle files land on local disk. Choosing an instance without local SSD for a 2 TB shuffle is a self-inflicted wound.

For SQL warehouses, serverless is usually cheaper in practice despite the higher per-DBU rate, because startup is seconds and idle time is near zero. Classic warehouses on 45-minute auto-stop lose that comparison badly.

Governance so the savings persist

Cost work decays without ownership. Three mechanisms hold it:

Tag everything, enforce with policies. Cluster policies can require tags and cap node types and counts.

{
  "custom_tags.team":        { "type": "allowlist", "values": ["platform", "analytics", "ml"] },
  "custom_tags.cost_center": { "type": "regex", "pattern": "^CC-[0-9]{4}$" },
  "node_type_id":            { "type": "allowlist", "values": ["r6gd.xlarge", "r6gd.2xlarge", "r6gd.4xlarge"] },
  "autotermination_minutes": { "type": "range", "maxValue": 30, "defaultValue": 20 },
  "num_workers":             { "type": "range", "maxValue": 24 },
  "spark_version":           { "type": "regex", "pattern": "^15\\.[0-9]+\\.x-scala2\\.12$" }
}

Tags flow into system.billing.usage.custom_tags, so per-team chargeback becomes a GROUP BY. Without policies, tags are optional, and optional tags are absent tags.

A weekly report, delivered. I schedule the top-offenders query to a table and post the diff versus last week to the team channel. Visible numbers change behavior more reliably than a policy document.

Table maintenance on a schedule. Compaction and vacuum are themselves compute, but skipping them costs more: small files inflate scan time on every downstream read, and untombstoned files inflate storage. Managed tables under Unity Catalog get predictive optimization, which handles most of this without a job.

Pitfalls

Scheduled jobs on all-purpose compute. The most expensive default in Databricks, and it usually happens because someone promoted a notebook to a schedule from the UI. Audit for it monthly.

Photon everywhere by policy. Doubling the DBU rate on a Python UDF pipeline that gets 10 percent faster is a pure loss. Measure per job, and re-measure after code changes.

Spot drivers. Saves a few dollars, occasionally loses an entire multi-hour run. Never worth it.

Autoscaling short jobs. Ramp-up time is a meaningful share of a sub-10-minute run, and you get the underprovisioned portion at full price anyway.

Optimizing DBUs while ignoring instance choice. The cloud instance bill is the larger half for many workloads. Wrong instance family, no local SSD for shuffle, or oversized drivers all show up in cloud cost rather than DBUs, so a DBU-only view misses them.

Chasing query optimization first. It is real work with real returns, but compute configuration is faster and arithmetic. Do the configuration audit in week one, then optimize queries with the time you bought.

FAQ

How much can I realistically cut?

On a bill nobody has audited, 25 to 40 percent within two weeks, almost entirely from job compute migration, auto-termination, spot workers, and turning off idle warehouses. After that, the returns require actual query work.

Is serverless cheaper than job clusters?

Per DBU, no. In total, often yes for short or bursty workloads, because you pay no startup or idle time and no cloud instance bill separately. For a job that runs 40 minutes on a well-sized spot cluster, classic job compute usually still wins.

Should every job get its own cluster?

Every pipeline should, and tasks within a pipeline should share one via job_cluster_key so the boot cost is paid once. Sharing a cluster across unrelated scheduled jobs reintroduces the all-purpose problem with extra coupling.

Do system tables cost anything to query?

You pay the warehouse or cluster compute to run the query, which for these aggregations is cents. There is no separate charge for the system tables themselves, and they are the cheapest cost visibility available.

What about storage cost?

Usually a small fraction of compute, but it grows quietly from tombstoned files and abandoned checkpoints. A scheduled VACUUM with sane retention, as described in the transaction log post, handles the first. Orphaned streaming checkpoints need manual cleanup, and nobody ever does it.

What this means for your pipelines

Run the system tables query before you touch anything. The ranked list will tell you that a handful of entities are most of your bill, and it will contain at least one thing nobody remembers creating. That query takes ten minutes and reframes the entire conversation from “Databricks is expensive” to “these four jobs are expensive.”

Then work the structural levers in order: scheduled work off all-purpose compute, auto-termination to 20 minutes, spot workers with on-demand drivers, Photon where it is measured to help, and instance families with local SSD for shuffle-heavy jobs. None of it touches your code, none of it risks correctness, and together it is usually a third of the bill.

Keep it with tags, cluster policies, and a weekly report someone actually reads. Cost discipline is a maintenance habit, not a project — the same category as table compaction and dependency upgrades. Do the audit once a quarter and the number stops surprising anyone.