Fabric OneLake and Capacity: Shortcuts First, Then Stop Starving Spark

Monday 08:05, the F64 went yellow in Capacity Metrics. The month-end Power BI refresh had burst for 40 minutes. The Spark notebook that builds curated_sales sat in a queue for 47 minutes. Finance closed on Friday numbers. Nobody had “run out of warehouse.” We had one shared meter, and the semantic model won.

I had sold Fabric internally as Synapse with a nicer workspace. That was wrong. The Spark code mostly ports. The capacity model and OneLake do not. If you want the component map and the sequence, I wrote that in Synapse to Fabric and the Azure stack overview. This post is the two decisions that page me: copy versus shortcut, and who gets the CUs.

Shortcuts are not a copy

A OneLake shortcut points at ADLS Gen2, S3, or another OneLake folder. Synapse Spark can keep writing Delta. Fabric notebooks, the SQL analytics endpoint, and Direct Lake can read it. No dual-write window. No nightly reconcile.

flowchart TD
  adls["ADLS Gen2 Delta (Synapse still writes)"] --> sc["OneLake shortcut"]
  sc --> lh["Lakehouse"]
  lh --> spark["Fabric Spark"]
  lh --> pbi["Direct Lake / refresh"]
  spark --> cu["Same F SKU CUs"]
  pbi --> cu
  cu -->|"refresh bursts"| starve["Spark queued"]

Storage is a pointer. Compute is a shared budget. Those are different problems.

Copy when you need V-Order for Direct Lake on a hot table, or when the source cannot stay. Shortcut when the lake is already the system of record. I copied 18 TB once because a vendor slide said “migrate to OneLake.” We spent three weeks proving checksums. A shortcut would have been an afternoon.

Two catches I hit. A Hive-style Parquet folder is Files, not a Lakehouse table, until you define one. Writes through a shortcut skip some Fabric write optimizations, so Direct Lake on externally written Delta can feel like DirectQuery. Test the biggest fact before you promise executives faster tiles.

Capacity units are one pot

Dedicated SQL pools had DWUs you paused. Fabric bills capacity units on an F SKU. Notebooks, pipelines, warehouse queries, and Power BI rendering draw from the same pot. Smoothing averages a burst. Throttling, when you stay over, makes everything slow instead of failing one query. That is why the 08:05 incident looked like “Spark is broken.”

# Sketch: flag hours where interactive (PBI) and Spark overlap.
# Export CU usage from Capacity Metrics, then:
import pandas as pd

usage = pd.read_csv("capacity_cu_by_item_2026-08-24.csv")
usage["ts"] = pd.to_datetime(usage["timestamp_utc"])
hourly = (
    usage.groupby(["ts", "item_kind"], as_index=False)["cu_seconds"]
    .sum()
    .pivot(index="ts", columns="item_kind", values="cu_seconds")
    .fillna(0)
)
overlap = hourly[(hourly.get("SemanticModel", 0) > 0) & (hourly.get("Notebook", 0) > 0)]
print(overlap.sort_values("SemanticModel", ascending=False).head(20))

On that F64, the refresh used enough burst that background Spark waited. Fixes that actually worked: move the close refresh to an F32 reserved for BI, run the notebook after 06:00 UTC, and stop ad-hoc “Run all” on a 40-notebook workspace during close. Resizing the one F64 to F128 would have hidden the neighbor problem for a month and doubled the bill.

Not rebranded Synapse

Spark notebooks and Data Factory pipelines port with path and connection changes. The dedicated SQL pool does not. Fabric Warehouse has no DISTRIBUTION = HASH, no resource classes, and a different T-SQL surface. I still keep the dedicated pool running until Direct Lake covers the reports. Many pools exist only to feed import models. If Direct Lake over a shortcut Lakehouse is enough, retiring the warehouse beats porting it.

-- Dedicated SQL pool: layout is in the DDL. This does not port.
CREATE TABLE dbo.fact_sales
WITH (DISTRIBUTION = HASH(customer_key), CLUSTERED COLUMNSTORE INDEX)
AS SELECT * FROM staging.sales;

-- Fabric Warehouse: the engine owns layout. MERGE is the incremental.
MERGE dbo.fact_sales AS t
USING staging.sales_delta AS s
    ON t.sale_key = s.sale_key
WHEN MATCHED THEN UPDATE SET amount_cents = s.amount_cents
WHEN NOT MATCHED THEN INSERT (sale_key, customer_key, order_date_key, amount_cents)
     VALUES (s.sale_key, s.customer_key, s.order_date_key, s.amount_cents);

I script every dedicated-pool object and grep for DISTRIBUTION, RESOURCE CLASS, identity columns, and materialized views before I give a date. Last inventory: 380 objects, 41 that needed real work. That count is the estimate, not the TB on disk.

Pitfalls

Copying the lake into OneLake because the workspace is empty. Empty is fine. A shortcut fills it. Copies need sync jobs, and sync jobs need a reconcile nobody staffed.

One F SKU for BI and overnight Spark. Smoothing hides a burst until it does not. Separate capacities or a written schedule. Capacity Metrics on day one, not after the first close miss.

Promising Direct Lake speed on shortcut Parquet with no V-Order. Import-mode users will say Fabric is slower. They are not wrong. Rewrite the hot facts from Fabric or keep import.

Treating Fabric Warehouse as Synapse T-SQL. Resource classes are gone. A query that was isolated on a dedicated pool now competes with a refresh. That is a capacity conversation, not a hint in the DDL.

Starting the migration with the warehouse. Spark and Power BI on shortcuts deliver the visible wins. The pool can run a year.

Pausing a shared F SKU overnight. Dev capacities can pause. The one that serves 08:00 refresh cannot. Label them in the module or the portal so a cost-saving toggle does not become a Monday outage.

I also write CU budgets into the same place I write data contracts when a domain owns a capacity: a refresh window, a Spark window, and who gets paged when Metrics goes yellow. Without that, every notebook is “urgent.”

Start with a shortcut and the Capacity Metrics app. Watch two weeks of real refresh plus one Spark job before you resize. Fabric is a workspace you point at data you already have. The outage was never “we needed more OneLake.” It was one refresh and one notebook sharing a meter nobody had drawn on a whiteboard.