Redshift Sort Keys, WLM, and Spectrum: RA3 Tuning and the Concurrency Scaling Bill

The BI team turned on concurrency scaling because Monday’s stand-up was “the cluster is busy.” Four weeks later the Redshift bill had an $8,400 line that was not compute of the RA3 nodes we sized. Tableau had issued a few hundred concurrent SELECT * against a Spectrum table over a year of Parquet. Extra clusters spun, scanned S3, and spun down. The dashboard still took 40 seconds because nothing was sorted and nothing was partitioned.

I turned concurrency scaling off for the ETL queue, put the BI workload on its own WLM queue with a cap, and rewrote the Spectrum view to name columns and require dt. The $8,400 dropped. The 40 seconds dropped only after we fixed the sort key on the internal table the dashboard should have been using.

RA3 did not make this a Snowflake problem. The broader choice is in Redshift vs Snowflake. This post is the knobs that still exist after you pick Redshift.

flowchart TD
  q[Query] --> wlm[WLM queue]
  wlm --> local{Local table?}
  local -->|yes| sort[Sort key + dist key]
  local -->|Spectrum| spec[S3 scan]
  spec --> star{SELECT * ?}
  star -->|yes| bill[Bytes scanned]
  wlm --> burst{Queue saturated?}
  burst -->|concurrency scaling on| extra[Billed extra clusters]
  burst -->|off / cap| wait[Wait or timeout]

WLM decides who runs. Sort keys decide how much of a local table you read. Spectrum decides how much S3 you pay for. Concurrency scaling is a separate invoice.

RA3 still has a physics

Managed storage means you do not size DS2 disks. It does not mean every query is a metadata prune. Data is still distributed on a key (or AUTO, or EVEN) and still stored in sort order if you loaded it that way. A missing sort key on a 4 billion row fact is a full slice scan with nicer hardware.

create table analytics.fct_orders (
  order_id      varchar(32)  not null,
  customer_id   varchar(32)  not null,
  order_ts      timestamp    not null,
  amount_usd    numeric(12,2) not null,
  status        varchar(16)  not null
)
diststyle key
distkey (customer_id)
compound sortkey (order_ts, status);

-- Load in sort order when you can. COPY + unsorted region
-- is why VACUUM exists.
copy analytics.fct_orders
from 's3://acme-lake/out/fct_orders/dt=2026-08-21/'
iam_role 'arn:aws:iam::123456789012:role/RedshiftCopy'
format as parquet;

vacuum sort only analytics.fct_orders;
analyze analytics.fct_orders;

DISTKEY (customer_id) is for the join you run every hour against dim_customer. If the common join is something else, you picked a souvenir. EVEN is honest when there is no join key. ALL on a large fact is how you fill a node. AUTO is fine until you measure; then pin what you measured.

COMPOUND sort keys prefix-match. Filter on order_ts and you win. Filter only on status and the second column does not save you. Interleaved sort keys exist and I no longer use them: vacuum cost and planner surprises outran the benefit on every cluster I have owned.

VACUUM and ANALYZE are still jobs

RA3 reclaims deletes better than dense-compute nodes did. It does not keep sort order for you after a week of COPY into the unsorted region. Dashboards that were 2 seconds become 40 when 60% of the table sits unsorted.

VACUUM SORT ONLY on the hot facts after the load. ANALYZE so the planner sees the new stats. I run both in the ETL queue, not as a polite suggestion in a wiki. Skip them and you will “tune WLM” for a statistics problem.

Spectrum SELECT *

Spectrum is Redshift’s external table over S3. You pay for bytes scanned, with partition pruning only if the predicate is on a partition column the DDL declared. SELECT * from a 200-column wide Parquet landing table is an S3 scan dressed as SQL.

create external table spectrum.orders_raw (
  order_id     varchar(32),
  customer_id  varchar(32),
  amount_usd   double precision,
  order_ts     timestamp,
  status       varchar(16)
)
partitioned by (dt date)
stored as parquet
location 's3://acme-lake/landing/orders/';

-- Partition + columns. Anything else is a bill.
select order_id, amount_usd, status
from spectrum.orders_raw
where dt = date '2026-08-21'
  and status = 'paid';

Land with Spectrum. Serve from a local table that has a sort key. I do not point Tableau at spectrum.*. That is how concurrency scaling and Spectrum meet on the same invoice.

Concurrency scaling is a surprise because it works

When a queue is saturated, Redshift can run extra clusters and charge you for them by the second. BI bursts look great. A runaway dashboard refresh looks like a successful scale-out.

I disable concurrency scaling on ETL queues. ETL should wait or fail, not clone the cluster because a merge queued behind itself. For BI I set a WLM concurrency and a query timeout, and I alert on concurrency_scaling usage in the bill and in SVL_CONCURRENCY_SCALING_USAGE. “Make it faster” is a sort key. Extra clusters are a credit card.

Failure modes

AUTO everywhere, never measured. Fine at 100 GB. At 10 TB you own a mystery.

Sort key on a column nobody filters. Decorative.

VACUUM deferred for a month. Unsorted region is the table.

Spectrum as the semantic layer. Bytes × users.

Concurrency scaling as the default. ETL clones itself.

What to do Monday

For each slow dashboard, read STL_SCAN and SVL_QUERY_SUMMARY. If you scanned the fact, fix SORTKEY and VACUUM. If the scan is Spectrum, name columns and require a partition predicate. Turn concurrency scaling off on ETL. Cap it on BI and put a budget alarm on the usage view. RA3 bought you managed storage. It did not buy you a warehouse that sorts itself.