The AWS Data Engineering Stack: Which Service Does What (and What to Skip)

AWS has more than a dozen data services with overlapping names and mandates. Here is the map I wish someone had given me — what each service is actually for, and the combinations that show up in healthy production platforms.

You do not need the whole map. You need S3, a catalog, a transform engine, a serve path, and something that runs on a schedule. The rest is a reason, not a default.

flowchart LR
  src[Sources] --> ingest[Kinesis / DMS / Glue]
  ingest --> s3[S3 data lake]
  s3 --> xform[Glue / EMR / Lambda]
  xform --> serve[Athena / Redshift]

The foundation: S3

Everything on AWS data engineering revolves around S3. It is the lake, the staging area, the backup target, and the hand-off point between every other service. Two habits from day one:

Partition your prefixes (s3://lake/orders/dt=2026-08-24/) so query engines can prune.
Use lifecycle policies to move cold prefixes to cheaper tiers. Do not lifecycle-expire objects that Iceberg/Delta metadata still reference.

Add three more before the lake is “real”:

Bucket layout in git. s3://env-lake/{layer}/{domain}/… with env in the name or prefix. Prod and staging sharing a bucket is how a backfill wipes a dashboard.
Encryption and keys you can rotate. SSE-S3 is fine to start. SSE-KMS needs the same IAM thought as the data.
No “temporary” dumps at the bucket root. Temporary is forever.

S3 is not a database. ACID comes from a table format (Iceberg / Hive / Delta) plus a catalog. Raw JSON drops are bronze, not gold.

Ingestion

Kinesis Data Streams — real-time event streaming, AWS’s managed Kafka-shaped service. Choose it when you need seconds-level latency and you are all-in on AWS. Shards, not partitions; IAM, not SASL. Firehose is the lazy path into S3 (good) and the way people skip a schema (bad).

MSK (Managed Kafka) — when you want the Kafka ecosystem (Connect, consumer groups, replay habits). See Kafka vs Kinesis for the split. Do not run both for the same events.

DMS (Database Migration Service) — despite the name, it is the standard tool for continuous CDC from operational databases into S3 or Redshift. Full-load + CDC. Watch LOBs, timezone columns, and “we paused DMS for a weekend.” Treat the S3 target as bronze. Dedup in silver.

Glue jobs / Glue streaming — ingest when the source is already “a job” (JDBC, some SaaS connectors). Fine. Do not use Glue Python Shell as a hidden application server.

Lambda — perfect for small event-driven ingestion (file lands in S3 → validate → route). Resist writing whole pipelines in it; the 15-minute limit will find you. Also: 6 MB sync payloads, cold starts, and a concurrency bill that looks like a DDoS you paid for.

AppFlow / partner connectors — SaaS → S3 when you refuse to maintain a token script. Good for CRM extracts. Bad as the only transform layer.

# Lambda: route and record a pointer — not "run Spark"
def handler(event, context):
    for rec in event.get("Records", []):
        bucket = rec["s3"]["bucket"]["name"]
        key = rec["s3"]["object"]["key"]
        if not key.endswith(".json"):
            continue
        # validate header, write a quarantine or a catalog update
        ...

Transformation

Glue (Spark + catalog). Serverless Spark plus a metadata catalog. The catalog part is non-negotiable (Athena and EMR both use it). The Spark part is convenient but pricier per compute-hour than EMR for heavy, sustained jobs. Glue also means DPU math, cold start, and a job bookmark story you must test.

EMR (or EMR Serverless). Managed clusters for Spark / Trino / Flink at scale. More ops than Glue (or less, with Serverless). Meaningfully cheaper for large sustained workloads, especially on spot, if someone owns the cluster policy. EMR is not a failure. It is a reason: “this job is too big / too constant for Glue.”

Athena. Serverless SQL over S3, priced per data scanned — same instinct as BigQuery. Ideal for ad-hoc exploration and light scheduled transforms over well-partitioned Parquet / Iceberg. Terrible as an unpartitioned SELECT * factory. CTAS is a valid silver builder for modest volumes; it is not a substitute for Spark on a 20 TB daily MERGE.

Lambda + pandas. Allowed for small dimensions. Not allowed for “the orders fact, but we didn’t want Glue.”

-- Athena: prune or pay
select count(*)
from analytics.orders
where dt = '2026-08-24';

If dt is not a partition (or Iceberg hidden partition), that query is a scan event. Same lesson as the BigQuery partitioning guide, different console.

Serving

Athena — analysts, notebooks, light dashboards, “does this partition exist.” Put a workgroup per team with a bytes scanned cutoff. One shared workgroup with no limit is a cost incident.

Redshift (RA3 / Serverless). The warehouse. Competitive again when you need governed concurrency, materialized result habits, or BI that should not dry-run 400 TB. Spectrum / federated query reads S3 so you do not load every bronze file. Do not default to Redshift because 2016 architecture slides said so. Default to S3 + Athena/Iceberg; add Redshift when a named BI workload outgrows scan-tax SQL.

OpenSearch — full-text / log search. Not your warehouse. Not your lake.

API Gateway + Athena/Redshift — possible. Cache it. Do not expose ad-hoc SQL to the internet and call it a platform.

Orchestration

MWAA — managed Airflow. The default if the team knows Airflow. Version pins are real; read them. See Airflow vs Dagster vs Prefect.

Step Functions — serverless state machines. Great for event-driven, Lambda-heavy flows; awkward for classic daily batch with 80 dbt models. Use it to fan out file arrivals, not to reimplement Airflow.

Glue Workflows / EventBridge Scheduler — fine for a handful of jobs. The day you need backfills and a UI your intern understands, you wanted MWAA (or another orchestrator).

SageMaker / Databricks on AWS — valid transform/serve paths when that is the team. They still land on S3. They do not replace the catalog conversation.

The Glue Data Catalog is the contract

Athena, EMR, Redshift Spectrum, and many Glue jobs agree on names through the catalog (or through Iceberg catalogs that Glue can participate in). If every team registers orders in a different database with a different dt type, you do not have a lake. You have a junk drawer with IAM.

Habits:

One catalog database per domain (sales, billing), not per person.
Tables point at prefixes, not at a single file that someone will overwrite.
Crawlers are a bootstrap, not a religion. Prefer explicit table DDL / Iceberg commits for gold.
Lake Formation on top when you need column-level grants across engines. Lake Formation first, before you have tables, is how onboarding takes a month.

-- Explicit gold beats a crawler surprise
create external table sales.orders_gold (
  order_id bigint,
  amount decimal(12, 2)
)
partitioned by (dt string)
stored as parquet
location 's3://prod-lake/gold/sales/orders/';

Iceberg tables replace a lot of this Hive DDL. The rule stays: one writer, one name, catalog in git.

A reference architecture that works

For a mid-size platform, this combination covers most needs without exotic services:

CDC from Postgres/MySQL via DMS into S3 (bronze)
Glue catalog over everything; Glue or EMR Spark builds silver/gold Parquet or Iceberg
Athena for exploration; Redshift only if BI concurrency or a named workload requires it
MWAA (or Step Functions for the event-shaped bits) orchestrating it all

OLTP --DMS--> s3://.../bronze/...
                | Glue/EMR MERGE
                v
         s3://.../silver|gold/   + Glue/Iceberg catalog
                |                    |
           Athena workgroups    Redshift Spectrum / unload
                |
              MWAA schedule

Streaming variant: producers → Kinesis or MSK → Firehose or a Spark/Flink job → the same bronze prefix. Do not invent a second lake for “real-time.”

flowchart LR
  land[S3 bronze prefix] --> ddl[Glue catalog table]
  ddl --> compute[Athena or Glue Spark]
  compute --> gold[S3 gold / Iceberg]
  gold --> serve[Athena workgroup or Redshift Spectrum]

Files, then a name, then compute, then a warehouse that reads the same prefix. Skip a box and you have a junk drawer.

-- Athena: one bronze day into a partitioned silver prefix.
-- Prune on dt. Do not CTAS the whole lake "to be safe."
create table sales.orders_silver
with (
  format = 'PARQUET',
  parquet_compression = 'SNAPPY',
  external_location = 's3://prod-lake/silver/sales/orders/',
  partitioned_by = ARRAY['dt']
) as
select
  order_id,
  customer_id,
  amount,
  dt
from sales.orders_bronze
where dt = '2026-08-30';

Register the table in the same catalog database you already use for gold. A second sales_tmp database is how two teams query two different orders.

Service cheat sheet

Need	Reach for	Skip at first
Files / lake	S3	EFS as a lake
CDC from a database	DMS	Custom binlog-on-EC2
Batch Spark, light	Glue	EMR “because Spark”
Batch Spark, heavy/sustained	EMR / EMR Serverless	Glue jobs stacked 40-wide
SQL on the lake	Athena	Redshift for every explore
Concurrent BI warehouse	Redshift	Athena-only for 200 dashboards
Seconds-level events	Kinesis or MSK	Polling S3 every 10s
Small file routing	Lambda	Lambda-as-Spark
Schedule / retries	MWAA	Cron on a laptop
Event workflow	Step Functions	Step Functions for dbt Cloud

What to skip (at first)

Lake Formation as day-one gated community. Grow into it when two engines and a compliance review demand it.
Data Pipeline (legacy). If you still have it, plan a move.
OpenSearch for analytics unless you need full-text.
Redshift until a workload Athena/Iceberg cannot serve.
EMR until Glue’s bill or limits are a measured problem.
Kinesis + MSK + a third bus for one event type.
Macie / every security SKU as a substitute for bucket policies and no public ACLs. Do the boring IAM first.

Start with the boring five: S3, DMS, Glue, Athena, MWAA — you can build a serious platform with just those.

Cost habits (no fantasy unit prices)

Athena: workgroups, partition discipline, columnar formats. Scan is the bill.
Glue: DPUs × time. Idle bookmarks and tiny files both hurt.
EMR: spot + scale-down, or Serverless with a max. A cluster left at 20 nodes over the weekend is the invoice.
Kinesis: shards you provisioned and forgot. On-demand exists; measure.
S3: storage is usually not the crisis. GET/LIST on a million tiny files is.
Redshift: pause/resume or Serverless; RA3 managed storage is not an excuse to load bronze JSON.

Put one weekly review on Athena bytes, Glue job time, and any EMR/Redshift that did not scale to zero. Same instinct as a Snowflake Monday review — different system tables.

Pitfalls

Unpartitioned JSON in Athena. You will learn what a scan costs.
Crawlers as schema management. Gold types drift.
DMS to Redshift as the warehouse with no lake. Replay dies.
Lambda chaining until you have a distributed monolith with a 15-minute ceiling.
One IAM role for all Glue jobs. Least privilege is a table prefix, not a feeling.
Lifecycle expire on Iceberg/Delta data paths.
MSK and Kinesis for the same topic “in case.”
Redshift COPY from the internet every night instead of S3 in the same region.

Decision rules

New platform, AWS-only, SQL + batch → S3 + Glue catalog + Athena + MWAA. Add Glue Spark when SQL CTAS is not enough.
Sustained heavy Spark → EMR (reason in the ticket: hours, data volume, libraries).
True streams (seconds, many consumers) → Kinesis if AWS-native and simple; MSK if Kafka ecosystem. See Kafka vs Kinesis.
BI concurrency / warehouse features Athena cannot give → Redshift, still fed from S3.
SaaS extracts → AppFlow or a Glue/Lambda job, bronze first.
Already on Databricks or Snowflake on AWS → keep S3 as the lake; do not copy the lake into a second branded storage without a writer rule.

If you cannot name the bronze prefix and the catalog database, you are not ready for EMR or Redshift. You are ready for a bucket policy and a table DDL.

FAQ

Do I need Redshift on day one? No. S3 + catalog + Athena covers exploration and modest scheduled SQL. Add Redshift when a named BI workload outgrows scan-tax queries or needs warehouse concurrency Athena will not give you.

Glue or EMR for Spark? Glue until the job is too big or too constant for DPU math. EMR when you can name the hours, the volume, and who owns the cluster policy. “Because Spark” is not a reason.

Can I skip the Glue Data Catalog? Not if Athena, EMR, and Spectrum are supposed to agree on names. A prefix with no table is a folder. Two teams registering orders in different databases is a junk drawer.

Is Lambda a pipeline? It is a router. File landed → validate → drop a pointer. Whole facts in pandas inside 15 minutes is how you discover the limit in production.

Kinesis and MSK for the same events? No. Pick one log of record. Dual-write “in case” is two replay runbooks and no source of truth.

When do I turn on Lake Formation? When two engines and a compliance review need column-level grants. Day-one Lake Formation, before you have tables, is how onboarding takes a month.

What this means for data engineers

S3 is the system of record for files. The catalog is the system of record for names. Glue and Athena are the default compute. Everything else is a measured exception.

Skip the services that exist to close a slide. Run the Monday review on scans, DPUs, and idle clusters. The AWS data map is wide on purpose; your platform should be narrow on purpose.