Topic
9 posts ·view all posts
AWSS3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
Amazon RedshiftCOMPOUND sort keys and DISTKEY still decide scans on RA3. VACUUM and ANALYZE are not optional. Spectrum SELECT * is an S3 bill. Concurrency scaling will surprise you.
KafkaWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
AWSState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.
AWSLambda is excellent glue and a terrible framework. Here are the limits that shape every design, the idempotency you must build yourself, and the point to stop.
AWSA production comparison of Redshift Serverless and Snowflake: how RPU-seconds differ from credits, where each one bends under concurrency, and what actually breaks in a migration.
AWSAthena bills you for bytes scanned. Here is the partition projection, Parquet layout, CTAS compaction, and workgroup guardrails I use to cut that number by 90 percent.
AWSWhen Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
AWSGlue vs EMR for production Spark: DPU pricing vs instance pricing, cold starts, library control, and the job size where teams switch.