S3 Patterns for Data Lakes cheat sheet
Prefix layout, partitioning, file sizing, lifecycle rules, and the CLI commands for operating a lake on S3.
Layout
s3://lake/bronze/orders/ingest_date=2026-08-01/part-0000.parquet- Hive-style key=value prefixes let Athena, Spark, and Glue prune without a full listing.
s3://lake/{bronze,silver,gold}/<domain>/<table>/- Layer, then domain, then table. Bucket-per-layer also works and simplifies coarse IAM boundaries.
Partition by ingest date, not by high-cardinality IDs- customer_id partitioning creates millions of tiny prefixes and makes every query slow to plan.
One partition column is usually enough- Multi-level partitioning multiplies small files. Add a second level only when queries always filter on both.
File sizing
Target 128–512 MB per Parquet file- The single highest-impact lake tuning decision. Thousands of small files dominate query time in scan overhead.
df.repartition(8).write.parquet(path)- Control output file count explicitly. Spark's default partitioning rarely matches a sensible file size.
Parquet with snappy or zstd- Columnar plus compression typically cuts scanned bytes ten-fold versus JSON, which is pure Athena savings.
Compaction job on the bronze layer- Streaming ingestion always produces small files. Schedule compaction rather than hoping it resolves itself.
CLI operations
aws s3 ls s3://lake/bronze/orders/ --recursive --human-readable --summarize- Object count and total size — the fastest small-file diagnosis available.
aws s3 sync ./local s3://lake/staging/ --exclude "*" --include "*.parquet"- Sync with filters. Prefer sync over cp for reruns; it skips unchanged objects.
aws s3api list-objects-v2 --bucket lake --prefix bronze/ --query 'length(Contents)'- Precise counting without transferring data, useful in validation scripts.
aws s3 cp s3://lake/... - | head -c 2000- Peek at an object's head without downloading the whole file.
aws s3 rm s3://lake/tmp/ --recursive --dryrun- Always dry-run first. There is no recycle bin unless versioning is enabled.
Lifecycle and storage classes
{ "Rules": [{ "ID": "bronze-tiering", "Filter": {"Prefix": "bronze/"}, "Transitions": [{"Days": 30, "StorageClass": "INTELLIGENT_TIERING"}], "Expiration": {"Days": 730} }] }- Tier raw data after a month and expire it after two years. Storage growth is the quiet half of lake cost.
AbortIncompleteMultipartUpload after 7 days- Failed multipart uploads bill as storage forever and are invisible in the console. Every bucket needs this rule.
Intelligent-Tiering for unpredictable access- Automatic tiering without retrieval fees. Safer than Glacier for a lake where an old partition may be queried.
NoncurrentVersionExpiration- With versioning on, overwritten objects accumulate silently. Expire noncurrent versions or storage doubles.
Access and security
Block Public Access at the account level- Non-negotiable. Bucket-level settings can be undone by a policy change; account-level cannot.
Condition: {"StringEquals": {"s3:prefix": ["silver/"]}}- Prefix-scoped IAM policies give a team read access to one layer without exposing the whole lake.
aws:SecureTransport false → Deny- A bucket policy that rejects plaintext HTTP. Standard audit requirement, one statement to satisfy.
SSE-KMS with a bucket key- Encryption at rest with an auditable key. Bucket keys cut KMS request costs dramatically on high-object-count lakes.
Access Points for multi-team buckets- Per-team endpoints with their own policies, instead of one bucket policy that grows past the size limit.
Performance
S3 scales per prefix, not per bucket- Thousands of requests per second per prefix. Date partitioning naturally spreads load; a single hot prefix does not.
aws s3 cp with --expected-size for large streams- Lets the CLI choose the right multipart chunk size, avoiding the 10,000-part limit on very large files.
S3 Select and Requester Pays- Niche. For most analytics, Athena or DuckDB over Parquet beats S3 Select on both cost and flexibility.
Same-region compute- Cross-region reads add latency and egress charges. Keep query engines in the bucket's region.
From DataLane — tutorials at/blog, practice SQL live in theplayground.