Kinesis Data Streams vs Firehose: Shards, On-Demand, and Landing Streams in S3

Kinesis Data Streams and Kinesis Data Firehose share a brand name and almost nothing else. Streams is a log: ordered shards, consumers with checkpoints, replay within the retention window. Firehose is a pipe: records go in, buffered files come out in S3, and there is no replay because there is no log.

Most teams I have worked with picked Streams by default because it sounds like the serious option, then spent months operating shard scaling for a workload that never had a real-time consumer. The first question is not throughput. It is: does any application read these records in flight? If the answer is “no, we just land them in S3 for batch,” Firehose alone is the answer and you can stop paying the shard tax.

This post covers the shard math that actually bites, when on-demand mode is worth its premium, and the buffering and partitioning settings that decide whether your S3 landing zone is queryable or a million-file swamp. For Kinesis versus Kafka — a different question — see Kafka vs Kinesis.

flowchart LR
  prod[Producers] --> q{Real-time consumer?}
  q -->|yes| kds["Kinesis Data Streams (shards, replay)"]
  q -->|no| fh["Firehose (buffer, deliver)"]
  kds --> app[Consumer app or Flink]
  kds --> fh2[Firehose as one consumer]
  fh --> s3[S3 landing zone]
  fh2 --> s3

Streams is for readers in flight. Firehose is for landing. They compose.

Shard math, and where it actually breaks

A provisioned shard takes 1 MB/s or 1,000 records per second on the write side, and serves 2 MB/s on the read side, shared across all standard consumers. The totals are easy to plan for. What breaks pipelines is the per-shard limit combined with a bad partition key.

Records route to shards by an MD5 hash of the partition key. If 20% of your traffic is one key — one large tenant, one busy device, the literal string "unknown" from a lazy default — that key’s shard hits ProvisionedThroughputExceededException while the other nine shards idle. Adding shards does not fix this. Nothing fixes this except a better key.

My rules for partition keys, learned the expensive way:

• High cardinality first: user_id, device_id, order_id. Never an enum, never a tenant ID with a whale tenant.
• If ordering only matters per entity, key on the entity. Kinesis guarantees order within a shard, so per-key order survives.
• If nothing needs ordering, a random suffix (f"{tenant_id}#{uuid4().hex[:8]}") spreads a hot key at the cost of per-tenant ordering. Write that trade-off down.

Read-side math matters too. 2 MB/s per shard is shared: two consumer applications each get half. Enhanced fan-out gives each registered consumer its own 2 MB/s pipe with ~70 ms latency, for a per-consumer-shard-hour fee. I turn it on when there are two or more consumers or when a p99 latency SLA exists, and not before.

On-demand mode: what you are actually buying

On-demand mode removes shard provisioning. The stream scales to double your previous 30-day peak automatically, and you pay per GB in and out instead of per shard-hour.

What it does not remove: partition-key discipline. On-demand still shards under the hood, and a hot key still throttles. I have watched a team switch to on-demand to “fix” throttling caused by a constant partition key. The invoice changed. The errors did not.

The pricing shape decides it. On-demand costs more per GB at steady, predictable throughput — provisioned shards you keep at 70% utilization are cheaper. On-demand wins when traffic is spiky or unknown: launch days, batch replays into the stream, anything with a 10x peak-to-trough ratio. My default: new streams start on-demand, and once I have 30 days of metrics showing a flat profile, I switch to provisioned and pocket the difference. The switch is one API call in either direction, twice per 24 hours.

Firehose buffering is a freshness contract

Firehose delivers when the buffer fills or the interval expires, whichever comes first: 1–128 MB, 60–900 seconds (for the S3 destination). These two numbers are the entire freshness and file size story of your landing zone.

Small buffers make fresh, tiny files. Tiny files make Athena and Spark slow — this is the same small-file disease covered in the S3 lake layout guide, except Firehose will manufacture it for you 24 hours a day. Large buffers make efficient files that are 15 minutes stale. Pick the staleness the consumer can actually tolerate; it is nearly always more than the 60 seconds people first configure.

The settings I run for a typical events stream, as Terraform:

{
  "extended_s3_configuration": {
    "bucket_arn": "arn:aws:s3:::acme-lake-raw",
    "prefix": "events/source=web/ingest_date=!{partitionKeyFromQuery:ingest_date}/",
    "error_output_prefix": "errors/events/!{firehose:error-output-type}/dt=!{timestamp:yyyy-MM-dd}/",
    "buffering_size": 128,
    "buffering_interval": 300,
    "compression_format": "UNCOMPRESSED",
    "data_format_conversion_configuration": {
      "enabled": true,
      "output_format_configuration": {
        "serializer": { "parquet_ser_de": { "compression": "SNAPPY" } }
      },
      "schema_configuration": {
        "database_name": "raw",
        "table_name": "web_events",
        "region": "us-east-1"
      }
    },
    "dynamic_partitioning_configuration": { "enabled": true },
    "processing_configuration": {
      "enabled": true,
      "processors": [
        {
          "type": "MetadataExtraction",
          "parameters": [
            {
              "parameter_name": "MetadataExtractionQuery",
              "parameter_value": "{ingest_date: .event_time[0:10]}"
            },
            { "parameter_name": "JsonParsingEngine", "parameter_value": "JQ-1.6" }
          ]
        }
      ]
    }
  }
}

Three decisions in there worth stealing. Parquet conversion at delivery time, so the landing zone is columnar from minute one and no “raw JSON to Parquet” compaction job ever exists. Dynamic partitioning extracting a date from the record body, so files land in Hive-style prefixes engines can prune. And an explicit error_output_prefix, because records that fail conversion go somewhere you can see, not into a retry void.

Dynamic partitioning has a cost per GB and a limit on active partitions per delivery stream (500 by default). Partition on date and maybe one low-cardinality field. Partitioning on user_id will hit the limit before lunch.

The pattern that composes: Streams in front, Firehose behind

When you do need in-flight consumers and an S3 archive, do not write your own S3 sink. Point Firehose at the stream as one more consumer. Streams handles fan-out and replay for the applications; Firehose handles batching, Parquet conversion, and delivery for the lake.

Producers write to Streams with explicit failure handling — PutRecords is partial-failure by design and the SDK will not retry the failed subset for you:

import json
import time
import boto3

kinesis = boto3.client("kinesis")
STREAM = "web-events"

def put_batch(records: list[dict], max_retries: int = 4) -> None:
    """PutRecords with retry on the failed subset only."""
    entries = [
        {"Data": json.dumps(r).encode(), "PartitionKey": r["session_id"]}
        for r in records
    ]
    for attempt in range(max_retries + 1):
        resp = kinesis.put_records(StreamName=STREAM, Records=entries)
        if resp["FailedRecordCount"] == 0:
            return
        # Keep only the entries that failed, preserving order.
        entries = [
            e for e, res in zip(entries, resp["Records"])
            if "ErrorCode" in res
        ]
        if attempt == max_retries:
            raise RuntimeError(
                f"{len(entries)} records failed after {max_retries} retries: "
                f"{resp['Records'][0].get('ErrorCode')}"
            )
        time.sleep(min(2 ** attempt * 0.2, 5.0))

That retry-the-subset loop is not optional. Under throttling, PutRecords returns HTTP 200 with FailedRecordCount > 0, and every team eventually ships a producer that silently drops those records because nobody read the response shape.

flowchart TD
  p[Producers] --> kds[Kinesis Data Streams]
  kds --> flink["Flink / consumer app (real-time)"]
  kds --> fh[Firehose]
  fh --> parquet["Parquet conversion + dynamic partitioning"]
  parquet --> s3["s3://acme-lake-raw/events/..."]
  s3 --> athena[Athena and Spark]

One stream, two contracts: milliseconds for apps, minutes for the lake.

Pitfalls

Choosing Streams when nothing reads the stream. If the only consumer is Firehose, delete the stream and use Firehose Direct PUT. You lose replay you were not using and drop a whole bill.

Low-cardinality partition keys. One hot key throttles one shard while the stream looks 20% utilized. CloudWatch WriteProvisionedThroughputExceeded per shard tells the truth; the stream-level graph lies by averaging.

60-second buffers everywhere. Fresh tiny files are a tax on every query forever. Buffer at 300–900 seconds unless a consumer can name the decision that needs sub-five-minute data.

Ignoring the PutRecords partial failure contract. HTTP 200 with failed records inside is the API working as designed. Retry the subset or lose data quietly.

Forgetting retention is the replay window. Streams defaults to 24 hours. If your consumer can be down for a weekend, pay for 7-day retention or accept the gap. Firehose has no replay at all — its retry lives in the error prefix.

Lambda transforms doing heavy work. The Firehose Lambda processor has a 5-minute ceiling and buffers of its own. Enrich lightly (parse, flatten, tag). Joins and lookups belong downstream in the warehouse or lakehouse.

FAQ

Can Firehose deliver to Snowflake or Redshift directly? Yes — Firehose has native Snowflake (Snowpipe Streaming) and Redshift destinations. I still land S3 in parallel for the replay and audit trail; a landing zone you own outlives any single warehouse decision.

How do I pick between Kinesis and MSK/Kafka? Different post — Kafka vs Kinesis — but the short version: Kinesis for AWS-native teams that want zero brokers, Kafka when you need the ecosystem, longer retention, or multi-cloud portability.

Does on-demand mode eliminate throttling? No. It scales aggregate capacity, but hot partition keys still map to individual shards with individual limits. On-demand removes capacity planning, not key design.

What about Kinesis Data Analytics / Managed Flink? Managed Flink is the right home for stateful stream processing — windows, joins, aggregations — reading from Streams. If your “processing” is stateless reshaping, the Firehose Lambda processor is cheaper and simpler. If it is stateless and batch-tolerant, do it in the warehouse after landing.

One Firehose per event type, or one big one? Per logical table. A mixed-schema Firehose defeats Parquet conversion (one schema per stream) and forces downstream splitting. The delivery stream is free-ish; the untangling job is not.

What this means for data engineers

Ask the consumer question first. No in-flight reader means Firehose Direct PUT, Parquet conversion on, dynamic partitioning by date, buffers at five minutes, done. That configuration is 90% of “streaming into S3” needs and it is nearly zero-ops.

When real-time consumers exist, run Streams with a high-cardinality partition key, start on-demand, and let 30 days of metrics decide whether provisioned shards are worth the switch. Put Firehose behind the stream for the archive instead of writing an S3 sink.

Either way, the S3 side is the part your future self queries. Spend your care on file sizes and prefixes — the lake layout guide is the other half of this post.