Kafka vs Amazon Kinesis: Control vs Less Ops

Both products are durable ordered logs. The product question is who runs the log and who can leave AWS without rewriting every producer and consumer.

I have shipped both. The wrong choice is usually not “Kafka is better technology.” It is “we bought a cluster we cannot operate” or “we painted ourselves into Firehose and now Spark on Azure needs the same events.”

Read Kafka fundamentals if partitions and consumer groups are still fuzzy. This post assumes that model and maps it onto Kinesis.

flowchart TD
  q[Need a stream?] --> cloud{AWS-only and simple?}
  cloud -->|yes| kin[Kinesis Data Streams]
  cloud -->|multi-cloud or rich connectors| kf[Kafka / MSK]
  kin --> fh[Firehose / Lambda]
  kf --> conn[Connect / Flink / Spark]

Same job, different vocabulary

Idea	Kafka	Kinesis Data Streams
Named stream	Topic	Stream
Parallel log	Partition	Shard
Cursor	Offset (per group)	Sequence number / iterator / checkpoint
Independent readers	Consumer groups	Apps with their own checkpoints (KCL, Lambda)
Replay	Seek / reset offsets	Get records from an older sequence or TRIM_HORIZON
Ops surface	Brokers, disks, ACLs (or MSK)	Shards, IAM, AWS APIs

Kinesis is not “Kafka with different names.” Retention defaults, fan-out, and how you scale are different enough that a naive port will either over-shard or stall a consumer.

Kafka (self-managed or MSK)

Topics, partitions, consumer groups, offsets, Kafka Connect, and the Flink / Spark / ksqlDB ecosystem. You — or MSK / Confluent Cloud — operate brokers.

What you buy:

• Portable clients. The same confluent-kafka or kafka-python producer runs against MSK, Confluent, or a laptop Compose stack.
• Many independent consumers without inventing a checkpoint store. Groups are first-class.
• Connect for JDBC, S3, Snowflake, Elasticsearch, and the long tail of systems that already ship a connector.
• Replay as a habit. Reset a group, spin a new group.id, dump a compacted changelog into a table.

What you pay for:

• Broker disk, partition count, ISR, and rebalances — even on MSK you still own topic design and consumer lag.
• Cross-AZ traffic and storage if you treat the cluster like infinite cheap disk. Retention is a budget line.
• A person who understands the fundamentals. MSK does not delete that requirement.

MSK is still Kafka. If the argument for Kinesis is “I do not want to learn Kafka,” MSK will not save you. If the argument is “I do not want to patch brokers at 2 a.m.,” MSK is the right Kafka.

Kinesis Data Streams

Shards instead of partitions. AWS IAM instead of Kafka ACLs. Firehose to S3. Lambda as a consumer. The Kinesis Client Library (KCL) checkpoints in DynamoDB.

What you buy:

• No broker fleet. You create a stream, pick on-demand or provisioned shards, and attach IAM.
• Native AWS glue. Firehose → S3 / Redshift / OpenSearch. EventBridge and Lambda triggers. CloudWatch metrics you already graph.
• A smaller conceptual surface for a single pipeline: produce with the AWS SDK, consume with Lambda or KCL, land in S3.

What you pay for:

• Weaker portability. Tomorrow’s Azure Spark shop does not speak Kinesis. You will export to S3 and start over.
• A thinner connector ecosystem. You will write more glue (Lambda → HTTP, custom pollers) that Kafka Connect would have covered.
• Shard math. Each shard has published ingest and read limits. Split/merge is an operation, not a config file you bump on Friday.

Kinesis on-demand removes some of the split/merge toil. It does not give you Kafka Connect or multi-cloud clients.

Scaling and fan-out

Kafka: add partitions (carefully), add consumers up to that count, add a new group when a new sink appears. Multiple groups read the same log. Broker disk does not multiply per group.

Kinesis: each shard has a write cap and a read cap. Classic shared throughput means too many concurrent consumers fight for the same shard reads. Enhanced fan-out gives each consumer its own throughput — and its own bill.

If you need five independent, high-volume readers, Kafka-shaped systems are usually less awkward. If you need one Lambda and one Firehose, Kinesis is enough.

Replay and retention

Kafka retention is a topic config (retention.ms, size, compaction). A new consumer group with auto.offset.reset=earliest replays whatever is still on disk. Compaction keeps latest-per-key for CDC.

Kinesis retention is a stream setting (hours, up to the current AWS maximum). Replay means starting an iterator at TRIM_HORIZON or a timestamp and building a new checkpoint. There is no compacted changelog topic in the Kafka sense.

Both can rewind. Kafka makes rewind a normal ops action. Kinesis makes it a “write a one-off consumer and do not clobber the prod DynamoDB lease table.”

flowchart TD
  need[Need last Tuesday again] --> which{Which product?}
  which -->|Kafka / MSK| grp[New group.id or reset a scratch group]
  grp --> disk[Read whatever retention still has]
  which -->|Kinesis| iter[GetShardIterator TRIM_HORIZON or AT_TIMESTAMP]
  iter --> lease[Checkpoint in a new DynamoDB table]
  lease --> live[Leave the prod KCL table alone]

Kafka rewind is a group offset. Kinesis rewind is a new iterator and a checkpoint you must not share with prod.

# One-off Kinesis replay. New application name = new lease table.
# AT_TIMESTAMP, then process; do not point this at the prod KCL table.
from datetime import datetime, timezone

it = kinesis.get_shard_iterator(
    StreamName="orders",
    ShardId=shard_id,
    ShardIteratorType="AT_TIMESTAMP",
    Timestamp=datetime(2026, 8, 25, 0, 0, tzinfo=timezone.utc),
)["ShardIterator"]

while it:
    page = kinesis.get_records(ShardIterator=it, Limit=1000)
    for rec in page["Records"]:
        upsert_orders([json.loads(rec["Data"])])
    it = page.get("NextShardIterator")
    if not page["Records"]:
        break

GetRecords is at-least-once if you crash mid-batch. The MERGE in upsert_orders is still the contract. Stop the loop when a page is empty; a stale iterator will otherwise spin.

A Kinesis producer that is not a toy

# pip install boto3
import json
import boto3

kinesis = boto3.client("kinesis", region_name="us-east-1")

kinesis.put_record(
    StreamName="orders",
    PartitionKey="customer-42",  # shard mapping, like a Kafka key
    Data=json.dumps({"order_id": 1001, "amount": 79.99}),
)

PartitionKey is the Kafka key analogue. Same customer → same shard → per-key order. Batch with put_records in production. Handle the partial-failure list in the response — Kinesis can accept some records and reject others in one call.

Lambda as a consumer is fine for light transforms. It is a bad warehouse loader if you need multi-minute MERGEs and exactly-once checkpoints. Use KCL or a container that owns the lease table.

Consumers: groups vs leases

Kafka’s cursor is a group offset in the cluster (or an external store if you roll your own). Resetting it is an admin action on that group. Two apps = two group.id values.

Kinesis’s cursor is usually a checkpoint you own:

• Lambda — AWS tracks the iterator for the event-source mapping. Replay is a function of batch window, bisect on error, and “start from beginning” on a new mapping. Easy until you need a second independent reader of the same records at a different pace.
• KCL — leases and checkpoints live in DynamoDB. The application name is the Kafka group.id. Clone a deployment with the same name and they fight for shards.

# Sketch: process a Kinesis batch in Lambda, then let AWS checkpoint.
# Fail the batch if the warehouse write fails — do not swallow.
def handler(event, context):
    records = [json.loads(r["kinesis"]["data"]) for r in event["Records"]]
    upsert_orders(records)  # idempotent MERGE

If upsert_orders raises, Lambda retries the batch. That is at-least-once. The same rule as Kafka: the sink must tolerate duplicates.

Firehose and Connect are not the same job

Kinesis Data Firehose is a managed hose into S3, Redshift, or OpenSearch. Buffering, conversion to Parquet, and delivery retries are AWS’s problem. Replay into a different grain later is your problem — you get what landed in the bucket.

Kafka Connect is a plugin runtime: JDBC, S3, Snowflake, HTTP sinks, and a long tail of vendor connectors. You operate workers (or buy Confluent / MSK Connect). You keep Kafka as the log of record and attach sinks without writing a Lambda per destination.

Rule of thumb: one path into the lake → Firehose is enough. Many destinations, or destinations that are not AWS → Connect (or a Flink/Spark job you already run).

Schema and contracts

Kafka shops usually put a schema registry in front of the topic (Avro / Protobuf / JSON Schema, backward compatibility). Kinesis has no equivalent first-class registry. You get IAM, a blob of bytes, and whatever convention the producers honor.

That is survivable if one team owns produce and consume. It is how a second team ships a renamed field and the warehouse loader writes nulls for a week. Put the contract in the same place you would for Kafka: a schema file in the repo, a CI check, and a dead-letter path. The quality post still applies: data quality checks.

Same pipeline, two implementations

Orders → lake → warehouse MERGE, sketched:

Kinesis: app put_records → stream orders → Firehose prefix s3://lake/orders/dt=... and/or a loader Lambda → MERGE in Redshift / Snowflake (via external tables) / Databricks.

Kafka / MSK: app producer → topic orders → Connect S3 sink or a consumer group warehouse-loader → same MERGE.

The warehouse SQL does not care which log you picked. The ops runbook does: iterator age vs consumer lag, DynamoDB leases vs group offsets, IAM vs ACLs.

Cost, without folklore

Neither product is “free because it is managed.”

• Kinesis: you pay for shards (or on-demand volume), PUT payload, extended retention, and enhanced fan-out. Firehose is another meter. The bill is readable in Cost Explorer if you tag the stream.
• MSK: you pay for brokers, storage, and I/O. Idle partitions still sit on disk. A forgotten 30-day retention on a chatty topic is an invoice, not a feature.
• Self-managed Kafka: the AWS bill is EC2 + EBS + cross-AZ. The real cost is the engineer who owns it.

I do not pick a product from a conference slide that says “Kinesis is cheaper.” I pick it from this stream’s volume, reader count, and who is on-call. Run the AWS calculator for your shard count and the MSK calculator for a three-broker baseline. Then add engineering hours.

Decision rule

Kinesis when all of these are true:

• The rest of the platform is AWS (S3 lake, IAM, Lambda, Firehose).
• You have one or two consumers, not a connector zoo.
• You want AWS to own the log servers.
• You can live without a Kafka-shaped ecosystem (Connect, ksqlDB, every hire’s muscle memory).

Kafka / MSK when any of these are true:

• A second cloud, on-prem Spark, or “we might leave AWS” is a real conversation, not a thought experiment.
• You need many independent consumer groups or a large Connect estate.
• The team already operates Kafka well, or you are buying Confluent / MSK specifically for that API.

Neither when a 5–15 minute batch from a database or object store meets the SLA. See the AWS map in the AWS data engineering stack before you add a stream “because real-time looks good on a diagram.”

When NOT to use each

Do not use Kinesis if you are standardizing on Kafka everywhere else and this is the one AWS-only island. You will maintain two client stacks and two replay runbooks.

Do not use self-managed Kafka if nobody in the team has operated brokers and leadership will not buy MSK or Confluent. You will rediscover disk, JVM, and ISR the hard way.

Do not use either as a request/response bus or a job queue with visibility timeouts. Use a queue or an orchestrator (Airflow / Dagster / Prefect).

Pitfalls

• Treating shards like infinite Kafka partitions. They are capacity units with documented throughput caps. Watch WriteProvisionedThroughputExceeded.
• One Lambda per record at high QPS. Batch, or use a consumer that is not billed per invocation.
• Sharing a KCL application name across two deployments. They steal shards. Same class of bug as sharing a Kafka group.id.
• Firehose as the only consumer, then needing replay into a warehouse with different grain. Firehose is a hose. It is not your replay story.
• “We will just mirror to Kafka later.” Dual-write without a contract is two sources of truth. Pick one log of record.

FAQ

Can I reset the live warehouse consumer group at noon? Not if that group is still loading gold. Spin a scratch group.id, replay into a _fix table, then swap. Resetting the prod group is how you double-count a week.

How do I rewind Kinesis without stealing shards from prod? New iterator, new KCL application name (new DynamoDB table), or a new Lambda event-source mapping. Reusing the prod lease table is the same class of bug as sharing a Kafka group.id.

Is enhanced fan-out the same as a second Kafka consumer group? It gives a reader its own shard throughput. You still own the checkpoint. A Kafka group is a first-class cursor in the cluster. EFO is a billing and throughput feature, not a group coordinator.

Does MSK give me Firehose? No. MSK is Kafka. You get Connect, consumer groups, and the usual clients. S3 landing is a sink you attach, not a checkbox on the stream.

When is Firehose enough? One path into the lake, Parquet conversion you are willing to live with, and no need to replay into a different grain later. The moment a second destination appears, you wanted a log of record plus a consumer — Kafka Connect or a job you already run.

Should staging and prod share a KCL application name? No. They will fight for the same leases. Name the application after the environment and the sink.

Production checklist

• Written decision: Kinesis vs MSK vs self-managed, with the multi-cloud constraint named.
• Partition key / Kafka key is the entity you upsert downstream.
• Retention covers rewind. Document how to run a replay without clobbering the live checkpoint / group.
• Sink is idempotent. At-least-once is the default on both products.
• Alarms on iterator age (Kinesis) or consumer lag (Kafka), not just “5xx from the producer.”
• IAM (Kinesis) or ACLs (Kafka) scoped per app. No shared god role.
• Load test at expected peak and at 2× peak before you call it production. Shard/partition counts are guesses until then.
• An owner for the stream. Managed does not mean unowned.

If the platform is AWS and the stream is a single path into S3, I reach for Kinesis without guilt. If the stream is the company’s event backbone, I want a Kafka-shaped log — on MSK if I am already in AWS. The logo on the console matters less than whether you can replay last Tuesday without opening a ticket to another team.