Kafka12 min read
Kafka vs Amazon Kinesis: Control vs Less Ops
When to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
- kafka
- aws
- streaming
All stacks · Streaming & CDC
Kafka
Event streaming, topics, Kafka Connect, and real-time pipelines.
Kafka
Kafka11 min read
Kafka Fundamentals: Topics, Partitions, and Consumer Groups Explained
The mental model every data engineer needs before touching Kafka: how topics, partitions, offsets, and consumer groups actually work together.
- kafka
- streaming
- tutorials
Kafka12 min read
Kafka Connect in Production: Converters, Dead Letter Queues, and Worker Sizing
The Connect settings that decide whether your connectors survive contact with real data: converter choice, error tolerance with a DLQ, SMT limits, and how to size and scale workers.
- kafka
- streaming
Kafka13 min read
Kafka Schema Registry: Evolving Event Schemas Without Breaking Consumers
Avro versus Protobuf, what each compatibility mode really allows, and the CI gate that stops a breaking schema change before it reaches a topic your consumers cannot read.
- kafka
- streaming
Kafka12 min read
Kafka Exactly-Once Semantics: What Transactions Actually Guarantee
Idempotent producers, transactions, and read_committed give you exactly-once inside Kafka. The moment you write to a database, you still need an idempotent sink. Here is where the boundary sits.
- kafka
- streaming
Kafka12 min read
Kafka Consumer Group Rebalancing: Why Your Consumers Stall and How to Stop It
Rebalances are the most common cause of Kafka consumer lag spikes. Here is the protocol, the four timeouts that matter, and the cooperative sticky config I ship by default.
- kafka
- streaming
Related reading
Docker4 min read
Docker Images: Pin Digests or Watch :latest Ship a Different Binary
A Friday CI green on python:3.11-slim:latest became a Saturday outage when prod pulled a new digest. Pin the bytes, promote the same image, and stop treating Compose as Kafka.
- docker
- ci-cd
Apache NiFi5 min read
NiFi Backpressure and Provenance: Unbounded Queues, Disk Fill, and When Kafka Connect Wins
FlowFiles pile up when a downstream processor stalls. Provenance has its own disk. Set queue limits, watch both repositories, and do not use NiFi as a Kafka bus.
- nifi
- ingestion
Apache Flink5 min read
Flink Event Time and Watermarks: Checkpoints, Savepoints, and the Sink Key You Still Need
Processing time lies after a restart. Watermarks drop late events on purpose. Checkpoints recover; savepoints migrate. Exactly-once to Kafka does not upsert Postgres.
- flink
- streaming
Apache Pulsar4 min read
Pulsar Multi-Tenant Messaging: Brokers, BookKeeper, and When Not to Leave Kafka
Brokers are stateless. BookKeeper holds the log. Pulsar wins on tenant isolation and geo. A healthy Kafka cluster is not a fashion problem.
- pulsar
- streaming
Debezium14 min read
Debezium on Postgres: Replication Slots, Snapshots, and Applying Changes Idempotently
A production walkthrough of Postgres CDC with Debezium: how the replication slot fills your disk, how to snapshot a large table without locking it, and how to apply changes so replays are safe.
- debezium
- kafka
- cdc
About Kafka
Kafka is the backbone of event-driven data platforms: a distributed, replayable log that decouples producers from consumers and makes real-time pipelines possible. It is also a system where defaults are rarely what production needs — acks, retention, partitioning, and consumer group behavior all reward understanding.
The guides here cover Kafka from a data engineer's seat: designing topics and keys, getting exactly-once guarantees where they exist (and not pretending they exist where they do not), running Kafka Connect for ingestion, and landing streams into warehouses and lakehouses without duplicates.
What you'll learn here
- Topics, partitions, keys, and ordering guarantees you can actually rely on
- Consumer groups, rebalancing, and offset management without lost or duplicate reads
- Delivery semantics: at-least-once vs exactly-once, idempotent producers, transactions
- Kafka Connect and Schema Registry for pipelines that survive schema change
- Streams into Snowflake, Delta, and Iceberg with dedup patterns that hold up
Frequently asked questions
When do I actually need Kafka?
When multiple consumers need the same events, when you need replay, or when sub-minute latency matters. If one system just needs rows moved to a warehouse hourly, batch ingestion is simpler and cheaper. Kafka pays off as an architecture, not as a fancy queue.
How many partitions should a topic have?
Enough for your target consumer parallelism with headroom — partitions are the unit of parallelism, and you cannot reduce them later without pain. For most application topics, 6–12 is a sane start; very high-throughput topics go higher deliberately, not by default.
Does Kafka guarantee ordering?
Only within a partition. Messages with the same key land in the same partition and stay ordered; there is no global order across partitions. Design keys around the entity whose order matters — user_id, order_id — and this is enough.
Is exactly-once real?
Within the Kafka ecosystem, yes: idempotent producers plus transactions give exactly-once from producer through Kafka Streams. End-to-end into external systems, it becomes exactly-once processing with at-least-once delivery — your sink still needs idempotent writes or dedup.
Kafka or Kinesis?
Kinesis if you are all-in on AWS and want zero cluster operations at modest scale. Kafka for ecosystem breadth (Connect, Streams, exactly-once), multi-cloud portability, and high-throughput economics. Managed Kafka (MSK, Confluent) covers the middle ground.
New Kafka posts, straight to your inbox
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.