Streaming Interview Questions cheat sheet
Delivery semantics, watermarks, state, lag, and the Kappa versus Lambda questions streaming interviews actually ask.
Semantics
Explain at-most-once, at-least-once, and exactly-once.- At-most-once can drop events. At-least-once can duplicate them and is the realistic default. Exactly-once processing is achievable inside one engine with transactions and checkpointed offsets; end-to-end it still requires an idempotent sink. 'We have exactly-once' without naming the sink is the answer interviewers fail.
How do you make at-least-once safe in a warehouse sink?- MERGE on a business event_id, or delete-insert by a deterministic partition. Dedup the batch first. Guard updates with event time so a redelivered old record cannot overwrite a newer one. Idempotent writes are cheaper and more debuggable than chasing broker-level exactly-once.
What does Kafka's enable.idempotence actually guarantee?- A single producer session will not write duplicates to a partition even with retries. It does not deduplicate across producer restarts with a new PID unless you also use transactions and a stable transactional.id. Consumers can still see duplicates if they reset offsets.
Why is 'exactly-once' across Kafka, a stream job, and Snowflake still a design problem?- Each hop has its own commit. You can align Kafka offsets with a sink transaction inside one engine. Once you cross a network boundary into a warehouse MERGE, you are back to idempotency. Draw the commit points; do not recite a product feature.
Time and windows
Event time versus processing time — which do you aggregate on?- Event time. Processing-time windows move when the consumer lags, so a daily total changes because your job was slow. That is indefensible in a report. Processing time is acceptable for operational metrics about the pipeline itself.
What is a watermark and what happens to late events?- A watermark is a threshold after which the engine considers a window closed and drops or side-outputs later events. It is the contract that bounds state. A ten-minute watermark means you accepted losing anything later than that in exchange for not growing state forever.
Tumbling versus sliding versus session windows?- Tumbling is fixed and non-overlapping — hourly counts. Sliding overlaps, so each event lands in multiple windows and state grows. Session windows close after a gap of inactivity and are unbounded per key until that gap. Cap session gap and idle state or one noisy key takes the job down.
How do you handle a watermark that is too aggressive?- Late events disappear and downstream totals look low. You either lengthen the watermark, send late events to a side output and merge them in batch, or accept a correction pipeline. Changing the watermark without a backfill plan just moves the error.
Why do out-of-order events happen even with a well-behaved producer?- Multiple partitions, retries, mobile clients with bad clocks, and upstream buffers. Order is per partition per key, not global. Designing as if a topic is a single total order is the usual streaming architecture bug.
State and scaling
Where does stream job state live, and why does that matter for recovery?- Keyed state on the workers, checkpointed to durable storage. Recovery replays from the last checkpoint. If state is larger than memory you need a disk backend (RocksDB) and incremental checkpoints. Full checkpoints of large state will not finish inside the interval.
What limits the parallelism of a streaming job?- The number of source partitions and the cardinality of the key you group by. You cannot have more useful workers than partitions. A skewed key makes one task the bottleneck regardless of cluster size.
How do you diagnose a job whose lag grows during business hours and shrinks at night?- Throughput is just below peak input rate. Scale consumers or reduce per-event work. If one partition's lag grows, you have a hot key. If all grow, you are under-provisioned. CPU high with lag flat is not the same incident as lag high with CPU low.
What is backpressure and what should you do when you see it?- A slow operator forces upstream to slow down. That is the system working. The fix is the slow operator — an external call, a shuffle, or a sink — not disabling backpressure. Disabling it turns a slow job into a crashed one.
Architecture
Lambda versus Kappa — which do you reach for in 2026?- Kappa: one streaming path, replay the log to reprocess. Lambda's batch-plus-speed layers mean two implementations that will drift. Keep Lambda only when a legacy batch system cannot be retired. Micro-batch (5–15 minutes) is the honest middle and often the right product.
When is streaming the wrong tool?- When the consumer looks at the number once a day, when the team cannot staff a stateful job, or when the source cannot provide ordered change data. A well-tested hourly batch with an SLA beats an unattended Flink job. Ask what decision changes with latency.
How do you design a medallion architecture for streams?- Bronze appends raw events immutably. Silver dedups and conforms. Gold aggregates to the serving grain. A logic bug is then a silver rebuild, not a re-ingest from the source. Skipping silver means every gold job reimplements dedup and they drift.
Where does a dead letter queue belong?- On every hop that parses or validates. A poison-pill event must not halt the consumer or be silently skipped. DLQ plus an alert plus a replay path is the minimum. A DLQ nobody reads is just a second place data goes to die.
Kafka-specific
How do you choose a partition count?- It caps consumer parallelism and cannot be reduced later. Start from expected peak throughput divided by per-partition capacity, then add headroom. Increasing partitions later breaks key-to-partition affinity, so do not treat it as free.
What is consumer lag and what SLO do you put on it?- The difference between log-end offset and committed offset, per partition. Alert on age (event time versus now) not just offset count, because a quiet partition with a stuck consumer looks like zero lag in messages. Lag is the freshness promise.
Why do consumer group rebalances stall processing?- Every member stops fetching while partitions are reassigned. A member that fails to join in session.timeout.ms triggers another rebalance. Cooperative sticky assignment plus reasonable max.poll.interval.ms stops the stop-the-world loop. Adding consumers without this config can make lag worse.
Log compaction versus time retention — which for what topic?- Time retention for event streams you replay by time. Compaction for changelog or CDC topics where you need the latest value per key forever. Using compaction on an event stream silently drops history. Using time retention on a state topic silently drops keys that did not change recently.
Operations
What do you monitor besides consumer lag?- Checkpoint duration and failure rate, watermark delay, state size, sink commit latency, and DLQ volume. A job can have zero lag and still be dropping late events or failing to checkpoint. Lag alone is necessary and not sufficient.
How do you replay a day of events without doubling counts?- Reset offsets for that window, and make the sink idempotent on event_id or replace a date partition. Replaying into an append-only table without a dedup key is how replay incidents happen. Dry-run the offset reset; the Kafka CLI requires --execute on purpose.
A replication slot on Postgres is holding WAL. What do you do?- The CDC consumer is down or slow. WAL will grow until the disk fills and the source database stops. Alert on slot lag, page immediately, and drop or advance the slot only with a plan to re-snapshot. This is the most common CDC production incident.
How do you test a streaming job?- Deterministic event-time unit tests with a test harness, plus a staging topic with production-shaped volume. Testing only with live data means you discover watermark and skew bugs in production. Include a late event and a duplicate in the fixture; those are the cases that break people.
From DataLane — tutorials at/blog, practice SQL live in theplayground.