Kafka Schema Registry: Evolving Event Schemas Without Breaking Consumers

The producer team shipped on a Thursday. They renamed amount to amount_cents and multiplied by 100, which was a genuinely correct fix to a long-standing bug. The schema registry accepted it, because the subject was on NONE compatibility — a setting somebody had flipped eighteen months earlier during an incident and never flipped back.

By Friday morning the fraud service was throwing deserialization errors on every record, the warehouse loader had written a day of nulls into a column it still called amount, and the finance mart was reporting revenue at one one-hundredth of reality. Three separate teams spent a day on a change that CI could have rejected in four seconds.

Schema Registry is not a nice-to-have piece of Kafka governance. It is the only thing standing between a producer’s refactor and every consumer that ever read that topic. This post covers how compatibility modes actually behave, the practical differences between Avro and Protobuf when you have to live with the choice for years, and the CI gate that makes the whole thing enforceable rather than aspirational.

What the registry does, and what it does not

Schema Registry stores schemas under subjects and hands back an integer ID. The serializer prepends a magic byte and that four-byte ID to every message payload; the deserializer reads the ID, fetches the writer schema (cached), and decodes.

The default subject naming strategy is TopicNameStrategy: <topic>-value and <topic>-key. That means one schema lineage per topic, which is what you want for a single-event-type topic and actively wrong for a topic carrying several event types. For those, RecordNameStrategy or TopicRecordNameStrategy gives each record type its own compatibility lineage. Choosing TopicNameStrategy for a multi-type topic forces you into a union schema that gets uglier every quarter.

What the registry does not do: it does not validate that the data matches the schema at the broker (unless you enable broker-side schema validation in Confluent Server), it does not stop a producer from registering a new schema at runtime if auto.register.schemas is left on, and it does not know anything about semantics. Renaming amount to amount_cents while changing the unit is compatible in several modes and catastrophic in every one of them.

flowchart LR
  prod["Producer"] --> ser["Serializer"]
  ser --> reg["Schema Registry"]
  reg --> id["Schema id"]
  ser --> msg["magic byte + id + payload"]
  msg --> topic["Topic"]
  topic --> deser["Deserializer"]
  deser --> reg
  deser --> app["Consumer app"]

The ID travels with every record. The schema itself is fetched once and cached.

Compatibility modes, stated precisely

The names are confusing because they describe which direction of reading must keep working, not which side changes. Here is the version I keep in my head.

BACKWARD (the default): a consumer using the new schema can read data written with the previous schema. You may delete a field, and you may add a field that has a default. You may not add a required field. Upgrade order: consumers first, then producers.

BACKWARD_TRANSITIVE: same, but against every previous version, not just the last one. Matters when consumers replay old topic history — which, given Kafka’s whole value proposition, they will.

FORWARD: a consumer on the old schema can read data written with the new schema. You may add a field, and you may delete a field that has a default. Upgrade order: producers first.

FULL: both directions against the previous version. You may only add or remove fields that have defaults.

FULL_TRANSITIVE: both directions against all versions. This is the mode people pick when they want to feel safe, and it makes the schema nearly immutable after a few versions. Choose it only for schemas that genuinely must never move, like a payments ledger event.

NONE: anything goes. The setting that caused the Thursday incident.

My default is BACKWARD for internal event topics and FULL_TRANSITIVE for anything crossing a company or regulatory boundary. BACKWARD fits how deployments actually work: you roll out consumers, verify, then roll out producers. It also matches Kafka’s replay semantics, where a new consumer routinely reads data written six months ago.

Set it per subject, not globally, and set it in code:

# Per-subject override. Global default stays BACKWARD.
curl -X PUT http://schema-registry:8081/config/payments.ledger-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL_TRANSITIVE"}'

Avro: defaults are the whole game

In Avro, whether a change is compatible comes down almost entirely to whether the new field has a default. A field with a default can be absent from the writer’s data and the reader fills it in. A field without one cannot.

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.acme.orders.v1",
  "fields": [
    { "name": "order_id",    "type": "string" },
    { "name": "customer_id", "type": "string" },
    { "name": "amount_cents","type": "long" },
    { "name": "currency",    "type": "string", "default": "USD" },
    {
      "name": "status",
      "type": { "type": "enum", "name": "OrderStatus",
                "symbols": ["PLACED", "PAID", "CANCELLED"],
                "default": "PLACED" },
      "default": "PLACED"
    },
    {
      "name": "coupon_code",
      "type": ["null", "string"],
      "default": null
    },
    { "name": "occurred_at", "type": { "type": "long", "logicalType": "timestamp-millis" } }
  ]
}

Every evolvable field in there carries a default. coupon_code is a union with null first, which is required for the default to be null — Avro takes the default from the first branch of a union, and ["string", "null"] with "default": null is invalid. That ordering trips up nearly everyone on their first nullable field.

The enum has a symbol default too. Without it, a producer emitting a new REFUNDED symbol breaks every older reader; with it, older readers see PLACED instead of throwing. That is a lossy fallback you should choose consciously, because silently mapping refunds to placed orders is its own kind of incident. When an enum is likely to grow, I use a plain string with validation in the application and skip the schema-level enum entirely.

The changes Avro considers breaking, in the order I see them cause outages:

• Adding a required field (no default).
• Renaming a field. Avro matches on name, so a rename is a delete plus an add, and the delete side breaks readers expecting it. Use aliases if you truly must rename.
• Narrowing a type: long to int, double to float, union to non-union.
• Changing a field’s meaning while keeping its type. The registry cannot see this and it is the most damaging change of all.

Widening is safe in the promotion direction Avro allows: int to long, float to double, string to bytes.

Protobuf: field numbers change the calculus

Protobuf identifies fields by number, not name, which makes renames free and deletions safer. In proto3 every scalar field has an implicit default, so the “did you add a default” question mostly evaporates.

syntax = "proto3";
package com.acme.orders.v1;

import "google/protobuf/timestamp.proto";

message OrderPlaced {
  string order_id     = 1;
  string customer_id  = 2;
  int64  amount_cents = 3;
  string currency     = 4;

  // Field 5 was `status_string`, removed in v3. Never reuse the number.
  reserved 5;
  reserved "status_string";

  OrderStatus status = 6;

  // optional gives explicit presence: we can tell "no coupon" from "".
  optional string coupon_code = 7;

  google.protobuf.Timestamp occurred_at = 8;
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0; // required zero value, acts as the fallback
  ORDER_STATUS_PLACED      = 1;
  ORDER_STATUS_PAID        = 2;
  ORDER_STATUS_CANCELLED   = 3;
}

The reserved declarations are not decoration. Reusing field number 5 for a different type after removing the original means an old consumer decodes the new bytes into the old field and gets garbage that does not throw. Reserve every number and name you retire, forever.

Which one to pick? My practical split: Protobuf when the same events are consumed by application services in several languages and gRPC is already in the stack, because the codegen story is better and field numbers make long-term evolution genuinely easier. Avro when the primary consumers are data platforms — Spark, Flink, the lake, Hive-compatible tooling — because Avro schemas map cleanly onto columnar formats and every engine in the lakehouse world already speaks it. JSON Schema exists in the registry too; I use it only for topics where a human debugging with kafka-console-consumer is a primary use case, and I accept the size cost.

Wire it into the producer without auto-registration

The single configuration change that prevents the most damage:

from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

sr = SchemaRegistryClient({"url": "http://schema-registry:8081"})

with open("schemas/order_placed.avsc") as f:
    schema_str = f.read()

serializer = AvroSerializer(
    sr,
    schema_str,
    conf={
        # Production: schemas are registered by CI, never by a running app.
        "auto.register.schemas": False,
        # Look up the id for this exact schema; fail fast if it is not registered.
        "use.latest.version": False,
        "normalize.schemas": True,  # whitespace/ordering changes do not create versions
    },
)

With auto.register.schemas=False, a producer carrying an unregistered schema fails at startup with a clear error instead of quietly minting version 27 of a subject at 3 a.m. Registration becomes a deliberate, reviewed step in the deployment pipeline. I have never regretted this setting; I have repeatedly regretted its absence.

Then gate the registration itself in CI:

#!/usr/bin/env python3
"""CI gate: check every schema against the live registry before merge."""
import sys
from pathlib import Path

from confluent_kafka.schema_registry import Schema, SchemaRegistryClient

SUBJECTS = {
    "schemas/order_placed.avsc": "orders.placed-value",
    "schemas/payment_settled.avsc": "payments.ledger-value",
}


def main(register: bool) -> int:
    sr = SchemaRegistryClient({"url": "http://schema-registry:8081"})
    failed = []
    for path, subject in SUBJECTS.items():
        schema = Schema(Path(path).read_text(encoding="utf-8"), schema_type="AVRO")
        # test_compatibility checks against the subject's configured mode,
        # including all prior versions when the mode is TRANSITIVE.
        if not sr.test_compatibility(subject, schema):
            level = sr.get_compatibility(subject)
            failed.append(f"{path} is not {level}-compatible with {subject}")
            continue
        if register:
            version_id = sr.register_schema(subject, schema)
            print(f"registered {subject} -> id {version_id}")

    for line in failed:
        print(f"INCOMPATIBLE: {line}", file=sys.stderr)
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main(register="--register" in sys.argv))

Run it without --register on every pull request and with --register on merge to main. A breaking change now fails a check with the subject name and compatibility level in the message, which is a four-second feedback loop instead of a cross-team Friday.

flowchart TD
  pr["Pull request changes .avsc"] --> ci["CI: test_compatibility"]
  ci -->|incompatible| block["Fail the check"]
  ci -->|compatible| merge["Merge to main"]
  merge --> reg["Register new version"]
  reg --> deploy["Deploy consumers first"]
  deploy --> prod["Deploy producers"]
  block --> bump["Bump to v2 subject or add a default"]

Compatibility is a CI check, not a runtime discovery. Consumers deploy before producers under BACKWARD.

When you actually need a v2 subject

Some changes are breaking on purpose and no default will save you: the grain changes, a field’s unit changes, one event splits into two. Forcing those through a compatible schema produces a record with amount, amount_cents, and amount_v2 where two are always null, and nobody can tell you which is authoritative.

Ship a new subject and a new topic instead: orders.placed.v2 with com.acme.orders.v2.OrderPlaced. Dual-write both topics for a window sized by how long consumers realistically take to migrate — measure it from your slowest team’s deploy cadence, not from optimism. Track consumer group lag on v1 to know when it is truly dead, then stop the dual write and set v1 retention to expire.

This is the same versioning discipline as a producer-owned data contract, with the registry playing the role the YAML file plays in a batch pipeline. The registry enforces structure; the contract adds freshness, volume, and ownership. Run both.

Pitfalls

Leaving auto.register.schemas=true in production. Any producer can create a new version. The registry becomes a log of what shipped, not a gate on what may ship.

A subject on NONE that nobody remembers setting. Audit GET /config for every subject quarterly. One NONE subject is enough for the Thursday incident.

Adding a field without a default under BACKWARD. It fails at registration, which is fine, and then someone “fixes” it by lowering the compatibility mode, which is not.

Reusing a Protobuf field number. Old readers decode new bytes into the old field without error. Reserve retired numbers and names permanently.

Union ordering in Avro. ["string", "null"] with a null default is invalid. Null goes first for nullable fields, every time.

Deploying producers before consumers under BACKWARD. The mode guarantees new readers can read old data. It says nothing about old readers and new data. Get the order right or your rollout is a partial outage.

Treating a semantic change as compatible. Changing cents to dollars in a long field passes every check and destroys every downstream number. No registry catches this. Code review does.

FAQ

Should I use BACKWARD or FULL?

BACKWARD unless a consumer you do not control cannot be upgraded on your schedule. FULL constrains producers to only add and remove defaulted fields, which is a real cost, and it buys nothing if you always deploy consumers first. Use FULL_TRANSITIVE for externally-consumed or regulated event streams.

Does the registry validate that data matches the schema?

The serializer does, on the client. Brokers do not by default, though Confluent Server can be configured for broker-side schema validation per topic. A producer using a raw byte serializer can write anything to a topic that has a registered schema, and the first consumer to hit it gets the error.

How do I handle a topic with multiple event types?

Use RecordNameStrategy or TopicRecordNameStrategy so each record type gets its own subject and compatibility lineage. With the default TopicNameStrategy you end up with a top-level union that has to be edited every time any event type changes.

What happens to old data when the schema evolves?

Nothing. The bytes on disk are unchanged and still carry their original schema ID. That is precisely why transitive compatibility matters for topics with long retention: a consumer reading from the beginning will encounter every historical schema version.

Can I delete a schema version?

You can soft-delete and, with a permanent delete, hard-delete it. Do not do this on a topic with retained data written under that version, because the deserializer will fail to resolve the ID. Deletion is for cleaning up a version registered by mistake and never used.

How does this interact with the warehouse?

Avro maps cleanly to the columnar formats warehouses ingest, so an additive schema change usually becomes a new nullable column. That still needs a downstream change to be useful, which is why I treat the registry as one half of the story and dbt-side incremental model handling as the other.

What this means for your pipelines

The technical part of schema evolution is small. Add defaults, reserve retired Protobuf numbers, put null first in Avro unions, pick BACKWARD. You could learn all of that in an afternoon. The part that fails in practice is organizational: nothing stops a producer from shipping a breaking change if the only enforcement point is a runtime error in someone else’s service.

So spend your effort on the two settings that move enforcement earlier. Turn off auto.register.schemas so a running application can never invent a schema version, and add the compatibility check to CI so a pull request fails before review rather than after deploy. Those two changes convert schema compatibility from a convention into a constraint, and conventions do not survive a Thursday deadline.

Then be honest about the changes that are genuinely breaking. A unit change, a grain change, an event that should have been two events — these deserve a v2 topic and a dual-write window, not a clever union that keeps the compatibility check green. The registry is very good at telling you whether the bytes will decode and completely blind to whether the numbers still mean the same thing. That second question is still yours.