Docker Images: Pin Digests or Watch :latest Ship a Different Binary

CI was green Friday at 16:40. Saturday at 02:10 the nightly orders job died on libpq symbol not found. Nobody had merged anything. The DAG still said image: python:3.11-slim:latest.

Overnight Docker Hub had published a new slim layer. The runner that built Friday already had the old image cached. Prod pulled fresh. Same tag, different bytes, different OpenSSL, and a wheel that no longer loaded. We spent three hours proving the pipeline code was innocent.

That is the :latest incident. It is not exotic. It is the default, and it will page you the first weekend a distro patch lands between your last cached pull and a cold cluster.

I do not let pipeline images float anymore. Pin the digest you tested. Promote that digest. Stop maintaining a Dockerfile per job.

Tags move. Digests do not

A tag is a pointer. python:3.11-slim today is not python:3.11-slim in six weeks. A digest is the content address of the manifest you actually ran.

# Resolve once, then freeze the digest in git.
#   docker pull python:3.11.9-slim-bookworm
#   docker inspect --format='{{index .RepoDigests 0}}' \
#     python:3.11.9-slim-bookworm
FROM python:3.11.9-slim-bookworm@sha256:4b7ce07002c69e8f3d704a9c4d4bf55b87c61beb77306baa7d8be1f17d4da3dc

WORKDIR /app
COPY requirements.lock /app/requirements.lock
RUN pip install --no-cache-dir -r /app/requirements.lock
COPY src /app/src
USER nobody
CMD ["python", "-m", "jobs.nightly_orders"]

The version tag in the name is documentation. The @sha256: is the contract. If the registry ever retags 3.11.9-slim, your FROM line still builds the image you reviewed.

CI and prod must run that same digest. Rebuild-from-Dockerfile in the prod account is a second roll of the dice: different buildkit cache, different apt index, different day. Build once in CI, push the content-addressed image, and have the scheduler pull by digest.

flowchart LR
  ci["CI build"] --> digest["Image digest"]
  digest --> stage["Staging job"]
  stage -->|"same digest"| prod["Prod job"]
  latest[":latest pull"] --> drift["Different bytes"]
  drift --> outage["Saturday symbol error"]

Promote the digest. A second build is a second image, even when the Dockerfile text did not change.

Blessed images, not forty Dockerfiles

The codebase I inherited had a Dockerfile in every job directory. Most of them started from python:3.11-slim and then diverged: one installed gcc “just in case”, one pinned libpq5, one did not. After a year, six jobs could not share a wheelhouse and nobody could say which base was supported.

The fix was three blessed images, built in one repo, scanned in one pipeline:

• pipeline-python — interpreter, lockfile installer, non-root user, CA certs
• pipeline-spark — the Spark distribution the cluster actually runs, not “whatever bitnami/spark is this week”
• pipeline-connect — the Connect worker plus the connector plugins we have tested, which is the same discipline as Kafka Connect in production

Jobs copy code in at runtime or as a thin final layer. They do not each invent a distro. When a CVE lands, we rebuild the blessed image, pin the new digest, and roll the fleet. Forty Dockerfiles cannot do that in an afternoon.

Compose is not prod Kafka

docker-compose.yml with one broker, a Zookeeper or KRaft controller on the same Docker network, and a topic created by a helper container will tell you the producer works. It will not tell you what happens when a replica is out of ISR, when a disk fills, or when an ACL denies the sink user.

I still run Compose for unit-level connector tests. I do not use it as evidence that a Connect image, a replication.factor=1 topic, and an in-process Schema Registry will survive the Monday traffic spike. The production Kafka questions — converters, DLQs, task state — live in that Connect post, not in a laptop YAML file that restarts clean every compose down.

The same lie shows up for Postgres and Redis Compose stacks. They are fixtures. Treat them as fixtures.

What I put in the job spec

The Airflow or Kubernetes object names the digest, not a floating tag. The deploy log echoes it. Rollback is previous_digest, not “whatever :stable means tonight.”

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/pipeline-python@sha256:4b7ce07002c69e8f3d704a9c4d4bf55b87c61beb77306baa7d8be1f17d4da3dc

If the platform only accepts a tag, the tag is immutable (2026-08-30-a3f1) and CI fails if that tag is ever moved. Mutable tags are :latest with better branding.

Pitfalls

Pinning the version tag and calling it done. python:3.11.9-slim-bookworm can still be retagged. Digest or it is not pinned.

Rebuilding in prod to “pick up patches.” You pick up an untested rootfs. Patch the blessed image in CI, then promote.

A Dockerfile per DAG. Drift is guaranteed. Centralize the base, keep jobs thin.

Trusting Compose Kafka for delivery guarantees. One broker cannot teach you ISR, ACLs, or a real DLQ path.

Leaving latest on a base “just for local.” Someone will copy the compose service into a job spec. Make the local file pin too.

What this means for your pipelines

Image drift is a data incident that looks like a compiler error. The job code did not change. The bytes under it did. Pin the digest you tested, promote that digest from CI to prod, and keep the number of root images small enough that a CVE has one rebuild target.

Compose stays on the laptop. Kafka, Connect, and the warehouse do not. If you cannot point at a sha256 and say “that is what ran Saturday,” you cannot explain Saturday.