All stacks · Orchestration
Visual flow-based ingestion, routing, and edge collection.
Apache NiFiFlowFiles pile up when a downstream processor stalls. Provenance has its own disk. Set queue limits, watch both repositories, and do not use NiFi as a Kafka bus.
Apache NiFi is the visual, flow-based engine still used for edge collection, protocol translation, and “get the files off the plant network.” It is not a warehouse orchestrator and it is a poor dbt replacement — it is a router with provenance.
Treat it as ingestion and routing: back pressure, guaranteed delivery, and a graph you can explain to an ops team that does not want to write Python.
Yes, especially in industrial, government, and hybrid shops that need protocol conversion and guaranteed file movement. Greenfield SaaS teams usually pick Airbyte, Fivetran, or Kafka Connect instead.
Connect for Kafka-centric CDC and well-known source/sink pairs. NiFi when you have FTP, sensors, weird protocols, or operators who live in a canvas. They can coexist: NiFi lands files, Connect or a producer takes it from there.
Unbounded queues, tiny FlowFiles with huge provenance, and processors that pull entire files into heap. Bound the queues, batch, and treat provenance retention like a disk budget.
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.