Apache Kafka has become nearly synonymous with modern data engineering. It's in the architecture diagrams, the job descriptions, and the conference talks. The case for event-driven, streaming architecture is compelling in the abstract: real-time data flow, decoupled systems, the ability to replay events and rebuild state.
The question that gets less attention: for any given financial services organization, which problems actually require streaming, and which problems are being shoehorned into a streaming architecture because streaming is fashionable?
What Kafka Is Actually Solving
Kafka solves a specific set of problems well:
High-throughput, low-latency data transport. If your system needs to move millions of events per second with sub-second latency, Kafka is designed for this. Transaction processing at scale, real-time fraud detection, trading systems that need to process market data immediately — these are the use cases Kafka was built for.
Decoupled event-driven architecture. When multiple downstream consumers need to react to events independently, Kafka's consumer group model provides clean decoupling. The source system publishes events; consumers process them at their own pace without the source needing to know who's consuming.
Event replay and reprocessing. The ability to retain events and replay them — either to rebuild state after a failure or to onboard new consumers against historical data — is a genuine architectural advantage that batch systems don't offer cleanly.
Where It Doesn't Fit
The honest assessment for most Turkish financial services organizations: statutory reporting doesn't need Kafka. FATCA submissions happen monthly. CRS is annual. HAYMER reporting follows defined cycles. The data quality and lineage requirements for these submissions are stringent, but the latency requirement is measured in days, not milliseconds.
Introducing Kafka into a regulatory reporting pipeline for these use cases adds operational complexity — consumer group management, offset handling, schema registry, exactly-once semantics — without addressing any of the actual constraints. The compliance team doesn't need the fund manager's policy update to appear in the submission data in real-time. They need it to appear accurately, with full lineage, before the submission deadline.
Where It Does Fit in Financial Services
There are genuine real-time requirements in financial services where streaming architecture earns its complexity:
Fraud and anomaly detection. A payment that should be flagged needs to be flagged before it clears, not in the next batch run. The latency requirement is real, and it requires streaming infrastructure to meet it.
Customer-facing real-time operations. Mobile banking applications, payment status updates, balance displays — customers expect these to reflect current state, not yesterday's batch. The user experience requirement drives a genuine real-time need.
Risk monitoring. In insurance and pension, certain risk metrics need to be updated as events occur — a large policy change that affects the aggregate risk position should be visible to risk management before end of day, not at the end of the month.
Cross-system event coordination. When a payment in one system needs to trigger an immediate update in another, batch integration creates a latency window that may not be acceptable for operational reasons.
The pattern: real-time is justified when the latency of batch creates a specific, measurable operational problem. Not when streaming is the modern approach.
Operational Reality
Kafka is operationally non-trivial. It requires expertise to run reliably — broker configuration, replication factor, consumer group management, offset commit strategies, schema evolution. Organizations that adopt it without building or hiring that expertise end up with streaming infrastructure that is fragile, hard to debug, and whose failure modes are unfamiliar to the team.
I've seen organizations adopt Kafka for use cases that a well-structured PostgreSQL queue or a message broker like RabbitMQ would have served adequately, and spend months managing Kafka complexity that wasn't warranted by the actual requirements.
The right question before any streaming adoption: what is the latency requirement, and what is the throughput requirement? If the answer is "minutes or hours" and "thousands of events per day," Kafka is probably not the right tool. If the answer is "sub-second" and "millions of events per hour," the streaming architecture earns its complexity.
The Infrastructure Conversation
For organizations that do have genuine streaming requirements, the infrastructure investment is significant and worth planning for:
Schema registry for managing event schemas across producers and consumers. Dead letter queuing for events that fail processing. Monitoring that covers consumer lag, partition health, and end-to-end latency. Runbooks for common failure scenarios.
None of this is optional if you're running streaming infrastructure in a regulated environment where auditability matters. The event stream is a data asset like any other — it needs to be observable, its schema needs to be documented, and failures need to be detectable and recoverable.
Kafka is a powerful tool for the right problems. The discipline is knowing which problems those are. In regulated financial services, the honest answer for most organizations is that streaming infrastructure serves a subset of use cases — the real-time ones — and batch infrastructure, well-built and properly governed, serves the rest. The tooling conversation should follow from that assessment, not precede it.