Apache Kafka: The Reference Architecture for Event-Driven Data Streaming

TL;DR

In modern IT, data doesn’t rest; it flows. Apache Kafka serves as the central nervous system for these real-time data streams. While cloud services like AWS MSK provide the infrastructure, they often strip users of control over critical configurations and updates. Running Kafka on Kubernetes (via Operator) democratizes this technology: it combines the simplicity of a managed service with the flexibility of open source, allowing you to maintain full control over throughput, latency, and costs.

1. The Architecture Principle: The Distributed Commit Log

Databases store the current state. Kafka stores events that led to this state.

It is based on the principle of the Commit Log: An immutable, append-only sequence of records.

• Decoupling: Producers (e.g., the webshop) and consumers (e.g., the warehouse system) are unaware of each other. They communicate asynchronously via “topics.” If the warehouse system fails, Kafka buffers the orders until it is back online. Nothing is lost.
• High Throughput: By writing sequentially to disk (instead of random I/O), Kafka achieves write speeds that surpass traditional message queues (like RabbitMQ) by a factor of 100.

2. Core Feature: The Strimzi Operator (Infrastructure as Code)

Operating Kafka (with ZooKeeper) was notoriously complex in the past (“JVM hell”). On Kubernetes, this has fundamentally changed thanks to the Strimzi Operator (CNCF Sandbox Project).

Strimzi translates complex Kafka operations into native Kubernetes concepts.

• Kafka as YAML: You define a cluster with 3 lines of YAML (kind: Kafka). The operator spins up the pods, configures the network, and issues TLS certificates.
• Rolling Updates: A version update of Kafka is no longer a nerve-wracking weekend. You change the version in the YAML, and the operator rolls out the brokers one by one without interrupting the data stream.

3. Tiered Storage & KRaft (Future-Proofing)

Kafka is rapidly evolving. Proprietary cloud services often lag behind in versions.

• KRaft (ZooKeeper-less): Kafka is moving away from ZooKeeper. In self-managed environments, you can already use the new, leaner KRaft mode and reduce complexity.
• Tiered Storage: Kafka can be configured to automatically offload “cold” data (older than 2 days) to cost-effective S3 object storage. This allows for infinite “retention time” at a fraction of the cost of fast SSDs.

4. Operating Models Compared: AWS MSK vs. ayedo Managed Kafka

Here, it is decided whether your “nervous system” belongs to you or if you rent it.

Scenario A: AWS MSK (The Black Box with Fees)

MSK takes the work off your hands, but also the insight.

• Opaque Costs: In addition to instance costs, AWS often charges fees for data transfer between brokers (Cross-AZ Traffic). In high-volume systems, this item can double the bill.
• Restricted Config: You do not have full access to server.properties. Tuning parameters for extreme performance or specific security plugins are often locked.
• Forced Upgrades: AWS can force you to update versions when old versions are “deprecated.” This often doesn’t fit into the project schedule.

Scenario B: Kafka with Managed Kubernetes from ayedo

In the ayedo App Catalog, Kafka (powered by Strimzi) is provided.

• Full Control: You have access to every parameter of the JVM and broker config. You decide when to update.
• No Hidden Transfer Costs: Within the cluster, there are no artificial “managed service” fees for traffic. You pay for the compute power.
• Ecosystem Freedom: Use Kafka Connect, Schema Registry, or UI tools (like AKHQ) without relying on AWS-specific, paid “Connect” services.

Technical Comparison of Operating Models

FAQ: Kafka & Streaming Strategy

Do I need Kafka or is RabbitMQ enough?

The rule of thumb: If you want “Smart Broker, Dumb Consumer” (classic job queuing), choose RabbitMQ. If you need “Dumb Broker, Smart Consumer” (high throughput, replay capability, event sourcing), Kafka is indispensable. Kafka stores data, RabbitMQ deletes it after delivery. That is the fundamental difference.

How complicated is the ZooKeeper topic?

Traditionally, ZooKeeper was the most complex part. With modern Kafka versions (and in the ayedo stack), we increasingly use the KRaft mode. Here, the Kafka brokers themselves take over metadata management. ZooKeeper is eliminated as a separate component. This makes the cluster leaner, more stable, and easier to maintain.

What about Kafka Connect?

Kafka Connect is the framework for getting data in (Source) and out (Sink). AWS offers “MSK Connect” as an expensive add-on service. In the ayedo stack, you simply deploy Kafka Connect as another deployment type. You can use hundreds of open-source connectors (Debezium for SQL, S3, Elastic) without license fees.

How do I secure Kafka?

Kafka in the ayedo stack uses standard security mechanisms: mTLS (Mutual TLS) for encryption and authentication between client and broker, as well as SCRAM-SHA-512 or OIDC for user auth. With integrated ACLs (Access Control Lists), you precisely control which service can write to which topic.

Conclusion

A central nervous system must not become a black box. Those who use AWS MSK buy convenience with opacity and potentially exploding costs at high data throughput. Running Kafka on Kubernetes (via Strimzi) is today the industry standard for companies that take scalability seriously. With the ayedo Managed Stack, you get an enterprise streaming platform that is as flexible as your code and as robust as your infrastructure.