Apache Kafka Intro

Kafka is like a messaging system (like RabbitMQ) but designed for huge volumes of data and real-time streaming. It acts as a central hub where applications send and receive data (messages/events) in real time.

Producer ---> Topic ---> Partitions ---> Consumer
             (orders)   (P0  P1  P2)
                           ↓   ↓   ↓
                        Consumers Group

Kafka Components

• Topics:
  • The core abstraction in Kafka
  • Named channels to which messages are published
  • Divided into partitions for parallel processing
  • Messages in a partition are ordered with a sequential ID (offset)
• Brokers:
  • The servers that store and manage topics
  • Multiple brokers form a Kafka cluster
  • Each broker handles certain topic partitions
  • Provide redundancy and fault tolerance
• Producers:
  • Applications that publish messages to topics
  • Can send messages to specific partitions or let Kafka handle distribution
  • Support various delivery semantics (at-least-once, at-most-once, exactly-once)
• Consumers:
  • Applications that subscribe to topics and process messages
  • Track their position in each partition using offsets
  • Group into consumer groups for parallel processing
• Consumer Groups:
  • Collection of consumers that work together
  • Each partition is consumed by only one consumer in a group
  • Enables both queue-like (when in the same group) and pub/sub (when in different groups) patterns
• ZooKeeper/KRaft:
  • Coordinates the Kafka cluster
  • Manages broker metadata and leader election
  • Newer versions can use KRaft mode instead of ZooKeeper
• Connect API:
  • Framework for connecting Kafka to external systems
  • Standardized way to import/export data
• Streams API:
  • Library for stream processing applications
  • Enables transformation of data between input and output topics

How kafka work with (Queue, Pub/Sub)

1. Kafka as a Queue

• In a queue model, each message is consumed by only one consumer in a consumer group.
• Kafka achieves this with consumer groups:
  • A topic has partitions (parallel lanes of data).
  • Within a consumer group, Kafka assigns each partition to only one consumer.
  • This ensures load balancing (scaling horizontally) and no duplicate consumption inside that group.

👉 Example:

• Topic: orders with 3 partitions.
• Consumer group: order-service with 3 consumers.
• Each consumer gets one partition (Consumer A → P0, B → P1, C → P2).
• Each message in a partition is processed by one consumer only in that group.

2. Kafka as Pub/Sub

• In a publish/subscribe model, multiple consumers (in different groups) can read the same data independently.
• Kafka allows many consumer groups to subscribe to the same topic:
  • Group analytics reads every message.
  • Group fraud-detection also reads every message.
  • Group billing also reads every message.
• Each group gets its own offsets (read pointers), so they consume independently.

👉 Example:

• Topic: payments
• Producer sends payment events.
• Consumers:
  • dashboard-service shows metrics.
  • fraud-service checks suspicious payments.
  • audit-service stores history.
• All of them get all messages, independent of each other.