TL;DR
Every Kafka architecture is built using a few fundamental components:
- Broker
- Topic
- Partition
- Offset
- Producer
- Consumer
- Consumer Group
- Kafka Cluster
If Topic 1 explained why Kafka exists, Topic 2 explains how Kafka actually works.
Most Kafka interview questions, production issues, and architectural decisions revolve around these components.
Understanding them deeply is mandatory before learning Kafka internals, producer tuning, consumer tuning, troubleshooting, and architecture design.
Why Kafka Uses Multiple Building Blocks
Imagine an enterprise e-commerce system generating:
- 1 million orders per day
- 5 million payment events per day
- 20 million inventory events per day
- 50 million user activity events per day
A single server cannot efficiently process this volume forever.
Kafka solves this problem by distributing data across multiple servers while maintaining durability and scalability.
To achieve this, Kafka introduces several architectural building blocks that work together.
Before diving into each component, let's understand the overall flow.
Producer
↓
Topic
↓
Partition
↓
Broker
↓
Consumer Group
↓
Consumer
Every Kafka event passes through this architecture.
Kafka Broker
What Is a Broker?
A Kafka Broker is a Kafka server responsible for storing and serving data.
When people say:
"Kafka Cluster"
they are actually referring to a group of Kafka Brokers.
For example:
Broker-1
Broker-2
Broker-3
Together these form a Kafka Cluster.
Each broker is responsible for:
- Receiving messages
- Storing messages
- Serving messages to consumers
- Replicating messages
- Managing partitions
Think of a broker as a warehouse that stores events.
Why Multiple Brokers?
Imagine:
100 Million Events Per Day
One server may become:
- CPU bottleneck
- Disk bottleneck
- Network bottleneck
Kafka solves this through horizontal scaling.
Instead of:
1 Huge Server
Kafka prefers:
10 Smaller Servers
This makes scaling easier and cheaper.
Real Production Example
An e-commerce company may have:
Broker-1
Broker-2
Broker-3
Broker-4
Broker-5
If Broker-3 crashes:
Kafka can continue operating using replicas stored on other brokers.
This is one of the foundations of Kafka's fault tolerance.
Topic
What Is a Topic?
A Topic is a logical category used to organize events.
Think of a topic as a database table.
Example topics:
order-events
payment-events
inventory-events
customer-events
Every event related to orders goes into:
order-events
Every payment event goes into:
payment-events
Topics help organize and manage data.
Real Example
Order Service publishes:
{
"orderId": 1001,
"customerId": 501,
"amount": 2500
}
Kafka stores this event inside:
order-events
Consumers interested in orders subscribe to:
order-events
Important Concept
Topics are logical containers.
The actual data is stored inside:
Partitions
which we will discuss next.
Partition
What Is a Partition?
A Partition is the most important Kafka concept.
A topic is divided into one or more partitions.
Example:
order-events
may contain:
Partition-0
Partition-1
Partition-2
Partition-3
The actual messages are stored inside these partitions.
Why Partitions Exist
Suppose:
100 Million Events
stored in one location.
Reading and writing becomes slow.
Instead Kafka splits data across partitions.
Example:
Partition-0 → 25 Million Events
Partition-1 → 25 Million Events
Partition-2 → 25 Million Events
Partition-3 → 25 Million Events
Load becomes distributed.
Performance improves dramatically.
Partitions Enable Parallel Processing
Without partitions:
1 Consumer
must process everything.
With partitions:
Consumer-1 → Partition-0
Consumer-2 → Partition-1
Consumer-3 → Partition-2
Consumer-4 → Partition-3
Now processing happens in parallel.
This is one reason Kafka scales so effectively.
Interview Question
Why are partitions important?
Answer:
Partitions provide scalability, parallelism, and high throughput.
Offset
What Is an Offset?
Every message inside a partition receives a unique sequence number called an Offset.
Example:
Partition-0
Offset-0
Offset-1
Offset-2
Offset-3
Offset-4
Offsets identify the exact position of a message inside a partition.
Why Offsets Matter
Suppose a consumer processes:
Offset-0
Offset-1
Offset-2
and crashes.
After restart:
Kafka can resume from:
Offset-3
instead of reading everything again.
Offsets are the foundation of Kafka reliability.
Important Interview Point
Offsets are unique only within a partition.
They are not globally unique across the entire topic.
Producer
What Is a Producer?
A Producer is an application that sends data to Kafka.
Examples:
- Order Service
- Payment Service
- Mobile App Backend
- Inventory Service
Producer responsibility:
Create Event
↓
Send To Kafka
Real Example
Order Service creates:
{
"orderId": 1001,
"status": "CREATED"
}
Producer sends this event to Kafka.
Kafka stores it.
Producer's job ends.
Key Advantage
Producer does not care:
- Who consumes the event
- How many consumers exist
- Whether consumers are online
This creates loose coupling.
Consumer
What Is a Consumer?
A Consumer reads messages from Kafka.
Examples:
- Notification Service
- Analytics Service
- Fraud Detection Service
- Audit Service
Consumer responsibility:
Read Event
↓
Process Event
Real Example
Notification Service receives:
{
"orderId": 1001,
"status": "CREATED"
}
It sends:
Order Confirmation Email
Consumer processing is independent of producers.
Consumer Group
What Is a Consumer Group?
A Consumer Group is a collection of consumers working together.
This concept enables scalability.
Example:
Consumer Group
Consumer-1
Consumer-2
Consumer-3
Consumer-4
Why Consumer Groups Exist
Suppose:
1 Million Messages
need processing.
One consumer may be too slow.
Kafka distributes partitions across multiple consumers.
Example:
Consumer-1 → Partition-0
Consumer-2 → Partition-1
Consumer-3 → Partition-2
Consumer-4 → Partition-3
Each consumer handles part of the workload.
Critical Rule
Inside a consumer group:
One Partition
↓
One Consumer
at a time.
This prevents duplicate processing.
Production Interview Question
If a topic has:
4 Partitions
and consumer group has:
6 Consumers
how many consumers remain idle?
Answer:
2 Consumers
Because only four partitions exist.
Kafka Cluster
What Is a Kafka Cluster?
A Kafka Cluster is a collection of Kafka Brokers working together.
Example:
Broker-1
Broker-2
Broker-3
Broker-4
Broker-5
Together they provide:
- Scalability
- Availability
- Fault Tolerance
- Load Distribution
Why Clusters Matter
Single Broker:
Single Point Of Failure
Cluster:
High Availability
If one broker fails:
Other brokers continue serving data.
End-to-End Flow
Let's combine everything.
Step 1
Order Service creates event.
Producer sends:
{
"orderId": 1001
}
Step 2
Kafka stores event in:
order-events
Topic.
Step 3
Event is written into:
Partition-2
Step 4
Message receives:
Offset-150
Step 5
Broker stores event.
Step 6
Consumer Group reads event.
Step 7
Notification Service processes event.
Step 8
Offset committed.
Processing complete.
Production Support Perspective
Most Kafka production incidents involve these building blocks.
Consumer Lag
Consumer cannot keep up with partition data.
Partition Imbalance
Some partitions overloaded.
Offset Problems
Consumers reprocess messages.
Broker Failure
Kafka cluster loses node.
Consumer Group Rebalancing
Partitions reassigned unexpectedly.
Every Kafka troubleshooting exercise eventually comes back to understanding:
- Brokers
- Topics
- Partitions
- Offsets
- Producers
- Consumers
- Consumer Groups
Common Mistakes
Creating Too Few Partitions
Limits scalability.
Creating Excessive Partitions
Increases cluster overhead.
Ignoring Consumer Lag
Leads to delayed processing.
Using Random Topic Naming
Creates operational confusion.
Not Understanding Offsets
Causes duplicate processing issues.
Interview Questions
What is a Kafka Broker?
A Kafka server responsible for storing and serving data.
What is a Topic?
A logical category used to organize Kafka messages.
What is a Partition?
A subdivision of a topic used for scalability and parallelism.
What is an Offset?
A unique sequence number identifying a message inside a partition.
What is a Producer?
An application that sends messages to Kafka.
What is a Consumer?
An application that reads messages from Kafka.
What is a Consumer Group?
A group of consumers working together to process messages.
Why are partitions important?
They enable scalability, throughput, and parallel processing.
Can multiple consumers read the same partition in the same consumer group?
No. Only one consumer can read a partition within a consumer group at a time.
What is a Kafka Cluster?
A collection of Kafka brokers working together.
Revision Notes
Kafka Brokers are servers that store and serve data. Topics logically organize events. Partitions physically store events and enable scalability. Offsets uniquely identify messages within partitions and support recovery. Producers publish events. Consumers process events. Consumer Groups allow multiple consumers to share workload. Kafka Clusters provide scalability and fault tolerance. Understanding these building blocks is essential because every advanced Kafka concept is built on top of them.
Next Topic: Kafka Internal Architecture Deep Dive
We will cover:
- Kafka metadata management
- ZooKeeper vs KRaft
- Controller node
- Leader and follower replicas
- ISR (In-Sync Replicas)
- Leader election
- Replication internals
- How Kafka maintains consistency and fault tolerance
This is where Kafka starts becoming an architect-level topic.