TL;DR
Most engineers know Kafka as:
Producer → Kafka → Consumer
But internally Kafka is much more sophisticated.
When a producer sends a message, Kafka must answer several critical questions:
- Which broker should store the message?
- Which partition should receive the message?
- How is the message replicated?
- How does Kafka prevent data loss?
- What happens if a broker crashes?
- Who decides the new leader?
- How does Kafka maintain consistency across the cluster?
The answers lie in Kafka's internal architecture.
Understanding this topic is essential for:
- Kafka Developers
- Senior Java Engineers
- Production Support Engineers
- SREs
- Architects
Most real-world Kafka troubleshooting involves concepts covered in this chapter.
Why Kafka Internal Architecture Matters
Imagine a banking application processing:
- Fund transfers
- Credit card payments
- Loan transactions
- ATM transactions
Losing a message is unacceptable.
At the same time:
- Servers can fail
- Networks can fail
- Disks can fail
- Applications can crash
Kafka was designed to continue operating despite these failures.
To achieve this, Kafka uses:
- Replication
- Leader Election
- Metadata Management
- ISR Management
- Distributed Coordination
Kafka Cluster Architecture
A Kafka cluster consists of multiple brokers working together.
Example:
Broker-1
Broker-2
Broker-3
Broker-4
Broker-5
Each broker is an independent Kafka server.
Together they form:
Kafka Cluster
The cluster behaves like a single logical system even though data is distributed across multiple machines.
This distribution is the foundation of Kafka scalability and fault tolerance.
How Kafka Stores Data
Consider a topic:
order-events
Suppose it contains three partitions:
Partition-0
Partition-1
Partition-2
Kafka distributes these partitions across brokers.
Example:
Broker-1 → Partition-0
Broker-2 → Partition-1
Broker-3 → Partition-2
This distribution allows Kafka to utilize multiple servers simultaneously.
Without distribution:
Single Server
=
Single Bottleneck
With distribution:
Multiple Servers
=
Higher Throughput
Replication
Replication is one of Kafka's most important features.
Imagine:
Broker-1
stores:
Partition-0
If Broker-1 crashes:
Data Lost
This is unacceptable in enterprise environments.
Kafka solves this through replication.
Example:
Partition-0
Leader → Broker-1
Replica → Broker-2
Replica → Broker-3
Now the same data exists on multiple servers.
If one broker fails:
Data Still Available
This is the foundation of Kafka fault tolerance.
Replication Factor
The Replication Factor defines how many copies of data Kafka maintains.
Example:
Replication Factor = 3
Means:
1 Leader Copy
2 Replica Copies
Total:
3 Copies
Enterprise environments commonly use:
Replication Factor = 3
because it provides a good balance between reliability and storage cost.
Leader and Follower Replicas
Every partition has:
One Leader
Multiple Followers
Example:
Partition-0
Leader → Broker-1
Follower → Broker-2
Follower → Broker-3
This is extremely important.
Leader Responsibilities
The Leader handles:
- Producer writes
- Consumer reads
- Offset management
- Replication coordination
All reads and writes happen through the leader.
Follower Responsibilities
Followers continuously copy data from the leader.
They do not normally serve producers or consumers.
Their purpose is:
Backup
and
Failover
Why Kafka Uses Leaders
Imagine:
3 Brokers
all accepting writes simultaneously.
Questions arise:
- Which write arrived first?
- Which version is correct?
- How are conflicts resolved?
Kafka avoids these problems by using a single leader per partition.
This guarantees ordering and consistency.
In-Sync Replicas (ISR)
One of the most important Kafka concepts.
Interviewers love asking about ISR.
What Is ISR?
ISR stands for:
In-Sync Replicas
It represents followers that are fully synchronized with the leader.
Example:
Leader → Broker-1
Follower → Broker-2
Follower → Broker-3
If both followers are up to date:
ISR = {Broker-1, Broker-2, Broker-3}
All replicas are healthy.
Why ISR Exists
Suppose:
Broker-3
becomes slow.
It falls behind.
Now:
Broker-3
may not contain recent messages.
Kafka removes it from ISR.
Example:
ISR = {Broker-1, Broker-2}
Only healthy replicas remain.
This prevents Kafka from promoting stale replicas during failures.
Controller
Every Kafka cluster has one special broker called:
Controller
Think of it as:
Cluster Manager
Responsibilities include:
- Monitoring broker health
- Managing leader elections
- Detecting failures
- Updating metadata
Only one controller exists at a time.
Broker Failure Scenario
Suppose:
Leader
Broker-1
crashes.
Partition:
Partition-0
becomes unavailable.
Kafka immediately begins recovery.
Controller detects:
Broker-1 Failed
Then selects a new leader from ISR.
Example:
Old Leader → Broker-1
New Leader → Broker-2
Consumers continue processing.
This happens automatically.
Leader Election
Leader election is the process of selecting a new leader when the current leader fails.
Example:
Before failure:
Leader → Broker-1
Follower → Broker-2
Follower → Broker-3
Broker-1 crashes.
Controller chooses:
Broker-2
as new leader.
Result:
Leader → Broker-2
Follower → Broker-3
System continues functioning.
This mechanism is one reason Kafka is highly available.
Unclean Leader Election
Advanced but important concept.
Suppose:
Broker-1 Leader
fails.
ISR contains:
Broker-2
But Broker-2 also fails.
Only:
Broker-3
remains.
However Broker-3 is not fully synchronized.
Kafka now has two choices:
Option 1
Promote Broker-3.
Risk:
Data Loss
Option 2
Wait for ISR recovery.
Risk:
Temporary Unavailability
This tradeoff is called:
Unclean Leader Election
Enterprise systems usually prioritize data consistency over availability.
ZooKeeper Era
Historically Kafka depended on:
Apache ZooKeeper
for:
- Metadata management
- Broker coordination
- Leader election
- Cluster state management
Architecture:
Kafka Brokers
↓
ZooKeeper
ZooKeeper maintained cluster information.
Problems with ZooKeeper
Although powerful, ZooKeeper introduced challenges:
- Additional infrastructure
- Operational complexity
- Maintenance overhead
- Scalability limitations
Running Kafka meant running:
Kafka
+
ZooKeeper
which increased operational burden.
KRaft Mode
Modern Kafka introduces:
KRaft
which stands for:
Kafka Raft Metadata Mode
KRaft removes ZooKeeper completely.
Kafka now manages metadata internally.
Benefits:
- Simpler architecture
- Easier deployment
- Better scalability
- Lower operational overhead
Modern Kafka deployments increasingly use KRaft.
Metadata Management
Kafka maintains metadata about:
- Topics
- Partitions
- Brokers
- Leaders
- Replicas
- Consumer Groups
Example metadata:
Topic → order-events
Partition → 0
Leader → Broker-2
Replicas → Broker-1,2,3
This information is continuously updated.
Without metadata management, Kafka would not know where data resides.
End-to-End Internal Flow
Suppose Order Service publishes:
{
"orderId":1001
}
Step 1:
Producer contacts Kafka.
Step 2:
Kafka metadata identifies leader.
Step 3:
Message sent to partition leader.
Step 4:
Leader writes message to disk.
Step 5:
Followers replicate message.
Step 6:
ISR updated.
Step 7:
Acknowledgement returned.
Step 8:
Consumer later reads message.
This entire process happens within milliseconds.
Production Support Perspective
Many Kafka production incidents originate from internal architecture problems.
ISR Shrink
Followers falling behind.
Symptoms:
- Replication delays
- Increased risk
Leader Election Storms
Frequent leader changes.
Symptoms:
- Latency spikes
- Throughput reduction
Broker Failure
Broker unavailable.
Symptoms:
- Partition movement
- Rebalancing activity
Replication Delays
Followers cannot keep up.
Symptoms:
- High network usage
- Increased lag
Metadata Issues
Cluster coordination problems.
Symptoms:
- Client connection failures
- Producer timeouts
Understanding internal architecture is essential for troubleshooting these issues.
Common Mistakes
Replication Factor = 1
No fault tolerance.
Single broker failure can cause data loss.
Ignoring ISR Health
Cluster appears healthy but resilience decreases.
Excessive Broker Restarts
Can trigger unnecessary leader elections.
Not Monitoring Controller Events
Can hide cluster stability issues.
Poor Replica Distribution
Creates uneven load across brokers.
Interview Questions
What is a Kafka Broker?
A Kafka server responsible for storing and serving partition data.
What is Replication Factor?
The number of copies Kafka maintains for partition data.
Why does Kafka use leaders and followers?
To guarantee consistency while maintaining fault tolerance.
What is ISR?
In-Sync Replicas are replicas fully synchronized with the leader.
What is Leader Election?
The process of selecting a new partition leader after a failure.
What is the role of the Controller?
The controller manages broker failures, leader elections, and cluster coordination.
Why was ZooKeeper used?
ZooKeeper handled metadata management and cluster coordination.
What is KRaft?
Kafka's modern metadata management architecture that eliminates ZooKeeper.
Why is ISR important?
ISR ensures only fully synchronized replicas participate in failover decisions.
What happens if a leader broker fails?
The controller elects a new leader from ISR and processing continues.
Revision Notes
Kafka achieves fault tolerance through replication. Every partition has one leader and multiple follower replicas. Producers and consumers interact with leaders, while followers continuously replicate data. ISR represents replicas fully synchronized with the leader and plays a critical role during failover. The controller manages cluster coordination and leader elections. Historically Kafka used ZooKeeper for metadata management, but modern deployments increasingly use KRaft, which simplifies architecture by removing ZooKeeper. Understanding leaders, followers, ISR, replication, controller behavior, and leader elections is fundamental for designing, operating, and troubleshooting enterprise Kafka clusters.
Next Topic: Kafka Message Lifecycle
In the next topic we will follow a single message from creation to consumption and deeply understand:
- Producer → Broker communication
- Partition selection
- Message writes
- Replication flow
- Offset creation
- Consumer reads
- Offset commits
- Message delivery guarantees (At Most Once, At Least Once, Exactly Once)
This topic explains what actually happens when a Kafka message travels through the system.