Usually in a normal architecture, throughput is high only until your database becomes the bottleneck.
For example, in Discord, imagine every user sends a message and that message is first stored in a database and then emitted to other users.
If thousands of users are active at the same time and they are all messaging simultaneously, your database is cooked and deep-fried.
First of all,
- this will put a huge load on your DB.
second,
- it will become slower.
and third,
- it will not be truly realtime because of all those delays.
Also, there exist many other services that may need the same data for analytics, notifications, search, monitoring, and so on.
So my boy, in production, with that normal architecture, you are cooked.
Throughput
Throughput is like water flowing through a pipe.
The pipe defines how much water can flow through it at a given time.
Similarly, a database has a limit on how many operations it can handle per second. If too many requests arrive at the same time, the database becomes the bottleneck.
So, what if we could separate the process of receiving data from the process of storing or processing data?
So here comes Kafka. Kafka says,“I have high throughput.”
But we cannot simply say, “instead of storing data in a database, let’s store everything in Kafka.” That’s not what Kafka is primarily for.
Kafka is a distributed event streaming platform. It is designed to handle a large volume of events and allow different consumers to process those events indepedently.
Your database can still remain the primary storage system.
Kafka sits between the systems.
Producer and Consumer
let’s say we have a producer and a consumer.
The producer produces data.
The consumer consumes that data and does something with it.
Without Kafka, it could look something like this:
Producer
|
v
Database
|
v
Consumer
But if multiple services need the same data (as shown below), things become more complicated.
/ ----> Analytics
/
Producer -> Database --> Notification
\
\ ----> Search
Now the producer is tightly connected to the database and the consumers.
But With Kafka:
Here, the producer sends an event to Kafka. Kafka stores that event in a Topic.
Different consumers can then consume that event according to their own requirements.
This gives us something very important: Decoupling. i.e. The producer doesn’t need to know exactly who is going to consume the data (Producer and consumer don’t need to know about or directly depend on each other).
Kafka Topics
Inside Kafka, we have something called a Topic. A topic is basically a logical category of events.
For example, suppose we have an application that receives information about birds.
we could have:
bird-detections
as a topic.
A producer can publish bird detection events to that topic. Consumers can then subscribe to that topic.
You can think of a topic as a stream of related events.
Partitions
But we can go one step further.
A topic itself can be divided into multiple partitions.
suppose we have a topic called:
bird-detections
we could have:
bird-detections
Partition 0
Partition 1
Partition 2
Partition 3
Here each partition is an ordered sequence of events.
For example, if our application receives bird detections from Nepal, we could partition the data based on some meaningful key.
For example:
Partition 0 -> Himalayan region
Partition 1 -> Terai region
Partition 2 -> Hill region
Although in a real system, you would usually choose the partition key based on the access and distribution requirements of your application rather than simply dividing geographic regions.
The important thing is that partitions allow Kafka to scale horizontally.
You can see an architecture below,
Consumer and Partition
Now suppose we have four partitions.
P0
P1
P2
P3
If we have Two consumers. Kafka can distribute the partitions between them:
P0 -> Consumer 1
P1 -> Consumer 1
P2 -> Consumer 2
P3 -> Consumer 2
If we have three consumers (Consumer 3 gets data from two partition):
P0 -> Consumer 1
P1 -> Consumer 2
P2 -> Consumer 3
P3 -> Consumer 3
And if we have four consumers, Every consumer gets a data from each partition:
P0 -> Consumer 1
P1 -> Consumer 2
P2 -> Consumer 3
P3 -> Consumer 4
But what if we have five consumers? Consumer 5 will get nothing because we only have 4 partition which is assigned to each consumer already.
P0 -> Consumer 1
P1 -> Consumer 2
P2 -> Consumer 3
P3 -> Consumer 4
Consumer 5 -> nothing
This is an important rule:
One consumer can consume from multiple partitions, but within the same consumer group, one partition can be assigned to only one consumer at a time.
so:
1 Consumer -> Multiple Partitions
1 Partition -> 1 Consumer
Within the same consumer group.
This is one of the reasons why the number of partitions matters when designing a Kafka system.
Consumer Groups
Now we have a problem.
What if we want the same Kafka topic to be consumed independently by different applications? This is where Consumer Groups come in.
A consumer group is basically a group of consumers working together to consume a topic.
suppose we have:
Topic
P0
P1
P2
P3
and Consumer Group 1 has five consumers:
G1:
C1
C2
C3
C4
C5
Kafka distributes the four partitions among the consumers.
P0 -> C1
P1 -> C2
P2 -> C3
P3 -> C4
C5 -> nothing
Now suppose we have another consumer group:
G2:
C1
C2
The same four partitions can be consumed by this group too:
P0 -> C1
P1 -> C1
P2 -> C2
P3 -> C2
Notice something interesting. The partitions are being consumed independently by the two groups.
This is how Kafka can behave like both a queue and a publish/subscribe system, depending on how you use consumer groups.
Is Kafka a Queue or Pub/Sub?
The answer is: Both.
In a queue-like setup, multiple consumers can belong to the same consumer group. The work is distributed among them.
For example:
Topic
|
+-- P0 -> Consumer 1
|
+-- P1 -> Consumer 2
|
+-- P2 -> Consumer 3
|
+-- P3 -> Consumer 4
Each event is processed by one consumer within that group. But suppose we have multiple consumer groups:
-> Group A
/
Producer -> Kafka
\
-> Group B
both groups can consume the same events independently. This behaves much more like publish/subscribe.
For example:
-> Analytics Group
/
Producer -> Kafka
\
-> Notification Group
the analytics service can consume the event. The notification service can consume the same event. Neither one has to know about the other.
Broker
Now we need to understand where Kafka actually stores all of this data.
A Kafka server is called a Broker. A broker is responsible for storing and serving the data that belongs to the partitions assigned to it.
For example, suppose we have three Kafka servers:
Broker 1
Broker 2
Broker 3
These brokers together form a Kafka cluster.
Remember that a topic is divided into partitions. Kafka distributes these partitions across the brokers in the cluster.
For example, suppose our topic has three partitions:
Topic: orders
P0
P1
P2
Kafka could distribute them like this:
Broker 1 -> P0
Broker 2 -> P1
Broker 3 -> P2
In other words, the partitions are stored inside the brokers.
Kafka Cluster
│
├── Broker 1
│ └── P0
│
├── Broker 2
│ └── P1
│
└── Broker 3
└── P2
This is important because Kafka can now spread the workload across multiple machines.
For example, if producers are writing data to all three partitions, the writes can be handled by different brokers:
Producer
│
├──> Broker 1 -> P0
├──> Broker 2 -> P1
└──> Broker 3 -> P2
Instead of forcing a single machine to handle all the data, Kafka can distribute the storage and workload across the entire cluster.
However, this introduces another problem.
What happens if a broker fails?
Suppose Broker 1 suddenly crashes:
Broker 1 ❌
└── P0
If P0 exists only on Broker 1, then Kafka can no longer access P0.
That means producers and consumers that need P0 may not be able to operate normally until Broker 1 comes back.
This would be a serious problem in a distributed system. A single machine failure shouldn’t cause data to become unavailable.
This is where Replication comes in.
Kafka can keep multiple copies of each partition on different brokers.
For example, with a replication factor of 3, P0 could look like this:
Broker 1 -> P0 ← Leader
Broker 2 -> P0 ← Replica
Broker 3 -> P0 ← Replica
Now P0 exists on three different machines.
If Broker 1 fails:
Broker 1 ❌ -> P0
Broker 2 -> P0 ← can become Leader
Broker 3 -> P0 ← Replica
Kafka can promote one of the replicas to become the new leader.
So replication gives Kafka fault tolerance. Even if a broker goes down, the data can still be available from another broker.
This also gives us an important distinction:
- Partitioning distributes data across brokers to increase scalability and parallelism.
- Replication copies partitions across brokers to provide fault tolerance and availability.
Together, these are two of the fundamental ideas that allow Kafka to operate as a distributed event streaming system.
Replication
Kafka can maintain multiple replicas of a partition. Suppose P0 has three replicas:
Broker 1 -> P0
Broker 2 -> P0
Broker 3 -> P0
But these replicas don’t all independently act as the active partition. One replica becomes the leader. The others are followers.
P0 Leader
|
+--------+--------+
| |
Follower Follower
Producers normally write to the leader, and consumers normally read from the leader. The followers replicate the leader’s data.
If the leader fails, Kafka can elect another suitable replica as the new leader. This is how Kafka provides fault tolerance.
ZooKeeper
For a long time, Kafka used another system called ZooKeeper. ZooKeeper is a distributed coordination system.
Kafka used it for things such as:
- tracking which brokers are alive
- controller management
- controller election
- cluster metadata
- configuration and coordination state
So the architecture looked roughly like this:
ZooKeeper
/ | \
/ | \
Broker1 Broker2 Broker3
Kafka brokers depended on ZooKeeper for important cluster coordination tasks. But Kafka eventually introduced a different architecture.
KRaft
KRaft stands for: Kafka Raft.
It replaces Kafka’s dependency on ZooKeeper by moving cluster metadata management into Kafka itself.
Instead of having:
Kafka
|
v
ZooKeeper
|
+---- Broker 1
+---- Broker 2
+---- Broker 3
we now have Kafka controllers managing the metadata using the Raft consensus protocol.
Roughly:
Kafka Controllers
/ | \
/ | \
C1 C2
\ | /
\ | /
Kafka Cluster
/ | \
B1 B2 B3 B=Broker
The controllers use Raft to agree on the cluster’s metadata. So instead of Kafka depending on an external ZooKeeper cluster, Kafka can manage this coordination itself. This simplifies the architecture and removes an entire external system that Kafka previously needed.
What Actually Changed?
Suppose we have three Kafka brokers:
Broker 1
Broker 2
Broker 3
with ZooKeeper, we had something like:
ZooKeeper
/ | \
/ | \
B1 B2 B3
ZooKeeper was responsible for coordination and cluster metadata.
with KRaft:
Kafka Controllers
/ | \
C1 C2 C3
\ | /
\ | /
Brokers
/ | \
B1 B2 B3
The controller quorum manages Kafka’s metadata using Raft.
So the big philosophical change is simple:
Kafka no longer needs ZooKeeper to manage its cluster metadata. Kafka itself becomes responsible for that coordination.
The Bigger Philosophy of Kafka
And this is where Kafka becomes interesting.
Kafka isn’t simply:
“a faster database.”
It is also not simply:
“a queue.”
The bigger idea is decoupling systems through an ordered, durable event stream. Instead of making every service directly communicate with every other service:
Service A -> Database -> Service B
-> Service C
-> Service D
we can introduce Kafka:
-> Service B
/
Service A -> Kafka -> Service C
\
-> Service D
Now Service A doesn’t need to know exactly how Service B, C, or D works. It just produces an event.
Kafka stores and distributes that event. The consumers decide what to do with it. This is why Kafka becomes extremely useful in large distributed systems.
You can add new consumers without changing the producer.
You can scale consumers using partitions.
You can tolerate broker failures using replication.
You can replay events when needed.
And you can build multiple independent pipelines from the same stream of data.
So the philosophy is not really about Kafka being “fast.”
It’s about moving systems from tightly coupled request-based communication toward loosely coupled event-driven architecture.
and once your system starts looking like:
Producers
|
v
Kafka
|
+------> Analytics
|
+------> Search
|
+------> Notifications
|
+------> Data Warehouse
|
+------> Machine Learning
You start realizing why Kafka exists in the first place. Not to replace your database. Not to magically make everything realtime. But to become the high-throughput event backbone connecting different parts of your system.
and that, my boy, is the philosophy of Kafka.