Visualizing Kafka

Basics

Before we begin, let’s make sure we’re on the same page about Kafka. It’s event streaming software. It allows backend services (usually in micro-services architecture) to communicate with each other.

Two services communicating via Kafka

Producers and Consumers

Producers and consumers are services that listen to or send messages in Kafka. These services are your backend services.

Consumer and Producer
A service listening to messages and consuming them

Topics

Topics are addresses that producers can send messages to. Other services can listen to these topics.

A service emitting a message and a service receiving a message from a Kafka topic
A consumer group listening to topic B
A message going into a single service with a consumer group
Producer sending message to Kafka topic.
Message getting stored on the queue
A copy of the message being stored on the queue and sent to consumer.
Sending a second message to Topic A
Second message being stored.
Messages being queued up in topics

Partitions

I lied. A Kafka topic is not really a single queue, but actually composed of many queues called partitions! They help a topic scale.

A topic with two partitions
A message entering a topic, going to a partition
A message being consumed from a partition
A producer writing to a topic, which is writing to multiple partitions
A producer sending messages (with possibly different entity/user ids) to different partitions
Messages being consumed in order from partitions.
Regardless of why different message types are mapped into single partitions, they maintain order
Two perfectly valid scenarios. Each partition maintains order for its own messages.

Infrastructure

If we take a step back, let’s look at our first chart. What is the Kafka cloud?

Two services communicating via Kafka
Zookeeper routing traffic in and out of the Kafka cluster.
Zookeeper maintaining a set of nodes
Two topics with two partitions
Numbered partitions
Topic A, Partition #1
A message sent to the leader.
Sending the message to the consumer and duplicating it on all of the followers
Message in each partition copy
Two partition copies
Partition #1 and #2 in our cluster.
Both clusters
What we had before.
What we have now!

Conclusion

I hope you have a better understanding of Kafka now. I hope these visualizations helped you figure which questions to ask and what to google for. There are incredible guides on each one of the principles in this article.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Timothy Stepro

Timothy Stepro

Software engineer by day, software engineer by night.