Diving into Kafka's Protocol
Diving Into is a blogpost serie where we are digging a specific part of of the project’s basecode. In this episode, we will digg into Kafka’s protocol.
The protocol reference
For the last few months, I worked a lot around Kafka’s protocols, first by creating a fully async Kafka to Pulsar Proxy in Rust, and now by contributing directly to KoP (Kafka On Pulsar). The full Kafka Protocol documentation is available here, but it does not offer a global view of what is happening for a classic Producer and Consumer exchange. Let’s dive in!
Common handshake
After a client established the TCP connection, there is a few common requests and responses that are almost always here.
The common handhake can be divided in three parts:
- Being able to understand each other. For this, we are using API_VERSIONS to know which versions of which TCP frames can be uses,
- Establish Auth using SASL if needed, thanks to SASL_HANDSHAKE and SASL_AUTHENTICATE,
- Retrieve the topology of the cluster using METADATA.
All exchange are based between a Kafka 2.0 cluster and client.
All the following diagrams are generated with MermaidJS.
Producing
The PRODUCE API is used to send message sets to the server. For efficiency it allows sending message sets intended for many topic partitions in a single request.
Consuming
Consuming is more complicated than producing. You can learn more in The Magical Group Coordination Protocol of Apache Kafka By Gwen Shapira, Principal Data Architect @ Confluent and also in the Kafka Client-side Assignment Proposal.
Consuming can be divided in three parts:
- coordinating the consumers to assign them partitions, using:
- then fetch messages using:
- Send lifeproof to the coordinator using HEARTBEAT.
For the sake of the explanation, we have now another Broker1 which is holding the coordinator for topic ‘my-topic’. In real-life, it would be the same.
Thank you for reading my post! Feel free to react to this article, I am also available on Twitter if needed.