In this article we are going to teach you on Architecture of Kafka. producers, topics, partitions, brokers, and consumers are the main components in Kafka. Following image shows the Kafka architecture.
Kafka cluster: Cluster is a group of one or more Kafka brokers.
Kafka Brokers: Brokers are individual servers. Brokers receive messages from producers, store them on disk, and deliver them to consumers.
Topic: In Kafka topic is nothing but a category.
Kafka producer: Producers sends data to a Kafka topic. It publishes data to the Kafka cluster for consumption.
Kafka consumer: Consumer subscribes to the topic and receives data sent by the producer.
In Kafka, the producer and consumer are independent of each other. The producer can keep sending messages without waiting for the consumer to process them, and the consumer can read messages without affecting the producer.
Zookeeper: Zookeeper is a distributed coordination service used for managing Kafka brokers, maintaining metadata, and facilitating leader election for partitions. It helps in maintaining the overall state of the Kafka cluster and ensuring fault tolerance.
Partitions: Partitions are separate shelves in a topic to store messages. This allowing for parallel processing and scalability. that means consumers can read from different partitions at the same time, making data processing faster. Additionally, partitions help ensure that messages are stored in order and replicated for fault tolerance.