History of Kafka :

Apache Kafka was developed at LinkedIn and later became an open sourced Apache project in 2011. Scala and Java are used to write Kafka. Apache Kafka is publish-subscribe based fault tolerant messaging system. It is fast, scalable and distributed by design.


When we are processing big data, we have to deal with huge volumes of data in this case usually we will face kinds of issues. one is to collect the large data or messages and another is to analyze the collected data so to over come these problems we need a new efficient messaging system.

What is messaging ? 

Messaging is a system which will transfer the data between two applications asynchronously. here asynchronously means if the producers sends the data at this moments then the consumers may or may not consume the data or messages at the same time. There are two kinds of message patterns are available

  1. Point to point messaging : Message is delivered to only one recipient and it uses queues
  2. Publish – subscribe messaging :Message is delivered to multiple recipients and it uses topics



Benefits of KAFKA:

  1. Reliability − Distributed,replicated, partitioned and fault tolerance.
  2. Scalability − It can scale easily without down time.
  3. Durability − It uses Distributed commit log means the messages will persists to the disk as fast as possible, so that it is durable.
  4. Performance − It has high throughput for both publishing and subscribing. we will get the stable and constant performance even if we run for long duration with billion’s of messages.

Fundamentals and Terminology of KAFKA: 


kafka overview


In this diagram, a topic is configured into three partitions. Partition 1 has two offset factors 0 and 1. Partition 2 has four offset factors 0, 1, 2, and 3. Partition 3 has one offset factor 0. The id of the replica is same as the id of the server that hosts it.

Screen Shot 2016-07-16 at 12.30.04 AM

Assume, if the replication factor of the topic is set to 3, then Kafka will create 3 identical replicas of each partition and place them in the cluster to make available for all its operations. To balance a load in cluster, each broker stores one or more of those partitions. Multiple producers and consumers can publish and retrieve messages at the same time.

Topics: A stream of messages belonging to a particular category is called a topic. Data is stored in topics.

Topics are split into partitions. For each topic, Kafka keeps a mini-mum of one partition. Each such partition contains messages in an immutable ordered sequence. A partition is implemented as a set of segment files of equal sizes.

Partition: Topics may have many partitions, so it can handle an arbitrary amount of data.

Partition offset: Each partitioned message has a unique sequence id called as offset.

Replicas of partition: Replicas are nothing but backups of a partition. Replicas are never read or write data. They are used to prevent data loss.

Brokers: Brokers are simple system responsible for maintaining the pub-lished data. Each broker may have zero or more partitions per topic. Assume, if there are N partitions in a topic and N number of brokers, each broker will have one partition.

Assume if there are N partitions in a topic and more than N brokers (n + m), the first N broker will have one partition and the next M broker will not have any partition for that particular topic.

Assume if there are N partitions in a topic and less than N brokers (n-m), each broker will have one or more partition sharing among them. This scenario is not recommended due to unequal load distri-bution among the broker.

Kafka Cluster: Kafka’s having more than one broker are called as Kafka cluster. A Kafka cluster can be expanded without downtime. These clusters are used to manage the persistence and replication of message data.

Producers: Producers are the publisher of messages to one or more Kafka topics. Producers send data to Kafka brokers. Every time a producer publishes a message to a broker, the broker simply appends the message to the last segment file. Actually, the message will be appended to a partition. Producer can also send messages to a partition of their choice.

Consumers: Consumers read data from brokers. Consumers subscribes to one or more topics and consume published messages by pulling data from the brokers.

Leader: Leader is the node responsible for all reads and writes for the given partition. Every partition has one server acting as a leader.

Follower: Node which follows leader instructions are called as follower. If the leader fails, one of the follower will automatically become the new leader. A follower acts as normal consumer, pulls messages and up-dates its own data store.


let us know if you have any questions with the comments


Leave a Reply