Learn Apache Kafka

Implement Kafka Console Consumer

Kafka Console Consumer

Consumers read data from brokers. Consumers subscribes to one or more topics and consume published messages by pulling data from the brokers.

 

Steps to consume the messages from the topic:

Note: - Zookeeper as well as the kafka server should be up and run.

Step1: Open the command prompt and execute 'kafka-console-consumer' on the command line. This will help the user to read the data from the Kafka topic and output it to the standard outputs.

Note: Choose '.bat' or '.sh' as per the operating system.



รจ The highlighted text represents that a 'bootstrap-server' is required for the consumer to get connected to the Kafka topics. Also, a 'topic_id' is required to know from which topic the consumer will read the messages.

Step2: After knowing all the requirements, try to consume a message from a topic using the command:

Syntax: -

kafka-console-consumer -bootstrap-server localhost:9092 -topic <topic_name>

 

E.g: -

cmd>kafka-console-consumer -bootstrap-server localhost:9092 -topic cfamily

 

Note: Bootstrap server is the Kafka server, having port number=9092.

 

 

In the previous section, three messages were produced to this topic. But, in the above snapshot, 0 messages could be seen. It is because Apache Kafka does not read all the topics. A Kafka consumer will consume only those messages which are produced only when the consumer was in the active state. This can be categorized as a disadvantage of Apache Kafka.

 

Let's understand:

Open a new terminal. Launch the Kafka console producer. Keep both producer-consumer consoles together and start writing messages from producer and see consumer is consuming messages or not.

 

 Now, produce some messages in the producer console.

 

After doing so, press Ctrl+C and exit.

 

It is seen that all messages which are currently produced by the producer console are reflected in the consumer console. It is because the consumer is in an active state.

 

Reading whole messages

Apache Kafka allows to produce millions of messages. Sometimes, a consumer may require to read whole messages from a particular topic.

 

To do so, use '-from-beginning' command with the above kafka console consumer command as:

cmd>kafka-console-consumer -bootstrap-server 127.0.0.1:9092 -topic cfamily -from-beginning

This command tells the Kafka topic to allow the consumer to read all the messages from the beginning (i.e., from the time when the consumer was inactive).