Kafka Part 2: Kafka Command Line Interface (CLI)

ALSO CHECK: Kafka Interview Questions And Answers

Now we will create topics from CLI.

• Open another command prompt, execute 'cd D:\Softwares\kafka_2.12-2.3.0\bin\windows'.
• Make sure zookeeper and kafka broker is running.
• Now execute 'kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1'. This will create a kafka topic with name 'first_topic', with 3 partitions and replication-factor as 1 (in our case we cannot mention replication-factor more than 1 because we have started only one broker).
• After executing the above command you will get a message 'Created topic first_topic'.
• How can we check if our topic is actually created? In the same command prompt, execute 'kafka-topics --zookeeper 127.0.0.1:2181 --list'. This will list all the topics which are present.
• To get the more details about the topic which is created, execute 'kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --describe'

Kafka Console Producer CLI

• Open command prompt, execute 'cd D:\Softwares\kafka_2.12-2.3.0\bin\windows'
• Now execute 'kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic'. After executing this, you can produce messages.
• We can also set acks properties, 'kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --producer-property acks=all'

What will happen if I run kafka-console-producer for the topic which is not present? 

• To check, lets execute 'kafka-console-producer --broker-list 127.0.0.1:9092 --topic new_topic' (FYI, new_topic is not present).
• When you execute it and produce first message you will get a warning 'Error while fetching metadata with correlation id 3 : {new_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)'. You will get this warning only after sending 1st message.
• This warning occurred because the topic was not present, when we send the 1st message a new_topic will be created but sice the leader is not selected that's why LEADER_NOT_AVAILABLE warning is shown. But since producer has a capability to recover itself, it will wait till the leader was available. 
• Let's check how this new_topic is created. To do this execute 'kafka-topics --zookeeper 127.0.0.1:2181 --topic new_topic --describe'

• We can see the new_topic is created with one partition. This is the default behavior. 
• We can change the default number of partitions by configuring it in server.properties. In the properties file we need to updated 'num.partitions'

Kafka Console Consumer CLI

• Open a new command prompt, execute 'cd D:\Softwares\kafka_2.12-2.3.0\bin\windows'.
• Now execute this, 'kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic'. You will see it will return nothing, because it only read from the time when it is launched. It will intercept only the new messages.
• To verify this, lets open another console for producer and execute 'kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --producer-property acks=all'. If you produce anything on this console, it will be automatically consumed by the consumer (which is in another console).

How can we read all the messages from kafka-console-consumer?
To do this we need to execute, 'kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning'. It will print all the messages (old and new).

Kafka Consumers in Group

• Open a new command prompt, execute 'cd D:\Softwares\kafka_2.12-2.3.0\bin\windows'.
• Execute 'kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-first-application'. Here 'my-first-application' is the group id. After executing we will not get anything, now lets start producing the messages on the same topic from a different command prompt. You will not notice the new messages will appear in the consumer console.
• Now open another console, and execute the same command ('kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-first-application'). And now start producing messages from the producer console, you will notice some of the messages will appear on consumer console 1 and some on consumer console 2.

How to read messages from the beginning by using group id?

• Close the second consumer console. And on the first consumer console execute 'kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-second-application --from-beginning'. You will notice all the previous (and new one) all be printed on the console. 
• Now if you stop and re-execute the same command, you will notice none of the old messages will be printed even though we have used from-beginning. This is because we had mentioned executed the same command earlier, which had read all the messages for mentioned group id (my-second-application) and offsets are committed. Since offsets are committed, even if we mention from-beginning all the messages won't be consumed.

Kafka Consumers Groups CLI

• Command to print all the consumer groups: 'kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list'
• Command to get the details about a consumer group: kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --describe --group my-first-application

Resetting Offsets

• Command to reset offsets for the consumer groups: kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group my-first-application --reset-offsets --to-earliest --execute --topic first_topic

-K Himaanshu Shuklaa..