Apache Kafka Producer and Consumer in Scala

è In this Scala & Kafka tutorial, you will learn how to write Kafka messages to Kafka topic (producer) and read messages from topic (consumer) using Scala example; producer sends messages to Kafka topics in the form of records, a record is a key-value pair along with topic name and consumer receives a message from a topic.

Prerequisites:

è If you don’t have the Kafka cluster setup, follow the link to set up the single broker cluster.

Kafka Installation on Linux

Kafka Installation Windows

Table of contents:

Start Zookeeper
Start Kafka Broker
Create a Kafka Topic
Producer Program
Consumer Program

Start zookeeper with the default configuration: -

ZooKeeper is a high-performance coordination service for distributed applications and Kafka uses ZooKeeper to store the metadata information of the cluster. Kafka comes with the Zookeeper built-in; all we need is to start the service with the default configuration.

$zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

Start Kafka broker with the default configuration:-

A Kafka cluster consists of one or more brokers(Kafka servers) and the broker organizes messages to respective topics and persists all the Kafka messages in a topic log file for 7 days. Depends on your replication factor of the topic, the messages are replicated to multiple brokers.

$kafka-server-start.sh /opt/kafka/config/server.properties

Create a Kafka topic “cfamily”:-

All Kafka messages are organized into topics and topics are partitioned and replicated across multiple brokers in a cluster. The producer sends messages to topic and consumer reads messages from the topic. The replication factor defines how many copies of the message to be stored and Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers.

Execute this command to create a topic with replication factor 1 and partition 1 (we have just 1 broker cluster).

E.g:-

$kafka-topics.sh --create --zookeeper localhost:2181 \
                  --replication-factor 1 --partitions 1 \
                   --topic cfamily

Kafka Maven Dependency:-

To work with Kafka we would use the following Kafka client maven dependency. Added this dependency to your scala project.

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.1.0</version>
</dependency>