Learn Apache Kafka

Kafka Consumer in Scala

Kafka Consumer scala example: -

This Kafka Consumer scala example subscribes to a topic and receives a message (record) that arrives into a topic. This message contains key, value, partition, and off-set. All messages in Kafka are serialized hence, a consumer should use deserializer to convert to the appropriate data type. Here we are using StringDeserializer for both key and value.

E.g:-

import java.util.{Collections, Properties}
import java.util.regex.Pattern
import org.apache.kafka.clients.consumer.KafkaConsumer
import scala.collection.JavaConverters._
object KafkaConsumerSubscribeApp extends App {
 
  val props:Properties = new Properties()
  props.put("group.id", "test")
  props.put("bootstrap.servers","localhost:9092")
  props.put("key.deserializer",
      "org.apache.kafka.common.serialization.StringDeserializer")
  props.put("value.deserializer",
      "org.apache.kafka.common.serialization.StringDeserializer")
  props.put("enable.auto.commit", "true")
  props.put("auto.commit.interval.ms", "1000")
  val consumer = new KafkaConsumer(props)
  val topics = List("cfamily")
  try {
    consumer.subscribe(topics.asJava)
    while (true) {
      val records = consumer.poll(10)
      for (record <- records.asScala) {
        println("Topic: " + record.topic() +
                 ",Key: " + record.key() + 
                 ",Value: " + record.value() +
                 ", Offset: " + record.offset() +
                 ", Partition: " + record.partition())
      }
    }
  }catch{
    case e:Exception => e.printStackTrace()
  }finally {
    consumer.close()
  }
}

 

How to Run Kafka Producer and Consumer?

1.     Run KafkaConsumerSubscribeApp.scala program

When you run this program, it waits for messages to arrive in “cfamily” topic.

2.     Run KafkaProducerApp.scala program

Run KafkaProducerApp.scala program which produces messages into “cfamily”.

O/P:-

Now, you should see the messages that were produced in the console. And on another console, you should see the messages that are consuming.