Scala-Kafka-Avro (Producing and Consuming Avro messages)


This post will show you how to write and read messages in Avro format to/from Kafka.

Instead of using with plain-text messages, though, we will serialize our messages with Avro. That will allow us to send much more complex data structures over the wire.


Apache Avro is a language neutral data serialization format. A avro data is described in a language independent schema. The schema is usually written in JSON format and the serialization is usually to binary files although serialization to JSON is also supported.

Let’s add Avro dependency in build:

"org.apache.avro"  %  "avro"  %  "1.7.7"

We will consider schema like this:

    "namespace": "kakfa-avro.test",
     "type": "record",
     "name": "user",
         {  "name": "id", "type": "int"},
         {   "name": "name",  "type": "string"},
         {   "name": "email", "type": ["string", "null"]}

You can instantiate schema as follows:

val schema: Schema = new Schema.Parser().parse(SCHEMA_STRING)

Here, SCHEMA_STRING is the JSON listed above as…

View original post 616 more words


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s