The avrolixr's discuss from avvo

Serialization error: Unknown magic byte!

Hello!
Guys, I have an error with Serialization of data.
Message serialized successfully pushed to kafka. But on the other side of Kafka another application reads the Kafka queue and deserializes my message.
But deserializarion causes an exception:

greenRiver[ERROR] Processing geronimo_entities offset 20
greenRiver[ERROR] message: Objavro.codenullavro.schema{"type":"record","name":"entities","fields":[{"name":"content","type":["string","null"]},{"name":"content_type_id","type":["int","null"]},{"name":"created_by","type":["int","null"]},{"name":"id","type":["int","null"]},{"name":"inserted_at","type":["string","null"]},{"name":"kind","type":["string","null"]},{"name":"schema_version","type":["string","null"]},{"name":"scope","type":["string","null"]},{"name":"storefront","type":["string","null"]},{"name":"updated_at","type":["string","null"]}],"namespace":"com.foxcommerce.geronimo"}���*k|^��Y_�5�x��{"title":"Some title foooooo","tags":["tag","another"],"body":"Lorem ipsum","author":"John Doe" ,2017-06-28T09:27:51.0ZBlogPost,2017-06-28T06:31:54.0Z1foo.bar,2017-06-28T09:27:51.0Z���*k|^��Y_�5�x
greenRiver[ERROR] Error deserializing avro message with key entities_16: error org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] 	Objavro.codenullavro.schema{"type":"record","name":"entities","fields":[{"name":"content","type":["string","null"]},{"name":"content_type_id","type":["int","null"]},{"name":"created_by","type":["int","null"]},{"name":"id","type":["int","null"]},{"name":"inserted_at","type":["string","null"]},{"name":"kind","type":["string","null"]},{"name":"schema_version","type":["string","null"]},{"name":"scope","type":["string","null"]},{"name":"storefront","type":["string","null"]},{"name":"updated_at","type":["string","null"]}],"namespace":"com.foxcommerce.geronimo"}���*k|^��Y_�5�x��{"title":"Some title foooooo","tags":["tag","another"],"body":"Lorem ipsum","author":"John Doe" ,2017-06-28T09:27:51.0ZBlogPost,2017-06-28T06:31:54.0Z1foo.bar,2017-06-28T09:27:51.0Z���*k|^��Y_�5�x
greenRiver[ERROR] Not processed: geronimo_entities offset: 20
greenRiver[ERROR] Failure during processing geronimo_entities offset 20: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Here is the schema

%{fields: [%{"name" => "content", "type" => ["string", "null"]},
   %{"name" => "content_type_id", "type" => ["int", "null"]},
   %{"name" => "created_by", "type" => ["int", "null"]},
   %{"name" => "id", "type" => ["int", "null"]},
   %{"name" => "inserted_at", "type" => ["string", "null"]},
   %{"name" => "kind", "type" => ["string", "null"]},
   %{"name" => "schema_version", "type" => ["string", "null"]},
   %{"name" => "scope", "type" => ["string", "null"]},
   %{"name" => "storefront", "type" => ["string", "null"]},
   %{"name" => "updated_at", "type" => ["string", "null"]}], name: "entities",
  namespace: "com.foxcommerce.geronimo", type: "record"}

Here is the message

%{"content" => "{\"title\":\"Some title foooooo\",\"tags\":[\"tag\",\"another\"],\"body\":\"Lorem ipsum\",\"author\":\"John Doe\"}",
  "content_type_id" => 1, "created_by" => 4, "id" => 16,
  "inserted_at" => "2017-06-28T09:27:51.0Z", "kind" => "BlogPost",
  "schema_version" => "2017-06-28T06:31:54.0Z", "scope" => "1",
  "storefront" => "foo.bar", "updated_at" => "2017-06-28T09:27:51.0Z"}```

Here is the encoded message

{:ok,
 <<79, 98, 106, 1, 3, 232, 8, 20, 97, 118, 114, 111, 46, 99, 111, 100, 101, 99,
   8, 110, 117, 108, 108, 22, 97, 118, 114, 111, 46, 115, 99, 104, 101, 109, 97,
   172, 8, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 99, ...>>}

We're using confluent platform

Any advices?

Many thanks!

Unable to decode a message

Our schema is this

{
  "type" : "record",
  "name" : "SpecialtySuggestion",
  "namespace" : "AvvoEvent",
  "doc" : "The payload for an avvo question specialty suggestion",
  "fields" : [ {
    "name" : "question_id",
    "type" : "int",
    "doc" : "The question id"
  }, {
    "name" : "specialty_scores",
    "type" : {
      "type" : "array",
      "items" : {
        "type" : "record",
        "name" : "SpecialtyScore",
        "doc" : "The payload for an avvo question specialty score",
        "fields" : [ {
          "name" : "score",
          "type" : "float",
          "doc" : "The score"
        }, {
          "name" : "specialty_id",
          "type" : "int",
          "doc" : "The specialty id"
        } ]
      }
    },
    "doc" : "scores"
  }, {
    "name" : "event",
    "type" : {
      "type" : "record",
      "name" : "Event",
      "doc" : "The main class for this protocol",
      "fields" : [ {
        "name" : "name",
        "type" : "string",
        "doc" : "The name of the event"
      }, {
        "name" : "timestamp",
        "type" : "int",
        "doc" : "The timestamp of the event"
      }, {
        "name" : "app_id",
        "type" : "string",
        "doc" : "The id of the generating application"
      } ]
    },
    "doc" : "Basic event data"
  } ]
}

If you need data to debug this, let me know.

avvo / avrolixr Goto Github PK

avrolixr's Issues

Serialization error: Unknown magic byte!

Unable to decode a message

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent