Avro formatter plugin for Embulk

Avro formatter plugin for Embulk.

Overview

• Plugin type: formatter

Support avro types

Support all avro basic types.

• string
• int
• long
• float
• double
• boolean
• enum
• fixed
• array
• map
• record

But typecasting is restricted by embulk column type. See. AvroValueConverters,

Configuration

• avsc: avro schema (avsc) filepath (string, required)
• codec: avro codec type (enum: deflate, bzip2, xz, snappy, optional)
• compression_level: avro codec compression level (integer, optional, for only deflate and xz codec)
• skip_error_record: If you want to skip error record, set true (boolean, default: false)

Example

out:
  type: file
  path_prefix: ./out_
  file_ext: avro
  formatter:
    type: avro
    avsc: schema.avsc
    skip_error_record: true

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously