Git Product home page Git Product logo

avro-sandbox's Introduction

Avro Sandbox

Experiments with evolving Avro schemas, using fastavro for serialization and deserialization.

Purpose

Determine how Avro handles differences in writer and reader schemas.

From a reader schema perspective:

  • Which differences with writer schema are allowed?

  • Which differences with the writer schema would raise an error?

Some Findings

  1. Adding a default value to a field makes it "optional".

    • A reader schema may include new fields, with defaults, and still be able to read data from the writer schema that does not have the field (reader made a backwards compatible change).
    • If the reader schema does not include a field that the writer has added and specified a default, the reader can still read it (writer made a forwards compatible change).
  2. Fields are considered "required" unless a default is specified.

    • A reader cannot add a required field in a backwards-compatible manner.
    • A reader can remove a required field in a backwards-compatible manner.
    • A writer may add a new required field and still be considered a forwards-compatible change.
  3. Setting a field's type to a union of "null" and one or more other types, means it is "nullable" (can contain null, i.e. None in Python), but does not make it optional.

Run

pip install -r requirements.txt

pytest

avro-sandbox's People

Contributors

davidwmartines avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.