Comments (7)
Hey @sudeepdino008 thanks for the question. From the top of my head it should work, but I might need to check it. I think if you introduce new schema and mark it as BACKWARD or BACKWARD_TRANSITIVE it should work because you will have to update consumers first and they will read the latest schema from the registry I guess.
Could you try to emulate it in that order to prove that update of the consumer didn't help:
- Register full version of the schema
- Use writer to write a message
- Use reader to read the message (should use the full schema)
- Register new version of the schema with fields removed
- Update (restart, or drop cache) of the reader
- Use writer to write a message (with new schema)
- Use reader from step 5 and read new message (should use new schema)
I think that's the way it should work and of course you will have to configure schema registry compatibility for the schema before registering new (or if it will be correct by default you are good).
And if it will not work – let's dig deeper
from avrora.
Hi @Strech,
I've created this test - https://github.com/sudeepdino008/avrora/blob/master/test/avrora/schema/evolution_test.exs
Let me know if I'm not thinking about it right. This should be possible to do, right?
1) test from_json/2 schema evolution (Avrora.Schema.EvolutionTest)
test/avrora/schema/evolution_test.exs:10
** (MatchError) no match of right hand side value: {:error, :schema_mismatch}
code: {:ok, dpayload} = Avrora.Codec.Plain.decode(epayload, schema: newschema)
stacktrace:
test/avrora/schema/evolution_test.exs:27: (test)
from avrora.
It it helps, my use case is that I'm updating the schema on the reader side (with backwards compatibility ensured -- so only changes like deleting field, or adding new fields with default value), and expect the reader to be able to read data which is avro encoded by an older schema.
from avrora.
Ok, so I came to know about schema registries. Can you tell if I'm understanding this correctly?
the reader must know about the schema with which the writer encoded some data. You can't expect the reader to be able to use a latest schema which evolved in backwards-compatible way, to decode data which is being encoded by the reader using some earlier schema version.
Avro needs schema registries to support evolution decoding. The reader and writer both have access to such a registry, and "register" new schema versions while also maintaining a local cache. The encoding also adds a "schema version identifier", which is used on the reader side to figure out the schema version, and fetch it from cache/schema registry.
However, this means that the read can happen off the older schema version. If the evolved schema had a field deleted OR a field added with default value (since evolution is backwards compatible), this modification has to be done and maintained separately from the decoding process.
from avrora.
Yes, the writer will register (or you will register separately) new schema in the schema registry and it will be given an ID. Such ID is stored later in the binary message writer will generate/obtain.
When reader will try to decode such message it will check for the ID if schema registry is enabled.
But you can emulate evolution by scenario I presented to you when local cache is busted in order to obtain the latest version of the schema from local file.
from avrora.
You are welcome
from avrora.
Thank you for the time :)
I'm experimenting with the OCF format + hooks in avrora now to skip certain fields in latest schema (which is linked to the original usecase)
from avrora.
Related Issues (20)
- Empty user-agent in header when calling registry
- hook to skip certain fields. HOT 2
- Fix warnings introduced in elixir 1.15 HOT 2
- logicalType: date HOT 5
- Allow mix task to accept a Client to be used for registration
- Drop unnecessary configuration options
- Remove plain format from decode/2 and encode/2
- Warnings on compilation HOT 4
- Maintain a CHANGELOG? HOT 5
- Dialyzer issues with private clients HOT 2
- "Messages encoded with OCF are wrapped in a List" what for? HOT 1
- Allow 3 named types to be used as root schemas
- Get schema name on decode HOT 3
- Add deprecation warning for encode with plain format
- Private clients hardcode schemas path at compile time HOT 6
- Any way to configure the registry_url at startup? HOT 8
- Option for decoding with tagged union values HOT 7
- unnamed_type error with primary type schema HOT 11
- Add option to enable SSL verification of registry requests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avrora.