salsify / avro-builder Goto Github PK
View Code? Open in Web Editor NEWRuby DSL to create Avro schemas
License: MIT License
Ruby DSL to create Avro schemas
License: MIT License
Rake task fails to build avro files from dsl. Using ruby-2.6.5.
Building this DSL
# <rails_root>/avro/dsl/example.rb
namespace 'com.example'
fixed :password, 8
enum :user_type, :ADMIN, :REGULAR
record :user do
required :id, :long
required :user_name, :string
required :type, :user_type, default: :REGULAR
required :pw, :password
optional :full_name, :string
required :nicknames, :array, items: :string
required :permissions, :map, values: :bytes
end
running be rake avro:generate
generates this error
Generating Avro schema from /Users/steveburkett/openstax/event-capture-api/avro/dsl/nudged.rb
rake aborted!
ArgumentError: wrong number of arguments (given 1, expected 0)
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:74:in `to_h'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:74:in `to_h'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:79:in `to_json'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:48:in `block (3 levels) in define'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:38:in `each'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:38:in `block (2 levels) in define'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/bin/ruby_executable_hooks:24:in `eval'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/bin/ruby_executable_hooks:24:in `<main>'
Attempting to generate a schema for the following DSL (note the values
option isn't specified for the map) raises a NoMethodError
record :my_event do
required :my_map, :map
end
The stack trace is:
NoMethodError: undefined method `serialize' for nil:NilClass
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/map_type.rb:20:in `serialize'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/field.rb:80:in `serialized_type'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/field.rb:67:in `serialize'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `block (2 levels) in to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `map'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `block in to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/schema_serializer_reference_state.rb:22:in `definition_or_reference'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:50:in `to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/dsl.rb:80:in `to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/dsl.rb:85:in `to_json'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder.rb:9:in `build'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:37:in `block (3 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `each'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `block (2 levels) in define'
Currently, I don't think these are supported in the avro
gem.
The following basic example works with Ruby 2.6.2 but not with 2.6.5
require 'avro/builder'
Avro::Builder.build do
namespace 'com.example'
fixed :password, 8
enum :user_type, :ADMIN, :REGULAR
record :user do
required :id, :long
required :user_name, :string
required :type, :user_type, default: :REGULAR
required :pw, :password
optional :full_name, :string
required :nicknames, :array, items: :string
required :permissions, :map, values: :bytes
end
end
Error:
Traceback (most recent call last):
2: from schemas/example.rb:3:in `<main>'
1: from /Users/sushma/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/avro-builder-0.17.0/lib/avro/builder.rb:17:in `build'
/Users/sushma/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:80:in `to_json':
uninitialized constant Avro::Builder::DSL::JSON (NameError)
Errors in DSL files can result in stack traces like this:
ArgumentError: wrong number of arguments (given 1, expected 2..3)
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `instance_eval'
(eval):11:in `block in initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:38:in `instance_eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:38:in `block in create_and_configure_builtin_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:35:in `tap'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:35:in `create_and_configure_builtin_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:86:in `create_named_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:36:in `record'
(eval):3:in `initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `instance_eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder.rb:10:in `new'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder.rb:10:in `build'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:37:in `block (3 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `each'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `block (2 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/bin/ruby_executable_hooks:15:in `eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => avro:generate
(See full trace by running task with --trace)
It would be helpful if calls to instance_eval
included the source file to more easily track errors back to a line in a DSL file e.g. /Users/jturkel/salsify/alerts-service/avro/dsl/com/salsify/alerts/events/alert_created_event_value.rb
rather than (eval)
.
This would be used to skip generating an Avro JSON schema file (.avsc) for abstract types.
I have a schema file, in which I have two fields having the same schema, array of records
optional :categories, :array, items: :name_image_hash
optional :cart_images, :array, items: :name_image_hash
name_image_hash
looks like
record "name_image_hash" do
required :url, :string
required :name, :string
abstract true
end
and it is generating schema like
{
"name": "cart_images",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
"name": "name_image_hash",
"fields": [
{
"name": "url",
"type": "string"
},
{
"name": "name",
"type": "string"
}
]
}
}
],
"default": null
},
{
"name": "categories",
"type": [
"null",
{
"type": "array",
"items": "name_image_hash"
}
],
"default": null
}
cart_images
rendered properly, but categories
doesn't.
I am doing something wrong, or is it a bug ??
Earlier, I wanted to do this inline, but I am not able to get the right syntax.
See #12 for an initial implementation to hide this method.
I'm trying to write a record where one of the fields is an array where each item is a record. I can do this if I don't inline the inner-most record, e.g.
record :record_in_array do
required :name, :string
end
record :top_level do
required :array_of_records, :array, items: :record_in_array
end
Is it possible to inline the record_in_array
record? The following errors:
record :top_level do
required :array_of_records, :array, items: :record do
required :name, :string
end
end
#> NoMethodError: undefined method `required' for #<Avro::Builder::Field:0x007fdd3437d278>
When loading a DSL from a file infer the namespace from the file path relative to a configured load path.
The namespace should still be overridable in the DSL.
which would be the most efficient in terms of byte storage (and by how much) for dates in avro/kafka? is there anything else w/ how to store the date time?
(1) timestamp-millis (long).
(2) rfc3339 (string) for a datetime.
Thanks.
Optional fields are supported as a union of null
and another type.
An open question is how union should interact with required/optional.
#56 introduced mutli_json which broke pretty printing of generated schemas in applications that include Rails but not oj orYajl. There appears to be something about the Rails dependency that causes the MultiJson JSON gem adapter not to support pretty printing. Here's a reproducible test case:
require 'bundler/inline'
gemfile(true) do
source 'https://rubygems.org'
# gem 'oj'
# gem 'yajl-ruby'
gem 'multi_json'
gem 'rspec'
# Tests pass when Rails is not present
gem 'rails'
end
require 'rspec/autorun'
require 'multi_json'
describe do
let(:data) do
{ foo: { bar: 'baz' } }
end
let(:pretty_data) do
"{\n \"foo\": {\n \"bar\": \"baz\"\n }\n}"
end
it "pretty prints" do
puts "Using adapter #{MultiJson.adapter}"
expect(MultiJson.dump(data, pretty: true)).to eq(pretty_data)
end
end
Is this useful outside protocols?
When i build a dsl like
namespace 'org.openstax.interactions'
record :nudged do
required :user_uuid, :string
...
it creates avsc (in ./avro/schema/nudged.avsc) like this
{
"type": "record",
"name": "nudged",
"namespace": "org.openstax.interactions",
"fields": [
{
"name": "user_uuid",
However, using the schema store from avro_turf, setting the schema store:
AvroTurf::Messaging.new(
registry_url: 'http://localhost:8081/',
schemas_path: 'avro/schema'
)
This fails to work when i try to encode:
AvroTurf::Messaging.new(...).encode(data, schema_name: "org.openstax.interactions.nudged")
due to this error
"exception": "#<AvroTurf::SchemaNotFoundError: could not find Avro schema at `avro/schema/org/openstax/interactions/nudged.avsc'>",
Is there a way to tell avro-builder to build the schema in directory structure that avroturf expects, or to tell avroturf to look for schemas in the way avro-builder outputs?
thanks!!
This should include all the methods in the DSL, where they can be called, and options accepted.
Perhaps this should be a separate document instead of making the README extremely long.
Hi there,
How do i get the rake task to find DSLs under a Rails.root directory called 'oxavro'? i cant get this to work.
e.g., i put this
Avro::Builder.add_load_path('oxavro/dsl')
inside an initializer, but still not working. i think this gem is still using Rails.root/avro as the top level dir.
By default generated schemas are validated by attempting to parse using the avro
gem.
This is a placeholder for now to collect other validations that would be useful to give better feedback for disallowed situations.
Is there a way to have avro-builder skip certain file name masks (like *_swagger.rb) when it's generating avsc files?
I ask because i'd like to keep other (related) files side by side w/ the foobar.rb avro dsl file. like foobar_swagger.rb)
thanks!
Is there a way to import (or otherwize) bring in shared fields into a record? (if not..this would be soooo great)
e.g, like this
record :foobar do
required :highlight_id, :string
import '../shared/all_events'
end
It would be useful to optionally accept a single DSL file (or list of files) to regenerate instead of reprocessing all files.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.