zolyfarkas / avro Goto Github PK
View Code? Open in Web Editor NEWThis project forked from pimco/avro
Mirror of Apache Avro
License: Apache License 2.0
This project forked from pimco/avro
Mirror of Apache Avro
License: Apache License 2.0
currently reading the doc it is not clear what types are accepted.
While upgrading from version 1.9.0.11p
to 1.10.0.8p
, the org.apache.avro.LogicalType class appears to have methods deleted, with no mention of how to replace them. For example, #getLogicalTypeName() has been removed. However, in that case, I see in the code that the method was simply a wrapper for #getName(), so no issue there, although I would like some sort of deprecation message before it simply breaks.
There is another method that has been removed from LogicalType called #serialize() which I don't know how to resolve. We use this method and it has been removed with no deprecation. I checked the CHANGES.md document and the commit message as well, with no luck.
The change happened here. This is our usage:
private static void writeLogicalType(final Schema schema, final JsonGenerator writer, final Object value)
throws IOException {
// We are - do we have a logical type defined?
LogicalType logicalType = schema.getLogicalType();
if (logicalType != null) {
String typeName = logicalType.getLogicalTypeName(); // method no longer exists, will switch to "getName()"
// Convert the logical type and write - we assume these are implicitly primitives
if (DecimalEncoder.OPTIMIZED_JSON_DECIMAL_WRITE && "decimal".equals(typeName)) {
writer.writeNumber((java.math.BigDecimal) value);
} else if (DecimalEncoder.OPTIMIZED_JSON_DECIMAL_WRITE && "bigint".equals(typeName)) {
writer.writeNumber((java.math.BigInteger) value);
} else {
Object serialized = logicalType.serialize(value); // method no longer exists, replacement?
writer.writeObject(serialized);
}
} else {
// Write the best value we can find
writeJsonSimpleDataValue(value, writer);
}
}
"any" logical type can be applied to a record type with the fields: string avsc and bytes content.
where avsc will contain the schema, and content will contain the serialized content acording to this schema.
ExtendedJson(De|En)Coder will allow for a more natural serialization in JSON.
I was using this dependecy on my project and now I''m receiving the error below:
Authorization failed for https://dl.bintray.com/zolyfarkas/core/org/apache/avro/avro/1.8.1.48p/avro-1.8.1.48p.pom 403 Forbidden
I tried upgrade to new link, using github link but now I'm receiving an another error 401 - unauthorized.
Authentication failed for https://maven.pkg.github.com/zolyfarkas/*/org/apache/avro/avro/1.10.0.1p/avro-1.10.0.1p.pom 401 Unauthorized
Can help me?
Based on this article, Bintray is going away on May.1st.2021 and will no longer serve the binary for this project.
Do you have plans for moving the binary somewhere else, if yes, what's your timeframe.
Any recommendations of how to proceed with this after May 1st for projects that depends on the project?
https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/
Hi @zolyfarkas ,
Is there any plan to support higher avro versions?
We have integrated custom avro with confluent schema registry(Confluent-5.5.0), but during the upgrade found below issue and blocked on upgrade:
From 6.2.0, a method overload was deleted between Avro version 1.9.1 and 1.10.1. Avro version that is earlier than 1.10.1 might result in below error:
Caused by: java.lang.NoSuchMethodError: org.apache.avro.Schema.toString(Ljava/util/Collection;Z)Ljava/lang/String;
Any update or suggestion will be helpful. Thanks in advance.
I checked out the code from release tag avro-toplevel-1.10.0.9p, and was trying the build the project under lang/java/avro. However it fails with below exception
[ERROR] Failed to execute goal on project avro: Could not resolve dependencies for project org.apache.avro:avro:bundle:1.10.0.9p: The following artifacts could not be resolved: org.spf4j.avro:core-schema:jar:0.19, org.apache.avro:avro:jar:1.10.0.2p: Could not find artifact org.spf4j.avro:core-schema:jar:0.19 in github (https://api.github.com/users/zolyfarkas/repos/*) -> [Help 1]
[ERROR]
core-schema:jar:0.19, seems unavailable in maven repository as well.
When loading a json that in its schema uses logical type Date, the schema below
{
"type": "int",
"logicalType": "date"
}
I get the error
Caused by: java.lang.ClassCastException: class org.apache.avro.LogicalTypes$Date cannot be cast to class org.apache.avro.Date (org.apache.avro.LogicalTypes$Date and org.apache.avro.Date are in unnamed module of loader 'app')
lowered I'll put the complete traceback
org.apache.avro.AvroRuntimeException: Cannot convert 19073:Integer: expected generic type
at org.apache.avro.Conversions.convertToLogicalType(Conversions.java:259)
at org.apache.avro.generic.GenericDatumReader.convert(GenericDatumReader.java:195)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:134)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:152)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:223)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:213)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:144)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:115)
at com.grupozap.validator.validation.JsonToAvroDeserializer.validate(JsonToAvroDeserializer.java:27)
at com.grupozap.validator.validation.JsonToAvroDeserializer.validateRecord(JsonToAvroDeserializer.java:61)
at com.grupozap.validator.service.ValidatorService.valid(ValidatorService.java:23)
at com.grupozap.validator.controller.ValidatorController.post(ValidatorController.java:22)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class org.apache.avro.LogicalTypes$Date cannot be cast to class org.apache.avro.Date (org.apache.avro.LogicalTypes$Date and org.apache.avro.Date are in unnamed module of loader 'app')
at org.apache.avro.data.DateConverter.fromInt(DateConverter.java:91)
at org.apache.avro.data.DateConverter.fromInt(DateConverter.java:17)
at org.apache.avro.Conversions.convertToLogicalType(Conversions.java:245)
Hello, I had one case in a project where we have some timestamp fields printed as string (we cannot fix that) so when we try to use the Json decoder it fails. Do you think this feature could be added to the readLong function? I mean, to try to parse to one (or more, configurable) datetime formats? I have done some work on this and could open a MR in case you think it is relevant.
Tks
Hi, do you have any plans for supporting Avro 1.11?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.