Comments (7)
skydeep-9
commented on August 22, 2024
I have tried using version 1.0.0 and 1.0.1 both for software.amazon.glue:schema-registry-serde
from aws-glue-schema-registry.
blacktooth
commented on August 22, 2024
Hi,
Are you able to invoke Glue Schema Registry APIs from the environment / server where you are running the above producer? You can try using the AWS CLI to test it.
aws glue list-registries --region us-west-2
aws glue get-schema-version --schema-id "RegistryName=dtc_avro_registry,SchemaName=pixel_events_organic" --schema-version-number "VersionNumber=1"
The schema registry SerDe library invokes Glue APIs as part of serialization / deserialization.
from aws-glue-schema-registry.
skydeep-9
commented on August 22, 2024
Yes, I was able to
aws glue list-registries --region us-west-2
{
"Registries": [
{
"RegistryName": "dtc_avro_registry",
"RegistryArn": "arn:aws:glue:us-west-2:012732009581:registry/dtc_avro_registry",
"Description": "dtc Avro registry",
"Status": "AVAILABLE",
"CreatedTime": "2021-03-10T20:17:20.992Z",
"UpdatedTime": "2021-03-10T20:17:20.992Z"
}
]
}
aws glue get-schema-version --schema-id "RegistryName=dtc_avro_registry,SchemaName=pixel_events_organic" --schema-version-number "VersionNumber=1"
An error occurred (EntityNotFoundException) when calling the GetSchemaVersion operation: Schema is not found. RegistryName: dtc_avro_registry, SchemaName: pixel_events_organic, SchemaArn: null
from aws-glue-schema-registry.
blacktooth
commented on August 22, 2024
Do you mind sharing more information on what environment you are using to call SchemaRegistry service? From the error message, it looks like a SSL issue. Are you able to invoke other AWS services from this environment?
from aws-glue-schema-registry.
blacktooth
commented on August 22, 2024
We haven't heard back. Please feel free to re-open if you are still facing this.
from aws-glue-schema-registry.
k-chainalysis
commented on August 22, 2024
I had this same error calling Schema Registry from Lambda execution, and sure enough the execution role did not have IAM permissions to call Schema Registry. Thanks for the tip!
from aws-glue-schema-registry.
Paolo-de-vathaire
commented on August 22, 2024
@blacktooth Sorry to revive this old thread but it seems appropriate, I'm facing the same error as @skydeep-9.
Using aws glue list-registries --region eu-central-1 , I can also get the schema by id but I need to add the region to the command you provided, otherwise aws cli doesn't find it and returns EntityNotFoundException.
I could work around that using aws CLI by setting my default region to eu-central-1 but not with this lib.
Looking at the AWSSchemaRegistryClient.getSchemaVersionIdByDefinition method, it seems region is not sent as part of the query.
I'm correctly setting schema, registry and region as properties :
configProps.put(AWSSchemaRegistryConstants.AWS_REGION, "eu-central-1");
configProps.put(AWSSchemaRegistryConstants.REGISTRY_NAME, "XXX");
configProps.put(AWSSchemaRegistryConstants.SCHEMA_NAME, "XXX");
I'm using SASL_SSL with AWS_MSK_IAM to authenticate to AWS services.
Also this is the dependency I'm using to access Glue registry :
<dependency>
<groupId>software.amazon.glue</groupId>
<artifactId>schema-registry-kafkastreams-serde</artifactId>
<version>1.1.18</version>
</dependency>
Is that a simple bug or did I miss something ?
EDIT :
Here is the stacktrace I get :
com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to auto-register schema. Auto registration of schema is not enabled.
at com.amazonaws.services.schemaregistry.common.SchemaByDefinitionFetcher.getORRegisterSchemaVersionId(SchemaByDefinitionFetcher.java:81)
at com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistrySerializationFacade.getOrRegisterSchemaVersion(GlueSchemaRegistrySerializationFacade.java:86)
at com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer.serialize(GlueSchemaRegistryKafkaSerializer.java:113)
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1000)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:947)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:1050)
at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:799)
at org.springframework.kafka.core.KafkaTemplate.observeSend(KafkaTemplate.java:768)
at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:551)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException: Failed to get schemaVersionId by schema definition for schema name = XXX
at com.amazonaws.services.schemaregistry.common.AWSSchemaRegistryClient.getSchemaVersionIdByDefinition(AWSSchemaRegistryClient.java:155)
at com.amazonaws.services.schemaregistry.common.SchemaByDefinitionFetcher$SchemaDefinitionToVersionCache.load(SchemaByDefinitionFetcher.java:110)
at com.amazonaws.services.schemaregistry.common.SchemaByDefinitionFetcher$SchemaDefinitionToVersionCache.load(SchemaByDefinitionFetcher.java:106)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
at com.google.common.cache.LocalCache.get(LocalCache.java:4011)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)
at com.amazonaws.services.schemaregistry.common.SchemaByDefinitionFetcher.getORRegisterSchemaVersionId(SchemaByDefinitionFetcher.java:74)
... 14 more
Caused by: software.amazon.awssdk.services.glue.model.EntityNotFoundException: Schema version is not found. (Service: Glue, Status Code: 400, Request ID: 6b7b1216-ea03-4204-a46b-77b22305d02c)
from aws-glue-schema-registry.
Related Issues (20)
- schema-registry-serde:1.1.17 transient dependency org.json:json:jar:20230227 CVE-2023-5072 HOT 3
- Please update aws sdk v2 version. schema-registry-serde:1.1.17 conflicts with latest aws-sdk libs. HOT 2
- Allow customization of ObjectMapper / registration of default modules missing HOT 1
- `NoSuchMethodError` from `GlueSchemaRegistryDeserializerDataParser` when running on JDK 8 HOT 2
- JsonDeserializer does not allow enabling/disabling a feature for Jackson Object Mapper HOT 4
- Avro deserialization fails in 1.1.18 HOT 3
- ProtobufSchemaConverter fails in v1.1.18 HOT 1
- Vulnerability CVE-2024-21634 (High Severity) HOT 1
- AWS MSK and AWS Glue Schema Registry Configuration is Wrong
- v1.1.17 to v1.1.19 fails producing messages with error "java.lang.NoSuchFieldError: IDENTITY_PROVIDERS" on SelfManaged Kafka server with no authentication HOT 2
- Production of AVRO specific record seems to be creating a Generic Record - v1.1.19/Lambda/Java 17 - Protobuf reading a typed record doesn´t find the class to map to
- Can't enable BACKWARD_COMPATIBILTY mode in the s3 sink connector HOT 1
- New constructor for AWSSchemaRegistryClient request
- Test `GlueSchemaRegistryConfigurationTest.testBuildConfig_noRegionConfigsSupplied_throwsException` fails in some build scenarios but not others HOT 1
- Serialization errors when using JSON messages on MSK Connect together with GSR Confluent JDBC Sink
- Nested Protobuf Message with oneof fails to convert
- Question: Why is kafka streams caching disabled in the README.md kafka streams example?
- New release with updated kafka.scala.version HOT 1
- Kafka Connect with kafka-connect-s3 plugin getting error Can't redefine: com.amazonaws.services.schemaregistry.kafkaconnect.avrodata.Union HOT 2
- Unnecessary dependency on jimfs?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-glue-schema-registry.