Comments (5)
might be relevant to the investigation: https://forums.aws.amazon.com/thread.jspa?threadID=217922
Extract of this link:
Each supported instance type has a maximum supported payload for HTTP requests, option at this moment is 10MB and 100MB. So the only options at this moment, is to break it up in chunks, like you have mentioned. Just make sure that the format stays JSON.
The other option would be to have a client/application that reads the items line by line for a bulk process.
Please have a look at the limits that we currently have for ElasticSearch Service:
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html
If you have any questions, please let us know.
from snow-owl.
actually, my setup is a c5.large.elasticsearch which has 10 MiB as limit for "Maximum Size of HTTP Request Payloads"
Would it work if it was a c5.xlarge.elasticsearch
which has a 100Mib limit?
from snow-owl.
side question: after an import failure, is it rolled-back (like in a transactional database?) or should the part of the successfully imported data be cleaned manually in Elasticsearch?
from snow-owl.
I confirm upgrading to a c5.xlarge.elasticsearch
has resolved the problem. I manually deleted the indexes to clean the data imported only partially, and re-imported again successfully.
from snow-owl.
Hi @martinmart92,
Thank you for figuring out the problems and issues around Snow Owl and AWS Elasticsearch integration.
We think it would be beneficial to add some kind of configuration to Snow Owl's underlying Elasticsearch client, which would enable administrators to tweak the payload size of the HTTP requests sent by Snow Owl to ES. Unfortunately, this is not something we can plan for the upcoming release, but could be something for the next.
We will discuss it and get back to you if necessary. In the meantime, the higher tier AWS instance is a good workaround for the issue.
Cheers,
Mark
from snow-owl.
Related Issues (20)
- Missing results in GET `/snomed-ct/v3/{path}/members` HOT 9
- SnowOwl & AWS ElasticSearch
- ArrayIndexOutOfBoundsException on Import HOT 3
- Can we control how the searching works on the search APIs? HOT 5
- Error launching snow owl HOT 7
- SNOMED CT International Edition import not working HOT 2
- hapi-fhir-cli for upload of custom terminology HOT 1
- Opensearch integration HOT 2
- Target Platform is not able to Resolve HOT 3
- snowowl and ES / Opensearch on AWS HOT 2
- Snow Owl doesn't start from source in Eclipse HOT 3
- Publishing PreRelease content (to OntoServer) HOT 1
- Publication of Live and Pre-Release content to OntoServer Terminology Server HOT 4
- Concept Search API semanticTag filter issue HOT 3
- Contribution by code smell removal HOT 1
- UNABLE TO IMPORT SNOMED AFTER INSTALL HOT 7
- Support the "url" search parameter for both CodeSystem and ValueSet resources HOT 4
- SNOMED CT Import not working HOT 9
- Import of FHIR ValueSet HOT 2
- Log4j vulnerability HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snow-owl.