import Snomed error (413 Request Entity Too Large) AWS S3 about snow-owl HOT 5 CLOSED

might be relevant to the investigation: https://forums.aws.amazon.com/thread.jspa?threadID=217922

Extract of this link:

Each supported instance type has a maximum supported payload for HTTP requests, option at this moment is 10MB and 100MB. So the only options at this moment, is to break it up in chunks, like you have mentioned. Just make sure that the format stays JSON.

The other option would be to have a client/application that reads the items line by line for a bulk process.

Please have a look at the limits that we currently have for ElasticSearch Service:
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

If you have any questions, please let us know.

from snow-owl.

actually, my setup is a c5.large.elasticsearch which has 10 MiB as limit for "Maximum Size of HTTP Request Payloads"
Would it work if it was a c5.xlarge.elasticsearch which has a 100Mib limit?

from snow-owl.

side question: after an import failure, is it rolled-back (like in a transactional database?) or should the part of the successfully imported data be cleaned manually in Elasticsearch?

from snow-owl.

I confirm upgrading to a c5.xlarge.elasticsearch has resolved the problem. I manually deleted the indexes to clean the data imported only partially, and re-imported again successfully.

from snow-owl.

Hi @martinmart92,

Thank you for figuring out the problems and issues around Snow Owl and AWS Elasticsearch integration.
We think it would be beneficial to add some kind of configuration to Snow Owl's underlying Elasticsearch client, which would enable administrators to tweak the payload size of the HTTP requests sent by Snow Owl to ES. Unfortunately, this is not something we can plan for the upcoming release, but could be something for the next.
We will discuss it and get back to you if necessary. In the meantime, the higher tier AWS instance is a good workaround for the issue.

Cheers,
Mark

from snow-owl.