Unable to handle ElasticSearch "RequestEntityTooLarge" error correctly about fluent-plugin-aws-elasticsearch-service HOT 5 OPEN

* Log this as an "error" not a "warning"?

We can't do it. This warning is generated in Fluentd core.

If the plugin receives this error back from ES, write the "chunk" to disk somewhere (dead letter queue?) and "move on"

Adding 413 HTTP status code handling in ES plugin can achieve your feature request. WDYT?

from fluent-plugin-aws-elasticsearch-service.

can we make this as configurable? since I know the maximum size of HTTP Request Payloads varies with different instance type.

from fluent-plugin-aws-elasticsearch-service.

Adding 413 HTTP status code handling in ES plugin can achieve your feature request. WDYT?

Yep, sounds good to me! What are your thoughts as to how it should behave if it receives this message back? Should it just give up/delete that chunk? Is there any way to split it and resend or put it somewhere and move to the next chunk?

from fluent-plugin-aws-elasticsearch-service.

Supported splitting huge chunk requests (> 20MB) feature on ES plugin side:
uken/fluent-plugin-elasticsearch#535

Should we this feature configurable?

from fluent-plugin-aws-elasticsearch-service.

hi, any updates ? @cosmo0920 did you have a chance to fix the issue ?

from fluent-plugin-aws-elasticsearch-service.