I have 2 different matches to push tu AWS ElasticSearch, one for audit logs and the other for payloads.
When I configure audit logs, it works fine, but when i configure payloads, I get the following result repeatedly :
2016-10-20 18:26:18 +0000 [info]: reading config file path="/fluentd/etc/custom.conf"
2016-10-20 18:26:18 +0000 [info]: starting fluentd-0.12.28
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.6'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-concat' version '0.6.2'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.6.0'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-kinesis' version '1.1.1'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-parser' version '0.6.1'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.2'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-s3' version '0.7.1'
2016-10-20 18:26:18 +0000 [info]: gem 'fluent-plugin-systemd' version '0.0.4'
2016-10-20 18:26:18 +0000 [info]: gem 'fluentd' version '0.12.28'
2016-10-20 18:26:18 +0000 [info]: adding filter pattern="*.**" type="record_transformer"
2016-10-20 18:26:18 +0000 [info]: adding match pattern="service_audit" type="copy"
2016-10-20 18:26:18 +0000 [info]: adding match pattern="service_payload" type="copy"
2016-10-20 18:26:18 +0000 [info]: adding match pattern="fluent.**" type="copy"
2016-10-20 18:26:18 +0000 [info]: adding source type="forward"
2016-10-20 18:26:18 +0000 [info]: using configuration file: <ROOT>
<filter *.**>
@type record_transformer
<record>
product_name service
environment_tag intqa102
region us-east-1
hostname ip-10-185-73-159.nonprod.xyz-aws.io
</record>
</filter>
<source>
@type forward
port 24224
</source>
<match service_audit>
type copy
<store>
@type s3
s3_bucket prd00000280.open_platform_logs.nonprod1.us-east-1.xyz-aws
path service/intqa102/audit/
buffer_path /fluentd/log/s3_buffer_audit
s3_object_key_format %{path}application_log_ip-10-185-73-159.nonprod.xyz-aws.io_%{time_slice}_%{index}.%{file_extension}
s3_region us-east-1
storage_class REDUCED_REDUNDANCY
time_slice_format %Y%m%d%H
utc
format json
include_time_key true
flush_at_shutdown true
</store>
</match>
<match service_payload>
type copy
<store>
@log_level debug
@type aws-elasticsearch-service
type_name payload_log
logstash_format true
logstash_prefix intqa102
include_tag_key true
tag_key @log_name
time_key_format %FT%T.%L%z
flush_interval 5s
<endpoint>
url http://logs.open.platform.nonprod1.nonprod-xyz-aws.io:80
region us-east-1
</endpoint>
</store>
<store>
@type s3
s3_bucket prd00000280.open_platform_logs.nonprod1.us-east-1.xyz-aws
path service/intqa102/payload/
buffer_path /fluentd/log/s3_buffer_payload
s3_object_key_format %{path}payload_log_ip-10-185-73-159.nonprod.xyz-aws.io_%{time_slice}_%{index}.%{file_extension}
s3_region us-east-1
storage_class REDUCED_REDUNDANCY
time_slice_format %Y%m%d%H
utc
format json
include_time_key true
flush_at_shutdown true
</store>
</match>
<match fluent.**>
type copy
<store>
@type stdout
</store>
<store>
@type s3
s3_bucket prd00000280.open_platform_logs.nonprod1.us-east-1.xyz-aws
path service/intqa102/fluent/
buffer_path /fluentd/log/s3_buffer_fluent
s3_object_key_format %{path}fluent_log_ip-10-185-73-159.nonprod.xyz-aws.io_%{time_slice}_%{index}.%{file_extension}
s3_region us-east-1
storage_class REDUCED_REDUNDANCY
time_slice_format %Y%m%d%H
utc
format json
include_time_key true
flush_at_shutdown true
</store>
</match>
</ROOT>
2016-10-20 18:26:19 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-10-20 18:28:19 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"logs.open.platform.nonprod1.nonprod-xyz-aws.io", :port=>80, :scheme=>"http", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="BSIEG3OXZUHFJVASPPEB">, :region=>"us-east-1"}}
2016-10-20 18:28:19 +0000 fluent.info: {"message":"Connection opened to Elasticsearch cluster => {:host=>\"logs.open.platform.nonprod1.nonprod-xyz-aws.io\", :port=>80, :scheme=>\"http\", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id=\"BSIEG3OXZUHFJVASPPEB\">, :region=>\"us-east-1\"}}","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:20 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)
2016-10-20 18:28:20 +0000 fluent.warn: {"message":"Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:22 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"logs.open.platform.nonprod1.nonprod-xyz-aws.io", :port=>80, :scheme=>"http", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="BSIEG3OXZUHFJVASPPEB">, :region=>"us-east-1"}}
2016-10-20 18:28:22 +0000 fluent.info: {"message":"Connection opened to Elasticsearch cluster => {:host=>\"logs.open.platform.nonprod1.nonprod-xyz-aws.io\", :port=>80, :scheme=>\"http\", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id=\"BSIEG3OXZUHFJVASPPEB\">, :region=>\"us-east-1\"}}","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:22 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)
2016-10-20 18:28:22 +0000 fluent.warn: {"message":"Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:26 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"logs.open.platform.nonprod1.nonprod-xyz-aws.io", :port=>80, :scheme=>"http", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="BSIEG3OXZUHFJVASPPEB">, :region=>"us-east-1"}}
2016-10-20 18:28:26 +0000 fluent.info: {"message":"Connection opened to Elasticsearch cluster => {:host=>\"logs.open.platform.nonprod1.nonprod-xyz-aws.io\", :port=>80, :scheme=>\"http\", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id=\"BSIEG3OXZUHFJVASPPEB\">, :region=>\"us-east-1\"}}","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:26 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-10-20 18:28:20 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Could not push logs to Elasticsearch after 2 retries. Broken pipe (Errno::EPIPE)" plugin_id="object:d37f0c"
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.6.0/lib/fluent/plugin/out_elasticsearch.rb:333:in `rescue in send'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.6.0/lib/fluent/plugin/out_elasticsearch.rb:323:in `send'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.6.0/lib/fluent/plugin/out_elasticsearch.rb:308:in `write'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluentd-0.12.28/lib/fluent/buffer.rb:354:in `write_chunk'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluentd-0.12.28/lib/fluent/buffer.rb:333:in `pop'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluentd-0.12.28/lib/fluent/output.rb:338:in `try_flush'
2016-10-20 18:28:26 +0000 [warn]: /var/lib/gems/2.1.0/gems/fluentd-0.12.28/lib/fluent/output.rb:149:in `run'
2016-10-20 18:28:26 +0000 fluent.warn: {"next_retry":"2016-10-20 18:28:20 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Could not push logs to Elasticsearch after 2 retries. Broken pipe (Errno::EPIPE)","plugin_id":"object:d37f0c","message":"temporarily failed to flush the buffer. next_retry=2016-10-20 18:28:20 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Could not push logs to Elasticsearch after 2 retries. Broken pipe (Errno::EPIPE)\" plugin_id=\"object:d37f0c\"","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:26 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)
2016-10-20 18:28:26 +0000 fluent.warn: {"message":"Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:28 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"logs.open.platform.nonprod1.nonprod-xyz-aws.io", :port=>80, :scheme=>"http", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="BSIEG3OXZUHFJVASPPEB">, :region=>"us-east-1"}}
2016-10-20 18:28:28 +0000 fluent.info: {"message":"Connection opened to Elasticsearch cluster => {:host=>\"logs.open.platform.nonprod1.nonprod-xyz-aws.io\", :port=>80, :scheme=>\"http\", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id=\"BSIEG3OXZUHFJVASPPEB\">, :region=>\"us-east-1\"}}","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:28 +0000 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)
2016-10-20 18:28:28 +0000 fluent.warn: {"message":"Could not push logs to Elasticsearch, resetting connection and trying again. Broken pipe (Errno::EPIPE)","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
2016-10-20 18:28:32 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>"logs.open.platform.nonprod1.nonprod-xyz-aws.io", :port=>80, :scheme=>"http", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="BSIEG3OXZUHFJVASPPEB">, :region=>"us-east-1"}}
2016-10-20 18:28:32 +0000 fluent.info: {"message":"Connection opened to Elasticsearch cluster => {:host=>\"logs.open.platform.nonprod1.nonprod-xyz-aws.io\", :port=>80, :scheme=>\"http\", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id=\"BSIEG3OXZUHFJVASPPEB\">, :region=>\"us-east-1\"}}","product_name":"service","environment_tag":"intqa102","region":"us-east-1","hostname":"ip-10-185-73-159.nonprod.xyz-aws.io"}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent