configuration about fluent-plugin-aws-elasticsearch-service HOT 44 CLOSED

I've managed to make this work and wanted to share my findings.

Firstly, assigning no policy to the ES cluster and doing all policy assertions in the EC2 instance IAM role will not work. I suspect this is rooted in the issues various ruby projects have been having with the move to v4 IAM request signing. Whilst this plugin seems to have a dependency chain that includes versions explicitly released to work around this for whatever reason it just doesn't seem to work.

However, it is still possible to have credential-less access.

The first step is to assign an IAM instance role ROLE to your EC2 instances. Name it appropriately. The role should contain no policy: we're using the possession of the role as the authenticating factor and placing the policy against the ES cluster, thereby avoiding fluentd having to use the SDK to sign requests.

You should then configure the cluster policy thus, with appropriate substitutions for the capitalized terms:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT:role/ROLE"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:ACCOUNT:domain/ES_DOMAIN/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:ACCOUNT:domain/ES_DOMAIN/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "1.2.3.4/32",
            "5.6.7.8/32"
          ]
        }
      }
    }
  ]
}

Be sure to amend the region from eu-west-1 to the appropriate region for your cluster.

This will allow your fluentd hosts (by virtue of the possession of the role) and any traffic coming from the specified IP addresses (you querying Kibana) to access the various endpoints. Whilst not ideally secure (both the fluentd and Kibana boxes should ideally be restricted to the verbs they require) it should allow you to get up and ingesting logs without anything getting in your way, before you tighten down the policy.

Plugin configuration suitable for using with this policy should look something like this:

    type "aws-elasticsearch-service"
    logstash_format true
    include_tag_key true
    tag_key "@log_name"
    flush_interval 5s

    <endpoint>
      url https://CLUSTER_ENDPOINT_URL
      region eu-west-1
    </endpoint>

from fluent-plugin-aws-elasticsearch-service.

@Suresh7 @servomac
For me the issue was extra slash at the end of endpoint url. I removed that slash and the plugin works for me.
When you copy the endpoint url from elastic search service it has a trailing slash. Remove that slash before putting the endpoint url.

For Problems related to policy.
I created a new group and add my user to that group, then instead of setting policy for group I set the policy for the elasticsearch service and allow the access to my IAM group.

from fluent-plugin-aws-elasticsearch-service.

I was able to post directly from fluentd agent to aws-es without using IP address as whitelist and using IAM role.

Resolution

• Make sure the end point does not end with backslash
• Assign permissions on the IAM Policy (thats only for ease of use as it takes longer for ES policy to update)

from fluent-plugin-aws-elasticsearch-service.

I have submitted #8 to include a version of the above in the README and to correct the original source of this thread: the ruby formatting of the fluentd configuration required.

from fluent-plugin-aws-elasticsearch-service.

after a lot of debugging following error comes up by calling "/usr/sbin/td-agent" manually..

I'm sure, that I have correct IAM credentials configured in endpoint.

2015-10-23 15:38:51 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-23 15:38:52 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1be2f34"
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in `__rescue_from_not_found'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:in `client'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in `rescue in send'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:in `send'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in `write'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:in `write_chunk'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in `pop'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:in `try_flush'
  2015-10-23 15:38:51 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'

from fluent-plugin-aws-elasticsearch-service.

I have debugged more .. and in connection string seems the the secret is missing? or could it be misunderstanding the authorization way..

{:host=>"search-......eu-central-1.es.amazonaws.com", 
:port=>80, :scheme=>"http", :path=>"/", :aws_elasticsearch_service=> {:credentials=>#<Aws::Credentials access_key_id="AKIAI....KQ">, :region=>"eu-central-1"}}

thank you for any help

from fluent-plugin-aws-elasticsearch-service.

hi @fragtom .

it was so as not to "warning: toplevel constant Elasticsearch referenced".

"[warn]: temporarily failed to flush the buffer." is I do not know yet well ...
But, response message from the AWS is I am sure the "The security token included in the request is expired".
So, I removed the memorize of credentials.

Please try using the "v0.1.2".

thanks!

from fluent-plugin-aws-elasticsearch-service.

thank you for taking up the issue.. the warning isn't present now..

2015-10-24 20:22:19 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-10-24 20:22:19 +0000 [info]: starting fluentd-0.12.12
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.2'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.1'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.1.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '0.9.0'
2015-10-24 20:22:19 +0000 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
..
2015-10-24 20:22:19 +0000 [info]: adding match pattern="reform.**" type="record_reformer"
2015-10-24 20:22:19 +0000 [info]: adding match pattern="kubernetes.**" type="aws-elasticsearch-service"
2015-10-24 20:22:19 +0000 [info]: adding source type="tail"

but the 403 .. seems to wrong credentials...

2015-10-24 20:22:45 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-24 20:22:46 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:179e20c"
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:in `perform_request'
  2015-10-24 20:22:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request

what about the result of debugging statement ... could it be possible, that elasticsearch "connection" is not fully filled.. or is that an misunderstanding of mine? I cannot see "secret_access_key" ..

:aws_elasticsearch_service=> {:credentials=>#<Aws::Credentials access_key_id="AKIAI....KQ">, :region=>"eu-central-1"}}

from fluent-plugin-aws-elasticsearch-service.

There may be a problem with the "faraday_middleware-aws-signers-v4".
It is being check by applying the current patch.

I'm sorry, it is likely to take still time to fix...

from fluent-plugin-aws-elasticsearch-service.

I'm so sorry, but on that issue I couldn't send signed request successfully.. Sadly I'm not the ruby expert to add debug statements on the right position to follow up the request .. it seems all ok with your plugin from my perspective .. If you have any things I could do for you, don't hesitate.

 {:host=>"search-....eu-central-1.es.amazonaws.com", :port=>443, :scheme=>"https", :path=>"/", :aws_elasticsearch_service=>{:credentials=>#<Proc:0x0000000075aaf8@/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-aws-elasticsearch-service-0.1.2/lib/fluent/plugin/out_aws-elasticsearch-service.rb:58 (lambda)>, :region=>"eu-central-1"}, :protocol=>"https"}

from fluent-plugin-aws-elasticsearch-service.

Please try using the "v0.1.3".
"[warn]: temporarily failed to flush the buffer." should have been fixed.

thanks!

from fluent-plugin-aws-elasticsearch-service.

I've tested without success.. I'll post you debug output..

JFYI: thats our docker image for that.
https://hub.docker.com/r/organizeme/docker-fluentd-elasticsearch-aws/

2015-10-28 10:00:55 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-10-28 10:00:55 +0000 [info]: starting fluentd-0.12.12
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.3'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.1.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '0.9.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.9'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-td' version '0.10.27'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-10-28 10:00:55 +0000 [info]: gem 'fluentd' version '0.12.12'
2015-10-28 10:00:55 +0000 [info]: adding match pattern="reform." type="record_reformer"
2015-10-28 10:00:55 +0000 [info]: adding match pattern="kubernetes." type="aws-elasticsearch-service"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: adding source type="tail"
2015-10-28 10:00:55 +0000 [info]: using configuration file:

type tail format json_in_json time_key time path /varlog/containers/_.log pos_file /varlog/es-containers.log.pos time_format %Y-%m-%dT%H:%M:%S tag reform._ read_from_head true type record_reformer enable_ruby true tag kubernetes.${tag_suffix[3].split('-')[0..-2].join('-')} type tail format none path /varlog/monit.log pos_file /varlog/gcp-monit.log.pos tag kubernetes.monit type tail format none path /varlog/salt/minion pos_file /varlog/gcp-salt.pos tag kubernetes.salt type tail format none path /varlog/startupscript.log pos_file /varlog/es-startupscript.log.pos tag kubernetes.startupscript type tail format none path /varlog/docker.log pos_file /varlog/es-docker.log.pos tag kubernetes.docker type tail format none path /varlog/etcd.log pos_file /varlog/es-etcd.log.pos tag kubernetes.etcd type tail format none path /varlog/kubelet.log pos_file /varlog/es-kubelet.log.pos tag kubernetes.kubelet type tail format none path /varlog/kube-apiserver.log pos_file /varlog/es-kube-apiserver.log.pos tag kubernetes.kube-apiserver type tail format none path /varlog/kube-controller-manager.log pos_file /varlog/es-kube-controller-manager.log.pos tag kubernetes.kube-controller-manager type tail format none path /varlog/kube-scheduler.log pos_file /varlog/es-kube-scheduler.log.pos tag kubernetes.kube-scheduler type aws-elasticsearch-service logstash_format true include_tag_key true tag_key @log_name log_level info flush_interval 5s buffer_chunk_limit 512K buffer_queue_limit 128 max_retry_wait 300 url https://search------.eu-central-1.es.amazonaws.com/ region eu-central-1 access_key_id --- secret_access_key ---- 2015-10-28 10:00:55 +0000 [info]: following tail of /varlog/containers/fluentd-elasticsearch-172.31.5.40_kube-system_fluentd-elasticsearch-2f04135698fff595652f649ab2f840adfbabed408acefc190d762a062f228520.log 2015-10-28 10:00:55 +0000 [info]: following tail of /varlog/containers/fluentd-elasticsearch-172.31.5.40_kube-system_POD-73b4a8627ad690072517d4a5201b3cb6374b09847487a14598b2c1c4eb926849.log 2015-10-28 10:01:29 +0000 [info]: following tail of /varlog/monit.log

....

2015-10-28 10:02:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-28 10:02:33 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:1451748"
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in __raise_transport_error' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:227:inperform_request'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/faraday.rb:20:in perform_request' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:inperform_request'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:20:in block in ping' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/utils.rb:189:in__rescue_from_not_found'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/ping.rb:19:in ping' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:inclient'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in rescue in send' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:insend'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in write' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:inwrite_chunk'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in pop' 2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:intry_flush'
2015-10-28 10:02:32 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'

from fluent-plugin-aws-elasticsearch-service.

Error has immediately after starting the fluentd.
In this case, there is a possibility that has not been assigned a policy to IAM.
Please check the policies.

policy example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444210466000",
            "Effect": "Allow",
            "Action": [
                "es:ESHttpPost",
                "es:ESHttpGet",
                "es:ESHttpHead"
            ],
            "Resource": [
                "arn:aws:es:*:*:*"
            ]
        }
    ]
}

from fluent-plugin-aws-elasticsearch-service.

I've written policy on es-service, there I thought, that is no explicit policy in IAM is needed...

but for testing, I've allowed my user in aim .. but no data comes in logstash..

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"
    },

from fluent-plugin-aws-elasticsearch-service.

How policy on user ?

from fluent-plugin-aws-elasticsearch-service.

Ok, I would try ..

the user on IAM have the following policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"
    },

policy on elasticsearch service

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::MYACCOUNT:user/MY_ES_USER"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:domain/xtractorio/*"
    },

on a other environment the logstash-shipper collects data and stream it to es service via logstash-plugin, and there I have no IAM policy .. just the elasticsearch-policy..

just one difference between the environment is the region - eu-west-1 and eu-west-2..

could you give me a hint for debugging?

from fluent-plugin-aws-elasticsearch-service.

I've listened via tcpdump on port 80 (just for debugging)

HEAD // HTTP/1.1
User-Agent: Faraday v0.9.1
X-Amz-Date: 20151029T091329Z
Host: search--sdfsfd7tsz3rqmhtvi.eu-central-1.es.amazonaws.com
X-Amz-Content-Sha256: e3b0...8fc1c149afbf4c899....4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKdfskfjskldOA/20151029/eu-central-1/es/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=00ed1eda8ce90e6c27d4b3ee795f0cf8e90f0f17608554a71be58e5394c9f1e1
Content-Length: 0

HTTP/1.1 403 Forbidden
Content-Length: 788
Content-Type: application/json
Server: Jetty(8.1.12.v20130726)
x-amzn-RequestId: 51d800e3-7e1d-11e5-9215-29572a93ba20
Connection: keep-alive

from fluent-plugin-aws-elasticsearch-service.

important note: there is a difference between eu-west-1 and eu-central-1

I've used same setup and changed endpoint credentials

Content-Type: application/json; charset=UTF-8
Server: Jetty(8.1.12.v20130726)
Content-Length: 270
Connection: keep-alive
{"took":11,"errors":false,"items":[{"create":{"_index":"logstash-2015.10.29","_type":"fluentd","_id":"AVCzDTuooBdD20Kisvhf","_version":1,"status":201}},{"create":{"_index":"logstash-2015.10.29","_type":"fluentd","_id":"AVCzDTuooBdD20Kisvhg","_version":1,"status":201}}]}??1V??BBBo?6?B

I could give you some credentials for testing, if you need.. but in the past we have had differences between locations with signature-request, too. it seems, that eu-central-1 is more strict with s4-hmac signature process

from fluent-plugin-aws-elasticsearch-service.

Are different a resource of policy on user and a resource of policy on elasticsearch service.
Is this intended?

from fluent-plugin-aws-elasticsearch-service.

Yes, I've different user and various endpoints.. On both locations I use the same policy (changed only the arn's).

from fluent-plugin-aws-elasticsearch-service.

"Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"

Resource of policy on user is syntax error?
Please change the "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:domain:search/*" to try.

The documents :

Syntax:

arn:aws:es:<region>:<account_id>:domain:<domain-name>
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#d0e11445

from fluent-plugin-aws-elasticsearch-service.

Indeed, the syntax is documented different my approach. I've tested, but without success.

The syntax I've taken from template (inside the console.aws.com http://console.aws.com/), when I take the suggested syntax the grant to IP-range didn't work, it seems the documentation covered only arn for IAM users.

Am 30.10.2015 um 12:26 schrieb @ [email protected]:

"Resource": "arn:aws:es:eu-central-1:MYACCOUNT:searchdomain/*"

Resource of policy on user is syntax error?
Please change the "Resource": "arn:aws:es:eu-central-1:MYACCOUNT:domain:search/*" to try.

The documents :

Syntax:

arn:aws:es::<account_id>:domain:
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#d0e11445 https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#d0e11445
—
Reply to this email directly or view it on GitHub #2 (comment).

from fluent-plugin-aws-elasticsearch-service.

I am using the latest code and getting the Forbidden issue. I've provided the correct IAM user credentials.
There is a strange issue it pushed one index correctly on my amazon elastic service, but now it's failing.

2015-11-09 03:59:34 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2015-11-09 03:59:34 +0000 [info]: following tail of /etc/apache2/logs/local_api_access.log
2015-11-09 04:04:45 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-11-09 04:04:46 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f9fffb21b6c"
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:136:in `__raise_transport_error'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:228:in `perform_request'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:in `perform_request'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:191:in `__rescue_from_not_found'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:in `client'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in `rescue in send'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:in `send'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in `write'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:in `write_chunk'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in `pop'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:in `try_flush'
  2015-11-09 04:04:45 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'
2015-11-09 04:04:46 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-11-09 04:04:48 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f9fffb21b6c"
  2015-11-09 04:04:46 +0000 [warn]: suppressed same stacktrace

I even tried changing the policy and permit access to all
and I got this issue

2015-11-09 04:37:41 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-11-09 04:37:12 +0000 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:3fba62cc1530"
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.1.35/lib/aws-sdk-core/signers/v4.rb:96:in `+'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.1.35/lib/aws-sdk-core/signers/v4.rb:96:in `signature'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.1.35/lib/aws-sdk-core/signers/v4.rb:86:in `authorization'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.1.35/lib/aws-sdk-core/signers/v4.rb:38:in `sign'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-aws-elasticsearch-service-0.1.3/lib/fluent/plugin/out_aws-elasticsearch-service.rb:163:in `sign'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday_middleware-aws-signers-v4-0.1.4/lib/faraday_middleware/request/aws_signers_v4.rb:57:in `call'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `call'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `perform_request'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:in `perform_request'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:20:in `block in ping'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:191:in `__rescue_from_not_found'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:19:in `ping'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:in `client'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in `rescue in send'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:in `send'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in `write'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:in `write_chunk'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in `pop'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:in `try_flush'
  2015-11-09 04:37:41 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'
2015-11-09 04:38:10 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-11-09 04:37:14 +0000 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:3fba62cc1530"
  2015-11-09 04:38:10 +0000 [warn]: suppressed same stacktrace

from fluent-plugin-aws-elasticsearch-service.

I've successfully post index to the aws elastic search using this plugin.

from fluent-plugin-aws-elasticsearch-service.

@mzahidriaz
Thank you for the report!

from fluent-plugin-aws-elasticsearch-service.

Unable to use this plugin, Error:

2015-12-02 14:07:16 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-02 14:07:17 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f86b8f81adc"
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:136:in __raise_transport_error' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:228:inperform_request'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in perform_request' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:inperform_request'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:20:in block in ping' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/utils.rb:191:in__rescue_from_not_found'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/ping.rb:19:in ping' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:69:inclient'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:176:in rescue in send' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:174:insend'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.1.0/lib/fluent/plugin/out_elasticsearch.rb:168:in write' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:325:inwrite_chunk'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:304:in pop' 2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:321:intry_flush'
2015-12-02 14:07:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/output.rb:140:in `run'
2015-12-02 14:07:17 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-02 14:07:19 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f86b8f81adc"
2015-12-02 14:07:17 +0000 [warn]: suppressed same stacktrace
2015-12-02 14:07:19 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-02 14:07:22 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f86b8f81adc"
2015-12-02 14:07:19 +0000 [warn]: suppressed same stacktrace
2015-12-02 14:07:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-02 14:07:30 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f86b8f81adc"
2015-12-02 14:07:23 +0000 [warn]: suppressed same stacktrace
2015-12-02 14:07:30 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-02 14:07:46 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:3f86b8f81adc"
2015-12-02 14:07:30 +0000 [warn]: suppressed same stacktrace

from fluent-plugin-aws-elasticsearch-service.

@mzahidriaz Can you please share, how did you successfully implemented.
My IAM-Role as Admin permissions, My ES Access policy looks fine and I am passing credentials in the fluentd configuration, but still I got 403 Error

from fluent-plugin-aws-elasticsearch-service.

@Suresh7 Do you set type_name in the configuration?

from fluent-plugin-aws-elasticsearch-service.

Opening the access to anyone I can insert data into the elasticsearch service. Using the access_key_id and secret_access_key of a user, I have the same problem of access permissions (403):

Software versions:

[ec2-user@ip-10-0-0-34 conf]$ gem list | grep aws
aws-sdk (2.2.5)
aws-sdk-core (2.2.5)
aws-sdk-resources (2.2.5)
faraday_middleware-aws-signers-v4 (0.1.1)
fluent-plugin-aws-elasticsearch-service (0.1.4)
[ec2-user@ip-10-0-0-34 conf]$ fluentd --version
fluentd 0.12.18

Access policy for elasticsearch:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::$ID:user/thumbor"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:$ID:domain/elasticsearch/*"
    }
  ]
}

Fluentd aws-elasticsearch-service endpoint:

  <endpoint>
    url                 "https://$ENDPOINT/"
    region              "eu-west-1"
    access_key_id       "$ACCESS_KEY_ID"
    secret_access_key   "$SECRET_ACCESS_KEY"
  </endpoint>

Also, the IAM user i'm using (thumbor) has this as part of its custom policy:

        {
            "Sid": "AllowElasticsearch",
            "Effect": "Allow",
            "Action": "es:*",
            "Resource": "arn:aws:es:eu-west-1:$ID:domain/elasticsearch/*"
        }

from fluent-plugin-aws-elasticsearch-service.

Sorry to add to this conversation but I'm getting 403's and I'm a bit stumped, I'm using

~ $ fluentd --version
fluentd 0.12.23

Here is my IAM policy which is set in the ElasticSearch service

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<acount_id>:role/role" }, "Action": "es:*", "Resource": "arn:aws:es:eu-central-1:<account_id>:domain/<domain>/*" }, { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Resource": "arn:aws:es:eu-central-1:<account_id>:domain/<domain>/*", "Condition": { "IpAddress": { "aws:SourceIp": "<my_ip>/32" } } } ] }

The role is attached to my instance, however this role has no ES specific policies however it does have ECS policies. I presume this won't cause issues?

apologies @sampointer some weird formatting issue before, my IAM policy matches the example from what I can see

from fluent-plugin-aws-elasticsearch-service.

You're missing a bunch of wildcards. "Action": "es:", should be "Action": "es:*",and also "Resource": "arn:aws:es:eu-central-1:ACCOUNT:domain/domain/" should be "Resource": "arn:aws:es:eu-central-1:ACCOUNT:domain/domain/*"

See the example here: https://github.com/atomita/fluent-plugin-aws-elasticsearch-service#iam

from fluent-plugin-aws-elasticsearch-service.

The presence of the ECS policies shouldn't matter, as the IAM example uses the possession of the named role as the authenticating factor in the ES cluster IAM policy.

You say you're getting 403s. From where and hitting which url?

Not to nit-pick on formatting, but where above you say "Principal": { "AWS": "arn:aws:iam::<acount_id>:role/role" I presume you mean "Principal": { "AWS": "arn:aws:iam::<acount_id>:role/<role>" ?

Is the source host an ECS container or an EC2 instance?

from fluent-plugin-aws-elasticsearch-service.

I'm hitting the https endpoint - sorry for the confusion I do mean "Principal": { "AWS": "arn:aws:iam::<acount_id>:role/<role>"

it's a ECS container, endpoint configuration looks like

<endpoint> url https://<my_es>.eu-central-1.es.amazonaws.com region eu-cental-1 access_key_id secret_access_key </endpoint>

from fluent-plugin-aws-elasticsearch-service.

I'm having the same issue.

When I'm permitting my FLUENTD-ECS-TASK IAM role and my instance role aws-elasticsearch policy it's not working.

But by just adding the fluentd-instance public IP to the aws-ealsticsearch policy it immediately works.

I think it's a request signature issue.

from fluent-plugin-aws-elasticsearch-service.

What does the IAM policy simulator tell you?

from fluent-plugin-aws-elasticsearch-service.

The IAM role based policy permission is not working

It works only if I'm adding the fluentd public IP then the plugin can send request to AWS-ES

Following is my config:

My ES-DOMAIN policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::xxxxxxxxxxxx:role/fluentd-ecs-task-role",
          "arn:aws:iam::xxxxxxxxxxxx:role/fluentd-ec2-instace-role"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:xxxxxxxxxxxx:domain/my-domain/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:xxxxxxxxxxxx:domain/my-domain/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "FLUENTD_EC2_PUBLIC_IP"
        }
      }
    }
  ]
}

My fluentd config

<match fluent.**>
  @type null
</match>
<source>
  @type forward
  port 24224
  bind 0.0.0.0
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
  read_from_head true
</source>
<filter docker.**>
  @type concat
  key message
  stream_identity_key container_id
  multiline_start_regexp /^-e:2:in '\/'/
  multiline_end_regexp /^-e:4:in/
</filter>
<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>
<match **>
  @type aws-elasticsearch-service
  @log_level info
  include_tag_key true
  logstash_format true
  logstash_prefix logstash
  buffer_type memory
  buffer_chunk_limit 8m
  buffer_queue_limit 8192
  flush_interval 10s
  retry_limit 10
  disable_retry_limit
  retry_wait 1s
  max_retry_wait 60s
  num_threads 8
  reload_connections false
  resurrect_after 5
  <endpoint>
    url "${AWS_ELASTICSEARCH_ENDPOINT}"
    region "${AWS_ELASTICSEARCH_REGION}"
  </endpoint>
</match>

gem list

*** LOCAL GEMS ***

aws-sdk (2.5.11)
aws-sdk-core (2.5.11)
aws-sdk-resources (2.5.11)
cool.io (1.4.5)
did_you_mean (1.0.0)
elasticsearch (1.0.18)
elasticsearch-api (1.0.18)
elasticsearch-transport (1.0.18)
excon (0.52.0)
faraday (0.9.2)
faraday_middleware-aws-signers-v4 (0.1.1)
fluent-plugin-aws-elasticsearch-service (0.1.6)
fluent-plugin-concat (0.6.2)
fluent-plugin-elasticsearch (1.7.0)
fluent-plugin-secure-forward (0.4.3)
fluentd (0.12.29)
http_parser.rb (0.6.0)
jmespath (1.3.1)
json (2.0.2)
minitest (5.8.3)
msgpack (0.5.12)
multi_json (1.12.1)
multipart-post (2.0.0)
net-telnet (0.1.1)
oj (2.17.4)
power_assert (0.2.6)
proxifier (1.0.3)
psych (2.0.17)
rake (10.4.2)
resolve-hostname (0.0.4)
sigdump (0.2.4)
string-scrub (0.0.5)
test-unit (3.1.5)
thread_safe (0.3.5)
tzinfo (1.2.2)
tzinfo-data (1.2016.6)
yajl-ruby (1.2.1)

Fluentd version

bash-4.3#  fluentd --version
fluentd 0.12.29

from fluent-plugin-aws-elasticsearch-service.

What does the IAM policy simulator tell you about your policy? I've ingested way over a billion log items through IAM authentication: it works if you get the IAM correct.

There's a commenter above who also had problems with ECS and I see mentions of Docker in your setup.

from fluent-plugin-aws-elasticsearch-service.

I managed to solve this a few months ago. I didn't exactly remember what solved it (sorry :p), but my elasticsearch access policy has something like:

    {
      "Sid": "AllowFluentdUsingThumborUser",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ID:user/thumbor"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:ID:domain/logs/*"
    },
    {
      "Sid": "AllowFluentdUsingECS",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ID:role/ecsServiceRole"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:ID:domain/logs/*"
    },

And i'm using the access keys of the thumbor user. The gem list:

# gem list | grep aws
aws-sdk (2.2.9)
aws-sdk-core (2.2.9)
aws-sdk-resources (2.2.9)
faraday_middleware-aws-signers-v4 (0.1.1)
fluent-plugin-aws-elasticsearch-service (0.1.4)

from fluent-plugin-aws-elasticsearch-service.

@sampointer the IAM policy simulator does'nt have option to similate elasticsearch indices usage, it's limited the aws elasticserach domain management

from fluent-plugin-aws-elasticsearch-service.

Same as @mzahidriaz the problems is the slash at the end of endpoint url

from fluent-plugin-aws-elasticsearch-service.

I was getting 403 and was using:
url http://<my_es>.eu-west-1.es.amazonaws.com
when I changed it to
url https://<my_es>.eu-west-1.es.amazonaws.com
It was ok, so use https not http 💀
But it's in the README so shame on me.

from fluent-plugin-aws-elasticsearch-service.

The backslash removal worked for me too! thanks @adilnaimi

from fluent-plugin-aws-elasticsearch-service.

How many hours of people time have been wasted over bad logging when the URL ends with / ? You need to log the URL of the request that returns 403, that would ease debugging of this enormously.

from fluent-plugin-aws-elasticsearch-service.

Trailing backslash warning mechanism has been introduced in #67.
User can notice raised warnings.
We can close this issue.

from fluent-plugin-aws-elasticsearch-service.