Problem Unable to send logs to AWS ES and S3 at same time and noth

Yeah, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-ur

Thanks for the reply <a class="user-mention notranslate" data-hovercard-type="user" da

This issue should be continued to <a class="issue-link js-issue-link" data-error-text=

Query : Aws elastic search plugin will support multiple outputs(i.e s3) about fluent-plugin-aws-elasticsearch-service HOT 4 CLOSED

atomita commented on August 17, 2024

Query : Aws elastic search plugin will support multiple outputs(i.e s3)

from fluent-plugin-aws-elasticsearch-service.

Comments (4)

jayslife commented on August 17, 2024

Hello,

Its been awhile since i've worked with this plug-in and I remember having a similar issue. Take a look at the copy plugin https://docs.fluentd.org/output/copy. This is mostly likely what your looking for.

Jason

from fluent-plugin-aws-elasticsearch-service.

cosmo0920 commented on August 17, 2024

Yeah, @jayslife answer is right.
AWS ES Output plugin should emit nothing.
Instead, we can reprecate Fluentd events with copy plugin.

from fluent-plugin-aws-elasticsearch-service.

nareshnani529 commented on August 17, 2024

Thanks for the reply @cosmo0920 @jayslife

I tried with Copy plugin but not working. Could you give us an example here that will be great!

Config that I tried with :

<source>
        @type forward
        @id input_forward
</source>

<filter *.**>
  @type record_transformer
 <record>
    container_id ${record["container_id"]}
  </record>
</filter>

<match *.**>
        type aws-elasticsearch-service
        logstash_format true
        logstash_prefix ${tag[0]}
        include_tag_key true
        tag_key "Application_name"
        flush_interval 5s

    <buffer>
      flush_at_shutdown true
      flush_mode immediate
      flush_thread_count 8
      flush_thread_interval 1
      flush_thread_burst_interval 1
      retry_forever true
      retry_type exponential_backoff
    </buffer>

        <endpoint>
                url https://vpc-xxxx.us-east-1.es.amazonaws.com
                region us-east-1
                access_key_id "xxxx"
                secret_access_key "xxxx"
        </endpoint>
</match>

<match *.**>
type copy
      <store>
        type s3
        aws_key_id "xxxxx"
        aws_sec_key "xxxx"
        s3_bucket "logstream-testbucket"
        s3_region "us-east-1"
        path logs/
        buffer_path /var/log/td-agent/buffer/s3
        time_slice_format %Y-%m-%d/%H
        time_slice_wait 10m
        </store>
        </match>