sampointer / fluent-plugin-cloudwatch-ingest Goto Github PK

Hi,
I am getting the following error when I try to install this plugin on fluentds docker container

ERROR: Error installing fluent-plugin-cloudwatch-ingest:
ERROR: Failed to build gem native extension.

current directory: /usr/lib/ruby/gems/2.3.0/gems/strptime-0.1.9/ext/strptime

/usr/bin/ruby -r ./siteconf20171009-5-eusas6.rb extconf.rb
checking for rb_timespec_now()... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/bin/$(RUBY_BASE_NAME)
/usr/lib/ruby/2.3.0/mkmf.rb:456:in try_do': The compiler failed to generate an executable file. (RuntimeError) You have to install development tools first. from /usr/lib/ruby/2.3.0/mkmf.rb:541:in try_link0'
from /usr/lib/ruby/2.3.0/mkmf.rb:556:in try_link' from /usr/lib/ruby/2.3.0/mkmf.rb:765:in try_func'
from /usr/lib/ruby/2.3.0/mkmf.rb:1051:in block in have_func' from /usr/lib/ruby/2.3.0/mkmf.rb:942:in block in checking_for'
from /usr/lib/ruby/2.3.0/mkmf.rb:350:in block (2 levels) in postpone' from /usr/lib/ruby/2.3.0/mkmf.rb:320:in open'
from /usr/lib/ruby/2.3.0/mkmf.rb:350:in block in postpone' from /usr/lib/ruby/2.3.0/mkmf.rb:320:in open'
from /usr/lib/ruby/2.3.0/mkmf.rb:346:in postpone' from /usr/lib/ruby/2.3.0/mkmf.rb:941:in checking_for'
from /usr/lib/ruby/2.3.0/mkmf.rb:1050:in have_func' from extconf.rb:3:in

To see why this extension failed to compile, please check the mkmf.log which can be found here:

/usr/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/strptime-0.1.9/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /usr/lib/ruby/gems/2.3.0/gems/strptime-0.1.9 for inspection.
Results logged to /usr/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/strptime-0.1.9/gem_make.out

Do you think there's a way to use this plugin with the fluent-kubernetes-daemonset images instead of the one provided by default (https://github.com/ryotarai/fluent-plugin-cloudwatch-logs).

I'm having issues integrating it with CloudWatch and it seems like the author isn't worried about addressing an issue that makes it unusable (it least for me and a couple of people that have reported the issue).

I want to build a new image but I have no idea of how well will your plugin integrate with fluent-kubernetes-daemonset. Have you tried this or are you just using another image altogether?

Thanks in advance

Ideally this should be in two parts.

Functional

Configure fluentd such that it reads from a real log group and writes to a file or stdout, which can then be asserted on

Unit

To be done in tandem with a refactoring exercise. The run method should be pared down to a minimum. As functionality is moved out, it should be accompanied by tests.

Hi,

I am having issues with the event time from cloudwatch. It appears that fluentd takes the time when the events are received by fluentd and not when they are received by cloudwatch.

I have tested this by removing my state file and let fluentd add a lot of events again. They all had the same timestamp (while it were events of 4 days)

My config:

<source>
  @type cloudwatch_ingest
  region eu-central-1
  aws_logging_enabled true
  limit_events 10000
  state_file_name /var/log/fluentd/cloudwatch-security.state
  interval 30
  api_interval 5          # Time to wait between API call failures before retry
  limit_events 10000      # Number of events to fetch in any given iteration
  oldest_logs_first false  # When true fetch the oldest logs first
  log_group_name_prefix Server-SecurityLog

  tag eventlog.security

  <parse>
    @type cloudwatch_ingest
    expression /^(?<raw_message>.*)$/
    multiline true
    time_format %Y-%m-%d %H:%M:%S.%L
    event_time true         # take time from the Cloudwatch event, rather than parse it from the body
    inject_group_name false # inject the group name into the record
    inject_stream_name true # inject the stream name into the record
  </parse>

</source>

edit: the xml was not shown correctly :-)

Hello,

Has anyone managed to run this plugin on fluentd version above 1.4? I constantly get an error for any version from 14.1 to the latest 1.7.4. I believe it is related to ruby version. Unfortunately, I'm not very familiar with ruby, so can't be sure.
Here is an error:

fluentd/gems/fluentd-1.4.1/lib/fluent/plugin/parser_regexp.rb:44:in `configure': undefined method `named_captures' for "/^(?<message>.+)$/":String (NoMethodError)
        from /opt/bitnami/fluentd/gems/fluent-plugin-cloudwatch-ingest-1.7.0/lib/fluent/plugin/parser_cloudwatch_ingest.rb:29:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/plugin.rb:164:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/plugin_helper/parser.rb:90:in `block in configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/plugin_helper/parser.rb:85:in `each'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/plugin_helper/parser.rb:85:in `configure'
        from /opt/bitnami/fluentd/gems/fluent-plugin-cloudwatch-ingest-1.7.0/lib/fluent/plugin/in_cloudwatch_ingest.rb:86:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/plugin.rb:164:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/root_agent.rb:320:in `add_source'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/root_agent.rb:160:in `block in configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/root_agent.rb:156:in `each'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/root_agent.rb:156:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/engine.rb:131:in `configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/engine.rb:96:in `run_configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/supervisor.rb:801:in `run_configure'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/supervisor.rb:579:in `dry_run'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/supervisor.rb:597:in `supervise'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/supervisor.rb:502:in `run_supervisor'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
        from /opt/bitnami/ruby/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/bitnami/ruby/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/bitnami/fluentd/gems/fluentd-1.4.1/bin/fluentd:8:in `<top (required)>'
        from /opt/bitnami/fluentd/bin/fluentd:23:in `load'
        from /opt/bitnami/fluentd/bin/fluentd:23:in `<main>'

After February 22nd 2018 I will no longer be running this plugin in production myself, instead preferring Kinesis to AWS ES.

Rather than let this code rot and leave those who have invested time without a canonical version I'd instead welcome maintainers to whom commit and publish bits could be granted.

If you're a user of this plugin and would be willing to entertain such a thing please comment below. I'd be very happy to count thumbs-ups as votes of confidence.

Failing that, I'd probably consider 1.7.0 coming out of RC (whenever that might be) an opportune moment to make your own fork.

Should a fork become dominant I'd be happy to hand over the rubygems namespace to the obvious candidate.

It is a common pattern to use today's date as the log group name prefix, given this is the default assigned by AWS Lambda without explicit configuration.

Lambdas can survive to be re-used for up to 6 hours after their initial invocation, if invoked frequently. If this is the case they will continue to log to "yesterday's" log group prefix.

Functionality is required to:

• not rely on an external actor, such as a configuration management system, to amend the log_stream_name_prefix.
• introduce logic that will straddle log groups prefixed with today and yesterday's date for a configurable amount of time to ensure that logs from long-running lambdas are not missed.

2017-11-25 01:51:44 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'api_interval' parameter is required"
2017-11-25 01:52:08 +0000 [warn]: api_interval is deprecated for error_interval
2017-11-25 01:52:09 +0000 [warn]: #0 api_interval is deprecated for error_interval

Seems like the config or at least the config checks are inconsistent.

I am trying to install the plugin on an amazon ec2 instance. Getting the following error
`sudo td-agent-gem install fluent-plugin-cloudwatch-ingest
Building native extensions. This could take a while...
ERROR: Error installing fluent-plugin-cloudwatch-ingest:
ERROR: Failed to build gem native extension.

current directory: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/strptime-0.1.9/ext/strptime

/opt/td-agent/embedded/bin/ruby -r ./siteconf20171228-3075-ic69la.rb extconf.rb
checking for rb_timespec_now()... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.

Provided configuration options:
--with-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/opt/td-agent/embedded/bin/ruby
/opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:467:in try_do': The compiler failed to generate an executable file. (RuntimeError) You have to install development tools first. from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:552:in try_link0'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:567:in try_link' from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:747:in try_func'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:1032:in block in have_func' from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:923:in block in checking_for'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in block (2 levels) in postpone' from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in open'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:351:in block in postpone' from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:321:in open'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:347:in postpone' from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:922:in checking_for'
from /opt/td-agent/embedded/lib/ruby/2.1.0/mkmf.rb:1031:in have_func' from extconf.rb:3:in

To see why this extension failed to compile, please check the mkmf.log which can be found here:

/opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/strptime-0.1.9/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/strptime-0.1.9 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/strptime-0.1.9/gem_make.out`

Hi, the following api calls gets a rate limit exceeded throttling from AWS if you have too many log streams:

The throttling happens on "describe logs" and this limitation is hard capped (not possible to increase the number of calls per second towards that endpoint).

I made a pull request with some changes and I'm very open to criticism and or side effects I may have overlooked while doing it. I never did any real Ruby coding so more help is also appreciated.
#12