Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable about fortinet-2-elasticsearch HOT 6 CLOSED

I have uninstalled logstash-input-twitter and installed logstash-filter-tld version 3.1.3 but still gettting the same error

from fortinet-2-elasticsearch.

Nov 16 09:53:47 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:47,277][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:48 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:48,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:49 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:49,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:50 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:50,278][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:51 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:51,279][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:52 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:52,279][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:53 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:53,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:54 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:54,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:55 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:55,280][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:56 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:56,281][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:57 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:57,281][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:58 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:58,282][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:53:59 zavpemblogs31 logstash[1948]: [2023-11-16T09:53:59,282][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:54:00 zavpemblogs31 logstash[1948]: [2023-11-16T09:54:00,283][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry. Nov 16 09:54:01 zavpemblogs31 logstash[1948]: [2023-11-16T09:54:01,283][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

from fortinet-2-elasticsearch.

Those logs are not really helpfull.

I need the logs when logstash starts. Normally what comes before/next "running pipelines/not running pipelines"

In my case, that lines is:

[2023-11-20T15:49:28,392][INFO ][logstash.agent ] Pipelines running {:count=>17, :running_pipelines=>[:"syslog-fortinet-fortiedr_2_ecs-default", :"syslog-fortinet-fortigate-input-kv-client1", :"syslog-fortinet-fortimail-input-kv-client2", :"syslog-fortinet-forticlient_2_ecs-client2", :"syslog-fortinet-forticlient-input-kv-client2", :"syslog-fortinet-fortimail_2_ecs-client2", :"syslog-fortinet-fortigate-input-kv-client35424", :"syslog-fortinet-fortiedr-input-kv-default", :"syslog-fortinet-fortigate-input-kv-client15424", :"syslog-fortinet-fortigate-input-kv-client25424", :"syslog-fortinet-fortigate_2_ecs-client1", :"syslog-fortinet-fortigate_2_ecs-client3", :"syslog-fortinet-fortigate_2_ecs-client2", :"syslog-fortinet-common_ecs-output-default", :"syslog-fortinet-common_ecs-output-client3", :"syslog-fortinet-common_ecs-output-client1", :"syslog-fortinet-common_ecs-output-client2"], :non_running_pipelines=>[]}

from fortinet-2-elasticsearch.

I get the same error:
[2023-12-09T20:38:02,243][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_internal:[email protected]:9200/]}}
[2023-12-09T20:38:02,459][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Restored connection to ES instance {:url=>"https://logstash_internal:[email protected]:9200/"}
[2023-12-09T20:38:02,460][INFO ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Elasticsearch version determined (8.11.1) {:es_version=>8}
[2023-12-09T20:38:02,460][WARN ][logstash.outputs.elasticsearch][syslog-fortinet-common_ecs-output] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2023-12-09T20:38:02,475][WARN ][logstash.filters.grok ][syslog-fortinet-common_ecs-output] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-12-09T20:38:02,520][WARN ][logstash.filters.grok ][syslog-fortinet-common_ecs-output] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2023-12-09T20:38:02,540][ERROR][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline error {:pipeline_id=>"syslog-fortinet-common_ecs-output", :exception=>#<ArgumentError: wrong number of arguments (given 2, expected 1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:69:in parse'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:51:in default'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-tld-3.1.2/lib/logstash/filters/tld.rb:33:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/syslog-fortinet-common_ecs-output.conf"], :thread=>"#<Thread:0x263d4cf6 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-12-09T20:38:02,541][INFO ][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline terminated {"pipeline.id"=>"syslog-fortinet-common_ecs-output"}
[2023-12-09T20:38:02,551][ERROR][logstash.agent ] Failed to execute action {:id=>:"syslog-fortinet-common_ecs-output", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<syslog-fortinet-common_ecs-output>, action_result: false", :backtrace=>nil}
[2023-12-09T20:38:03,067][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
[2023-12-09T20:38:04,068][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
[2023-12-09T20:38:05,068][WARN ][org.logstash.plugins.pipeline.PipelineBus][syslog-fortinet-fortigate_2_ecsv2][29a6aa27ca7002ac905931a3f66296c9a559f80ec562f0a6bc6cce6e7d356a3a] Attempted to send event to 'syslog-fortinet-common_ecs-output' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

As soon as i uncomment the TLD Lines in Output all works.

from fortinet-2-elasticsearch.

it does not seem you are using logstash-filter-tld --version 3.1.3

[2023-12-09T20:38:02,540][ERROR][logstash.javapipeline ][syslog-fortinet-common_ecs-output] Pipeline error {:pipeline_id=>"syslog-fortinet-common_ecs-output", :exception=>#<ArgumentError: wrong number of arguments (given 2, expected 1)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:69:in parse'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/public_suffix-3.1.1/lib/public_suffix/list.rb:51:in default'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-tld-3.1.2/lib/logstash/filters/tld.rb:33:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/syslog-fortinet-common_ecs-output.conf"], :thread=>"#<Thread:0x263d4cf6 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

after these steps

Running Logstash 8.10
Elasticsearch 8.10
Removed logstash-input-twitter
Installed logstash-filter-tld --version 3.1.3

please restart logstash. Probably you already did, but just want to make sure.

For some reason your plugin is not loading on version 3.1.3

from fortinet-2-elasticsearch.

Had to update the tld Plugin manually to version 3.1.3 (this command gave me 3.1.2: bin/logstash-plugin install logstash-filter-tld)

Now it works fine. My issue is resolved. Thanks very much for your support :)

By the way: very nice Solution!

from fortinet-2-elasticsearch.