Comments (3)
gamefundas
commented on July 30, 2024
Do we have a resolution for this? I am seeing the same failure for ES 2.x
from cloudwatch-logs-subscription-consumer.
danielcbright
commented on July 30, 2024
I was/am having the same issue and now I've gotten a little further, I'm not Java programmer so pardon my ignorance. But I've updated the pom.xml to reflect a newer version of elasticsearch as shown:
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.0.0</version>
</dependency>
This caused a new issue during my Maven install:
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project cloudwatch-logs-subscription-consumer: Compilation failure: Compilation failure:
[ERROR] /opt/cwlsc/src/main/java/com/amazonaws/services/logs/connectors/elasticsearch/CloudWatchLogsElasticsearchDocument.java:[19,38] package org.elasticsearch.common.lang3 does not exist
[ERROR] /opt/cwlsc/src/main/java/com/amazonaws/services/logs/connectors/elasticsearch/CloudWatchLogsElasticsearchDocument.java:[79,21] cannot find symbol
[ERROR] symbol: variable StringUtils
[ERROR] location: class com.amazonaws.services.logs.connectors.elasticsearch.CloudWatchLogsElasticsearchDocument
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
I found out that the org.apache.commons:commons-lang3 artifact that is included in 1.6.x - 1.7.x of the elasticsearch mvn artifact see pom.xml here is not in 2.x.x see the 2.0.0 pom.xml here
So I added this dependency to the pom.xml for cloudwatch-logs-subscription-consumer
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.3.2</version>
</dependency>
And I changed src/main/java/com/amazonaws/services/logs/connectors/elasticsearch/CloudWatchLogsElasticsearchDocument.java to use the org.apache.commons lang3 instead of elasticsearch:
/*import org.elasticsearch.common.lang3.StringUtils;*/
import org.apache.commons.lang3.StringUtils;
After doing this, I can do a mvn clean install without issues.
The new issue I've run into is this:
2016-01-21 13:36:59,309 ERROR Worker - Worker.run caught exception, sleeping for 200 milli seconds!
java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/elasticsearch/common/settings/ImmutableSettings
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessorFactory.createProcessor(KinesisConnectorRecordProcessorFactory.java:53)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.V1ToV2RecordProcessorFactoryAdapter.createProcessor(V1ToV2RecordProcessorFactoryAdapter.java:36)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.createOrGetShardConsumer(Worker.java:486)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker.run(Worker.java:329)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorExecutorBase.run(KinesisConnectorExecutorBase.java:95)
at com.amazonaws.services.logs.connectors.samples.elasticsearch.ElasticsearchConnector.main(ElasticsearchConnector.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/common/settings/ImmutableSettings
at com.amazonaws.services.kinesis.connectors.elasticsearch.ElasticsearchEmitter.<init>(ElasticsearchEmitter.java:106)
at com.amazonaws.services.logs.connectors.samples.elasticsearch.ElasticsearchPipeline.getEmitter(ElasticsearchPipeline.java:38)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessorFactory.createProcessor(KinesisConnectorRecordProcessorFactory.java:46)
... 11 more
It looks like the ElasticsearchEmitter.java in the amazon-kinesis-connectors library has the import defined and that's what's breaking.
I'm going to add some logic to differentiate between 1.x and 2.x and test locally, if it works, I'll make a PR back to amazon-kinesis-connectors with the fix.
+@dvassallo for visibility, he's been great at helping me through this struggle :)
from cloudwatch-logs-subscription-consumer.
danielcbright
commented on July 30, 2024
OK, I just spent all day getting this to work, it is a two-part problem. The first part is that the amazon-kinesis-connectors library needs to be updated to support elasticsearch 2.0, I've created a PR based on my work today amazon-archives/amazon-kinesis-connectors#63.
The second part is that the current cloudwatch-logs-subscription-consumer uses org.elasticsearch.common.lang which WAS a shade of org.apache.commons.lang in elasticsearch 1.x, now shades are gone and you call the classes directly. A new PR has been made here that has examples on how to make it work for 2.x #11.
from cloudwatch-logs-subscription-consumer.
Related Issues (19)
- Cloudformation template failure HOT 2
- Fails to send JSON with a string prefix and no extracted fields to elasticsearch HOT 1
- not consuming existing flowlogs HOT 10
- keeps getting roll_back HOT 2
- How to map a geo_point in indices that are created automatically by elasticsearch-kopf? HOT 1
- How can I update Kibana version? HOT 2
- Issues with splitting string fields HOT 2
- Kopf-Elasticsearch Version Mismatch HOT 2
- Support for multiple log groups HOT 7
- Record failed with message: UnavailableShardsException
- How do you import these dashboard JSON templates into Amazon Elasticsearch Service? HOT 1
- Setup in an existing VPC HOT 1
- Consumer doesn't restart after instance reboot HOT 9
- Cloudformation doesn't start sample template HOT 2
- Trying to set this up. HOT 8
- SSL options for Elasticsearch consumer needed HOT 2
- CloudTrail to CloudWatch Issues HOT 2
- Can't be used with Amazon Elasticsearch service HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudwatch-logs-subscription-consumer.