elastic / elasticsearch-cloud-aws Goto Github PK

View Code? Open in Web Editor NEW

577.0 375.0 188.0 762 KB

AWS Cloud Plugin for Elasticsearch

Home Page: https://github.com/elastic/elasticsearch/tree/master/plugins/discovery-ec2

elasticsearch-cloud-aws's Introduction

IMPORTANT: Project MOVED!

If you are looking for a hosted solution of Elasticsearch on AWS, please visit http://www.elastic.co/cloud.

From elasticsearch 2.0 you need to look at the following links:

For older versions, look at the following documentation.

In order to install the plugin, run:

bin/plugin install elasticsearch/elasticsearch-cloud-aws/VERSION

Where VERSION is the one you want to install.

Elasticsearch	AWS Cloud Plugin	Docs
es-1.7	2.7.1	2.7.1
es-1.6	2.6.1	2.6.1
es-1.5	2.5.1	2.5.1
es-1.4	2.4.2	2.4.2
es-1.3	2.3.0	2.3.0
es-1.2	2.2.0	2.2.0
es-1.1	2.1.1	2.1.1
es-1.0	2.0.0	2.0.0
es-0.90	1.16.0	1.16.0

License

This software is licensed under the Apache 2 license, quoted below.

Copyright 2009-2014 Elasticsearch <http://www.elasticsearch.org>

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.

elasticsearch-cloud-aws's People

Contributors

Stargazers

Watchers

Forkers

nkvoll holdenk lukas-vlcek hknochi kckit organicveggie cwebberops open-source-gis netconstructor rollick barlem jamesboehmer steven-paster vinodc wikka proximamonkey joeyimbasciano ahmed26 giannioneill web5design imclab swoop-inc adeze coolrepo99 xinv andrewgross imotov suzuken kzwang jhsbeat sagarl maximmironov foundit zircote rjweiss aemruli tsaavik kryptx kurtado dandb dreamingbinary brwe henry-rath philwills ctyeong eredding-rmn bitsofinfo seallison hustcalm ishara agilejon ayumi hvivani mbit-cloud tlrx dldinternet brutasse krave-n ozbillwang festivalbobcats billthebest jomilanez ferhatsb agiza akashaio danisla mbfisher arkcore glenrsmith meltmedia emate malpani paul-e-cooley lmuscariello nagriar ssuprun ksmaheshkumar sunchen009 mkelkarbv rapid7 sidcarter ianblenke rmuir hectorgool gitbbrid jhmartin venkateshdaram434 crate modulexcite jainishshah17 vfrride bizhero poneyo kenden fsero schonfeld twik denzuko-forked mikeaddison93 rtkmhart

elasticsearch-cloud-aws's Issues

S3 blob storage gateway: deleting an index named x destroys data for any index with name beginning with x

S3 storage documentation

Please can you clarify in the documentation whether the S3 storage is deprecated or not. I see that using Remote Gateways is deprecated - but I don't yet understand whether a Local Gateway can work with S3 storage - presumably one file/directory for each node, not one file/directory for each cluster.

Thanks

The dependency on Elasticsearch should be in provided and not compile

Please change the ElasticSearch dependency to provided (and maybe testCompile) instead of compile and runtime.

Why? If you include the plugin in embedded ElasticSearch, and you exclude anything from the ElasticSearch dependency, it is undone by this dependency also including ElasticSearch. Therefore you have to tune both, when really this plugin only needs it as provided (since it runs with ElasticSearch or in ElasticSearch but never on its own).

Update to elasticsearch 1.1.0 / Lucene 4.7.0

Snapshot failures

It seems that if we run our index building process, things go fine. But if we've made changes and need to reindex, we started noticing ES filling the logs with lots of snapshot failures. It turns out that we were deleting and immediately recreating the index "too fast", and it seems that maybe S3 runs slower than ElasticSearch expects. It seems to fail to create certain files for certain shards.

As a workaround, our code now pauses for 10s after deleting the index, and the errors don't seem to occur anymore.

Below are some examples of the errors:

[2012-09-07 03:01:04,295][WARN ][index.gateway ] [Havok] [shanedev][2] failed to snapshot (scheduled)
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException: [shanedev][2] Failed to perform snapshot (index files)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:246)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:160)
at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:271)
at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:265)
at org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:1042)
at org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:528)
at org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:265)
at org.elasticsearch.index.gateway.IndexShardGatewayService$SnapshotRunnable.run(IndexShardGatewayService.java:366)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException: /var/data/elasticsearch/nodes/0/indices/shanedev/2/index/_0.fdt (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at org.elasticsearch.index.store.Store.openInputRaw(Store.java:314)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshotFile(BlobStoreIndexShardGateway.java:755)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:228)
... 10 more

[2012-09-07 03:01:41,407][WARN ][index.gateway ] [Havok] [shanedev][3] failed to snapshot on close
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException: [shanedev][3] Failed to perform snapshot (index files)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:246)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:160)
at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:271)
at org.elasticsearch.index.gateway.IndexShardGatewayService$2.snapshot(IndexShardGatewayService.java:265)
at org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:1042)
at org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:528)
at org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:265)
at org.elasticsearch.index.gateway.IndexShardGatewayService.snapshotOnClose(IndexShardGatewayService.java:310)
at org.elasticsearch.index.service.InternalIndexService.deleteShard(InternalIndexService.java:423)
at org.elasticsearch.index.service.InternalIndexService.access$000(InternalIndexService.java:87)
at org.elasticsearch.index.service.InternalIndexService$1.run(InternalIndexService.java:261)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.FileNotFoundException: /var/data/elasticsearch/nodes/0/indices/shanedev/3/index/_0.fdt (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:345)
at org.elasticsearch.index.store.Store.openInputRaw(Store.java:314)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshotFile(BlobStoreIndexShardGateway.java:755)
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:228)
... 13 more

Update to AWS Java SDK 1.5.3

Update to Elasticsearch 0.90.3

Issue reported in https://groups.google.com/d/msg/elasticsearch/bpdetIvIg5M/cMAUGwhRVf0J

No problem with elasticsearch 0.90.2 and the cloud-aws 1.12.0 plugin (other than the usual node discovery flakiness), but I'm unable to get elasticsearch 0.90.3 to start:

Initialization Failed ...

1) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]2) NoSuchMethodError[org.elasticsearch.discovery.zen.ZenDiscovery.<init>(Lorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/cluster/ClusterName;Lorg/elasticsearch/threadpool/ThreadPool;Lorg/elasticsearch/transport/TransportService;Lorg/elasticsearch/cluster/ClusterService;Lorg/elasticsearch/node/settings/NodeSettingsService;Lorg/elasticsearch/cluster/node/DiscoveryNodeService;Lorg/elasticsearch/discovery/zen/ping/ZenPingService;)V]

Do I need to wait for a new version of cloud-aws, or is there some other problem here?

My /etc/elasticsearch/elasticsearch.yml:

cluster.name: foo

plugin.mandatory: cloud-aws,lang-javascript

cloud:
    aws:
        access_key: ********
        secret_key: ********
        region: us-east-1

discovery:
    type: ec2
    ec2:
        ping_timeout: 15s

gateway:
    type: s3
    s3:
        bucket: bar

Old common-codecs version in use

Hi,

We're using elasticsearch-cloud-aws and it seems to conflict with our own plugin because of commons-codec

com.amazonaws/aws-java-sdk pulls version 1.3 which supports neither org.apache.commons.codec.binary.StringUtils nor org.apache.commons.codec.binary.Base64.encodeBase64URLSafeString(byte[])

Your version of httpclient should pull version 1.4 which is compatible with those two functions. Would it be possible to add the following exclusion to aws-java-sdk ?

<exclusion>
    <groupId>commons-codec</groupId>
    <artifactId>commons-codec</artifactId>
</exclusion>

Do you have an other solution ? We've tried adding common-codecs 1.4 into the lib directory and it works but it's not really satisfying.

Thanks

Startup fails after installing plugin

I added the plugin and can see it being loaded here:

discovery.type: ec2
discovery.ec2.groups: elasticsearch

I didn't add our access_keys and everything b/c the README said we didn't need to with the latest version.

The log has a bunch of lines like this:

[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/fd/ping, handlers: org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@32c5f9fe, org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@36ddc581
[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/publish, handlers: org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@639facbc, org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@19d1b44b
[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/join, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@8059dbd, org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@1693b52b
[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/join/validate, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@28b6e768, org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@3b5b25a1
[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/leave, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@1271ba, org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@5d038b78
[2013-08-09 15:05:27,432][WARN ][transport                ] [Elasticsearch-01] Registered two transport handlers for action discovery/zen/rejoin, handlers: org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@3ae86a5f, org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@13b9a2fd

If I remove:

discovery.type: ec2

The es process starts up just fine. Am I missing something? I can't seem to find an relevant errors for why the process wouldn't be starting.

Update to elasticsearch 1.0.0

Cross-region discovery

In addition to cross-availability-zone discovery within a particular region it would be really useful to implement cross-region discovery so that developers/ops can build truly distributed search index.

For example, discovery.ec2 can be prepended to region name so setting will be discovery.ec2.us-east. And all available options will go further as it's now.

Initialization fails with ClassNotFoundException[com.amazonaws.services.s3.AmazonS3] with local gateway

I'm using elasticsearch 0.90.10, elasticsearch-cloud-aws plugin 1.16.0

Upon starting elasticsearch, initialization fails with the following log. I am not using the S3 gateway.

2014-02-06 19:28:54,320[INFO ][node                     ] [ip-10-232-25-62-1] version[0.90.10], pid[11562], build[0a5781f/2014-01-10T10:18:37Z]
2014-02-06 19:28:54,333[INFO ][node                     ] [ip-10-232-25-62-1] initializing ...
2014-02-06 19:28:54,380[INFO ][plugins                  ] [ip-10-232-25-62-1] loaded [<<...>> cloud-aws], sites [<<...>>>]
2014-02-06 19:28:55,088[ERROR][bootstrap                ] {0.90.10}: Initialization Failed ...
- ExecutionError[java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3]
    NoClassDefFoundError[com/amazonaws/services/s3/AmazonS3]
        ClassNotFoundException[com.amazonaws.services.s3.AmazonS3]

Relevant settings in the elasticsearch.yml include:

discovery.type: ec2
cloud.aws.access_key: <<key>>
cloud.aws.secret_key: <<key>>
gateway.recover_after_nodes: 3

Adding the following line does not help.

gateway.type: local

Can anyone suggest any solution or workaround? Thank you!

Old Releases

Can a release for 1.10.0 be created? I believe that is the latest version of the plugin that will work with elasticsearch 0.20.6, which is the version tied to logstash 1.1.13.

Old httpclient version in use

Hi,

We're using elasticsearch-cloud-aws and it seems to conflict with our own plugin because of org.apache.httpcomponents/httpclient

You're forcing version 4.1.1 and we're using 4.3.2. As a result, some classes that we use are missing.

We've managed to make this work by removing the jars of httpclient that your plugin loads so that it uses ours and it seems to work but it's not the kind of hack we like to do in production.

Would it be possible on your side to update to the latest version of httpclient ? This new version marks alots of the 4.1 methods as deprecated but keeps them around.

Thanks for your help.
Regards

Multiple instances on the same server in race-condition

Hi, as asked here https://groups.google.com/forum/#!topic/elasticsearch/ZZsBNHOm8og I'm opening an issue here.

I have ES 0.90.10, I couldn't test with the latest release, I honestly don't know if maybe it was fixed since.

On the same server, I have a master/data instance of ES with the EC2 plugin and a second non-master/no-data one.

When the master instance is started first, and the non-master afterwards, both are in the same configured cluster.

If I start the non-master first, it stops working. I tried the following but without luck :

Hardcode the 9200/9300 for the master and 9201/9301 for the non-master
Enable master/data on the second instance (I end up with 2 separate cluster with the same name)
Remove the EC2 plugin configuration from the non-master and only enable unicast to localhost (it only ping localhost:9300 and not 9301)

Edit: Also tried to remove the EC2 plugin config on the master, I still have the issue, so maybe it's not related.

Closed: I will move this issue to elasticsearch

Cannot initialize on ES 1.1.0

I am getting the following error initializing the latest RC1 plugin on ES 1.1.0. This works in ES 1.0.1 and also when no cloud-aws config is set.

[2014-03-26 14:46:07,372][ERROR][bootstrap ] {1.1.0}: Initialization Failed ...

NoSuchMethodError[org.elasticsearch.discovery.zen.ZenDiscovery.(Lorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/cluster/ClusterName;Lorg/elasticsearch/threadpool/ThreadPool;Lorg/elasticsearch/transport/TransportService;Lorg/elasticsearch/cluster/ClusterService;Lorg/elasticsearch/node/settings/NodeSettingsService;Lorg/elasticsearch/cluster/node/DiscoveryNodeService;Lorg/elasticsearch/discovery/zen/ping/ZenPingService;Lorg/elasticsearch/Version;)V]2) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]

java.lang.NoSuchMethodError with 0.19.0

this is showing up in the logs, may just need a recompile and push to mvn.

Error injecting constructor, java.lang.NoSuchMethodError: org.elasticsearch.common.util.concurrent.EsExecutors.newScalingExecutorService(IIJLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/ThreadFactory;)Ljava/util/concurrent/ThreadPoolExecutor;
at org.elasticsearch.gateway.s3.S3Gateway.(Unknown Source)
while locating org.elasticsearch.gateway.s3.S3Gateway
while locating org.elasticsearch.gateway.Gateway
Caused by: java.lang.NoSuchMethodError: org.elasticsearch.common.util.concurrent.EsExecutors.newScalingExecutorService(IIJLjava/util/concurrent/TimeUnit;Ljava/util/concurrent/ThreadFactory;)Ljava/util/concurrent/ThreadPoolExecutor;
at org.elasticsearch.gateway.s3.S3Gateway.(S3Gateway.java:85)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)

Support group id in addition to group names

Based on issue #43, we want to support both notations:

group name first
group id if it does not work with group name

Anyone tried with 0.90.0.Beta1

I have a config in elasticsearch.yml and it is going to my nodes in my account turned trace on but I am not getting any discovery.ec2 indications in the log

I tried to set one as master and the other as false.

Classpath problem and NoSuchMethodError

With ES 1.0.1 (my own compile which works fine without the cloud-aws plugin) and cloud-aws 2.0.0.RC1, when I installed with bin/plugin and run ES I get:

[2014-03-14 17:19:08,792][INFO ][node                     ] [Mammomax] version[2.0.0-SNAPSHOT], pid[49038], build[b7a95d1/2014-03-13T19:51:39Z]
[2014-03-14 17:19:08,792][INFO ][node                     ] [Mammomax] initializing ...
[2014-03-14 17:19:08,869][INFO ][plugins                  ] [Mammomax] loaded [cloud-aws], sites [head, inquisitor, kibana]
{2.0.0-SNAPSHOT}: Initialization Failed ...
- NoClassSettingsException[Failed to load class setting [discovery.type] with value [ec2]]
    ClassNotFoundException[org.elasticsearch.discovery.ec2.Ec2DiscoveryModule]

I tried moving the cloud-aws jars to lib/cloud-aws, adding that to my classpath, and just having a es-plugin.properties in plugins/cloud-aws. That starts up but then I get:

{2.0.0-SNAPSHOT}: Initialization Failed ...
1) NoSuchMethodError[org.elasticsearch.discovery.zen.ZenDiscovery.<init>(Lorg/elasticsearch/common/settings/Settings;Lorg/elasticsearch/cluster/ClusterName;Lorg/elasticsearch/threadpool/ThreadPool;Lorg/elasticsearch/transport/TransportService;Lorg/elasticsearch/cluster/ClusterService;Lorg/elasticsearch/node/settings/NodeSettingsService;Lorg/elasticsearch/cluster/node/DiscoveryNodeService;Lorg/elasticsearch/discovery/zen/ping/ZenPingService;Lorg/elasticsearch/Version;)V]2) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]

I have the configuration set as in the example but I only have one node currently running.

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

Update to elasticsearch 1.0.0.RC1

Unable to read metadata from gateway if snapshot is created

See https://gist.github.com/nkvoll/219b706990e7287bf8d0 for full stack traces.

The issue seem to be that during initialization of the S3Gateway, the readMetaData of MetaData uses SnapshotMetaData to parse a field called snapshots. Its fromXContent method simply raises an UnsupportedOperationException, which causes Elasticsearch to exit (at https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/metadata/SnapshotMetaData.java#L358).

region setting from snapshot ignored

Hi,

when I set my region to eu-west-1 in elasticsearch.yml and then create repository, e.g.

curl -s -XPUT http://localhost:8200/_snapshot/s3_backups -d '{
        "type": "s3",         
        "settings": {
                "bucket": "mybucket",
                "region": "us-east-1",
                "compress" : true,
                "base_path" : "some/path"
       }
}'

Then when I try to do backup/restore of a snapshot, plugin still goes to bucket.s3-eu-west1.amazonaws.com - it then gets 301 redirect and it returns error to user. So two problems/bugs here:

If you get 301, just follow the redirect, don't fail
Do honor the region setting from the snapshot repository definition

Quick fix to this is to simply set your region to us-east-1 in the elasticsearch.yml - but this breaks other functionality (e.g. discovery).

java.lang.IllegalStateException when using 0.90.10 with 1.16.0

I've tried various version of elasticsearch-cloud-aws but all end up with the same exception.

I'm using openjdk-6-jre version 6b27-1.12.6-1~deb6u1 on Debian 6.0.8. Removing discovery.type: ec2 will let me boot Elasticsearch.

Any clue what's causing this? I'll dig some more on my end.

/etc/elasticsearch/elasticsearch.yml:

...
cloud.aws.region: eu-west–1
cloud.aws.access_key: XXXXXXXX
cloud.aws.secret_key: XXXXXXXXXXXXXXXXXX

discovery.type: ec2
discovery.ec2.groups: com.production.search

/var/log/elasticsearch/com.log:

...
[2014-01-30 11:28:11,184][DEBUG][indices.memory           ] [vespucci] using index_buffer_size [409.1mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
[2014-01-30 11:28:11,185][DEBUG][indices.cache.filter     ] [vespucci] using [node] weighted filter cache with size [20%], actual_size [818.3mb], expire [null], clean_interval [1m]
[2014-01-30 11:28:11,187][DEBUG][indices.fielddata.cache  ] [vespucci] using size [-1] [-1b], expire [null]
[2014-01-30 11:28:11,349][DEBUG][bootstrap                ] Exception
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.
  at org.elasticsearch.cache.NodeCache.<init>(Unknown Source)
  while locating org.elasticsearch.cache.NodeCache
Caused by: java.lang.IllegalStateException: This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.
        at org.elasticsearch.common.inject.internal.ConstructionContext$DelegatingInvocationHandler.invoke(ConstructionContext.java:102)
        at sun.proxy.$Proxy14.add(Unknown Source)
        at org.elasticsearch.cache.NodeCache.<init>(NodeCache.java:44)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
        at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
        at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
        at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:837)
        at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
        at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:57)
        at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
        at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:830)
        at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
        at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:174)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:68)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:201)
        at org.elasticsearch.bootstrap.ElasticSearch.main(ElasticSearch.java:32)
...

Repository specific credentials

In order to allow our customers to snapshot to and from private buckets it would be preferable if one could specify the credentials for the bucket when defining a repository. After a short glance at the code it looks doable by extending AwsS3Service to support multiple clients, one for each combination of region and credentials needed. This should make it possible to solve issue #54 as well.

Unless there are some major culprits you can think of or you think this feature is a step in the wrong direction, we would be happy to provide the initial implementation.

Upgrade to AWS SDK 1.3.3

Move tests to elasticsearch test framework

Migrated to 0.19, installed cloud-aws but EC2 discovery not found.

The cloud-aws plugin is installed:
[2012-03-10 01:44:23,709][INFO ][plugins ] [Tundra] loaded [], sites [bigdesk, river-twitter, mapper-attachments, cloud-aws, head]

Startup output:
INFO | jvm 1 | 2012/03/10 01:44:23 | {0.19.0}: Initialization Failed ...
INFO | jvm 1 | 2012/03/10 01:44:23 | - NoClassSettingsException[Failed to load class setting [discovery.type] with value [ec2]]
INFO | jvm 1 | 2012/03/10 01:44:23 | ClassNotFoundException[ec2]
STATUS | wrapper | 2012/03/10 01:44:25 | <-- Wrapper Stopped

Config:
bootstrap.mlockall: true
cloud.aws.access_key: ******************
cloud.aws.region: us-east-1
cloud.aws.secret_key: *****************
cluster.name: atreus
discovery.ec2.tag.Name: elasticsearch
discovery.type: ec2
gateway.s3.bucket: maamp-gateway
gateway.type: s3
path.data: /var/data/elasticsearch

Update to Elasticsearch 0.90.1

Works also with Elasticsearch 0.90.0

Startup failing with ec2 discovery for ES 90.4 / 90.5

very similar issue to #33

elasticsearch.yml

cloud:
  aws:
    access_key: XXXXX
    access_secret: XXXX
    region: ap-southeast-2
cluster:
  name: XXX
discovery:
  ec2:
    tag:
      project: XXX
      subtype: XXX
      type: XXX
  type: ec2

If I remove the ec2 discovery section, elasticsearch starts fine.

Otherwise, the log inits with

[2013-10-08 00:44:38,937][INFO ][node                     ] [Astron] version[0.90.4], pid[1787], build[8cb1a9b/2013-09-16T14:11:00Z]
[2013-10-08 00:44:38,938][INFO ][node                     ] [Astron] initializing ...
[2013-10-08 00:44:38,954][INFO ][plugins                  ] [Astron] loaded [cloud-aws], sites []
[2013-10-08 00:44:40,610][DEBUG][discovery.zen.ping.multicast] [Astron] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2013-10-08 00:44:40,615][DEBUG][discovery.zen.ping.unicast] [Astron] using initial hosts [], with concurrent_connects [10]
[2013-10-08 00:44:40,616][DEBUG][discovery.ec2            ] [Astron] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-10-08 00:44:40,617][DEBUG][discovery.zen.elect      ] [Astron] using minimum_master_nodes [-1]
[2013-10-08 00:44:40,618][DEBUG][discovery.zen.fd         ] [Astron] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-08 00:44:40,623][DEBUG][discovery.zen.fd         ] [Astron] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-08 00:44:41,208][DEBUG][discovery.ec2            ] [Astron] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-10-08 00:44:41,208][DEBUG][discovery.zen.elect      ] [Astron] using minimum_master_nodes [-1]
[2013-10-08 00:44:41,209][DEBUG][discovery.zen.fd         ] [Astron] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

Then will repetitively spit out the following

[2013-10-08 00:44:41,209][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/fd/masterPing, handlers: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@3e40f7a8, org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@1b435686
[2013-10-08 00:44:41,209][DEBUG][discovery.zen.fd         ] [Astron] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-08 00:44:41,209][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/fd/ping, handlers: org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@61e58565, org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@2398dc19
[2013-10-08 00:44:41,209][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/publish, handlers: org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@899ff91, org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@36598d00
[2013-10-08 00:44:41,210][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/join, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@313816e0, org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@5965e55e
[2013-10-08 00:44:41,210][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/join/validate, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@30b5ac2c, org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@673a95af
[2013-10-08 00:44:41,210][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/leave, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@3739491b, org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@11eea7f0
[2013-10-08 00:44:41,210][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/rejoin, handlers: org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@4b5386c2, org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@58b45130
[2013-10-08 00:44:41,681][DEBUG][discovery.ec2            ] [Astron] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-10-08 00:44:41,681][DEBUG][discovery.zen.elect      ] [Astron] using minimum_master_nodes [-1]
[2013-10-08 00:44:41,681][DEBUG][discovery.zen.fd         ] [Astron] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-08 00:44:41,682][WARN ][transport                ] [Astron] Registered two transport handlers for action discovery/zen/fd/masterPing, handlers: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@5efe087b, org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@3e40f7a8
[2013-10-08 00:44:41,682][DEBUG][discovery.zen.fd         ] [Astron] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]

Affects the following ES version and plugin combination

ES 90.5
Plugin 1.15.0

ES 90.4
Plugin 1.15.0

ES 90.3
Plugin 1.14.0

Similar to #33 if I remove ec2 discovery section, it loads fine

What am I missing?

Update to AWS SDK 1.7.3

Add plugin release semi-automatic script

Attempting to get AWS discovery working with logstash ES output

I have an existing 0.90.2 cluster with logstash elasticsearch outputs connecting to it using cloud-aws and it works fine. Now I am trying to build a new environment with ES 1.1.0 and cloud-aws 2.10. I have had to add a lot of jars to the logstash class path (all ES lib jars, and all the cloud-aws jars). I am seeing the following exception:

Exception in thread ">output" org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) A binding to org.elasticsearch.cloud.aws.AwsS3Service was already configured at org.elasticsearch.cloud.aws.AwsModule.configure(    AwsModule.java:31).
  at org.elasticsearch.cloud.aws.AwsModule.configure(AwsModule.java:31)

2) A binding to org.elasticsearch.cloud.aws.AwsEc2Service was already configured at org.elasticsearch.cloud.aws.AwsModule.configure(    AwsModule.java:32).
  at org.elasticsearch.cloud.aws.AwsModule.configure(AwsModule.java:32)

2 errors
    at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(org/elasticsearch/common/inject/internal/Errors    .java:344)
    at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(org/elasticsearch/common/inject/InjectorBuilder.java:151)
    at org.elasticsearch.common.inject.InjectorBuilder.build(org/elasticsearch/common/inject/InjectorBuilder.java:102)
    at org.elasticsearch.common.inject.Guice.createInjector(org/elasticsearch/common/inject/Guice.java:93)

Here is my class path:

/usr/local/elasticsearch/lib/jts-1.13.jar:/usr/local/elasticsearch/lib/lucene-core-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-memory-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-sandbox-4.7.0.jar:/usr/local/elasticsearch/lib/spatial4j-0.4.1.jar:/usr/local/elasticsearch/lib/log4j-1.2.17.jar:/usr/local/elasticsearch/lib/lucene-grouping-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-misc-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-spatial-4.7.0.jar:/usr/local/elasticsearch/lib/elasticsearch-1.1.0.jar:/usr/local/elasticsearch/lib/lucene-analyzers-common-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-highlighter-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-queries-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-suggest-4.7.0.jar:/usr/local/elasticsearch/lib/jna-3.3.0.jar:/usr/local/elasticsearch/lib/lucene-codecs-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-join-4.7.0.jar:/usr/local/elasticsearch/lib/lucene-queryparser-4.7.0.jar:$basedir/vendor/jar/cloud-aws/aws-java-sdk-1.7.3.jar:$basedir/vendor/jar/cloud-aws/commons-codec-1.4.jar:$basedir/vendor/jar/cloud-aws/commons-logging-1.1.1.jar:$basedir/vendor/jar/cloud-aws/elasticsearch-cloud-aws-2.1.0.jar:$basedir/vendor/jar/cloud-aws/httpclient-4.2.jar:$basedir/vendor/jar/cloud-aws/httpcore-4.2.jar:$basedir/vendor/jar/cloud-aws/jackson-annotations-2.1.1.jar:$basedir/vendor/jar/cloud-aws/jackson-core-2.1.1.jar:$basedir/vendor/jar/cloud-aws/jackson-databind-2.1.1.jar:$basedir/vendor/jar/cloud-aws/joda-time-2.3.jar:$basedir/vendor/jar/jruby-complete-1.7.11.jar org.jruby.Main

Big warning?

I think there is supposed to be a big warning that using this plugin on an already running elasticsearch install will wipe out all data. Not to mention my wildcard queries do not work since i installed this plugin. using ES 1.9

also everytime i restart ES now i loose all data again.

Logstash 1.4 + cloud-aws 2.1.0 + elasticsearch 1.1

I used to copy elasticsearch.yml in /usr/local/bin/logstash along with cloud-aws directory and it was working fine with logstash 1.3

Now with 1.4, I copied cloud-aws in /opt/logstash/vendor/ and I checked in the logs that it's properly loaded, but the yml file is not. I tried to put it a bit everywhere but no luck.

Is this plugin supported with the last version of logstash ?

Cluster in us-west-2a

I apoligize if this is not the place to post this. I have no other idea how to debug. I am trying to deploy two nodes in us-west-2a. I have followed the guidlines and have the following config.

cluster.name: prod-es

cloud:
    aws:
        access_key: HIDDEN
        secret_key: HIDDEN
        region: us-west-2a

discovery:
    type: ec2

I obtained access_key and secret_key by going to IAM section of AWS and clicking grant all accesses. See below for screenshot.

I then changed logging to trace as suggested and restarted both nodes. However, nothing happens and the http port 9200 is never actually functional. Below is the logging when I start one of the nodes. I can't make sense. If I remove region then everything is up but the cluster only has one node. When trying unicast, everything work. Are there any other debugging tools that I should use to see what is going on?

I can telnet to each box from the other. So I know firewall is not the issue.

[2013-06-06 05:40:25,788][INFO ][plugins                  ] [Elixir] loaded [cloud-aws], sites [head]
[2013-06-06 05:40:30,422][DEBUG][discovery.zen.ping.multicast] [Elixir] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2013-06-06 05:40:30,427][DEBUG][discovery.zen.ping.unicast] [Elixir] using initial hosts [], with concurrent_connects [10]
[2013-06-06 05:40:30,429][DEBUG][discovery.ec2            ] [Elixir] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-06-06 05:40:30,430][DEBUG][discovery.zen.elect      ] [Elixir] using minimum_master_nodes [-1]
[2013-06-06 05:40:30,432][DEBUG][discovery.zen.fd         ] [Elixir] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 05:40:30,462][DEBUG][discovery.zen.fd         ] [Elixir] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 05:40:32,325][DEBUG][discovery.ec2            ] [Elixir] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-06-06 05:40:32,328][DEBUG][discovery.zen.elect      ] [Elixir] using minimum_master_nodes [-1]
[2013-06-06 05:40:32,328][DEBUG][discovery.zen.fd         ] [Elixir] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 05:40:32,329][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/fd/masterPing, handlers: org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@566622c9, org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler@4b75e66f
[2013-06-06 05:40:32,329][DEBUG][discovery.zen.fd         ] [Elixir] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-06-06 05:40:32,329][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/fd/ping, handlers: org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@296ba81d, org.elasticsearch.discovery.zen.fd.NodesFaultDetection$PingRequestHandler@5aea1d29
[2013-06-06 05:40:32,330][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/publish, handlers: org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@40209d60, org.elasticsearch.discovery.zen.publish.PublishClusterStateAction$PublishClusterStateRequestHandler@1d3c2a84
[2013-06-06 05:40:32,332][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/join, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@590b4f1a, org.elasticsearch.discovery.zen.membership.MembershipAction$JoinRequestRequestHandler@757661a1
[2013-06-06 05:40:32,332][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/join/validate, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@2f03cd46, org.elasticsearch.discovery.zen.membership.MembershipAction$ValidateJoinRequestRequestHandler@2299c6c7
[2013-06-06 05:40:32,333][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/leave, handlers: org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@1dc94490, org.elasticsearch.discovery.zen.membership.MembershipAction$LeaveRequestRequestHandler@8b45937
[2013-06-06 05:40:32,333][WARN ][transport                ] [Elixir] Registered two transport handlers for action discovery/zen/rejoin, handlers: org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@784d2e57, org.elasticsearch.discovery.zen.ZenDiscovery$RejoinClusterRequestHandler@131d8f75
[2013-06-06 05:40:32,334][DEBUG][discovery.ec2            ] [Elixir] using host_type [PRIVATE_IP], tags [{}], groups [[]] with any_group [true], availability_zones [[]]
[2013-06-06 05:40:33,776][DEBUG][gateway.local            ] [Elixir] using initial_shards [quorum], list_timeout [30s]
[2013-06-06 05:40:35,117][DEBUG][gateway.local.state.meta ] [Elixir] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2013-06-06 05:40:35,312][DEBUG][gateway.local.state.meta ] [Elixir] took 192ms to load state
[2013-06-06 05:40:35,312][DEBUG][gateway.local.state.shards] [Elixir] took 0s to load started shards state
``

Optional credentials with latest AWS SDK

With the latest AWS SDK, credentials (access_key, secret_key) no longer need to be passed to the API but can be associated with EC2 instances [1].

[1] http://aws.typepad.com/aws/2012/06/iam-roles-for-ec2-instances-simplified-secure-access-to-aws-service-apis-from-ec2.html

Use instance metadata instead of external AWS API endpoint

The discovery mechanism requires an external API endpoint (ec2.us-east-1.amazonaws.com) call to gather information. This requires the security group to allow outbound access TCP/80 access to AWS's external resources.

It would be great if the discovery could use the instance metadata to gather information instead which doesn't require any outbound access. (http://169.254.169.254/latest/meta-data/security-groups/)

Plugin installation failed for 2.0.0-SNAPSHOT

Hi,

I'm trying to install the plugin for an ES 1.0 install using the 2.0.0-SNAPSHOT as directed in the readme, but it's failing to install. Here's the entirety of the output I received:

-> Installing elasticsearch/elasticsearch-cloud-aws/2.0.0-SNAPSHOT...
Trying http://download.elasticsearch.org/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip...
Trying http://search.maven.org/remotecontent?filepath=elasticsearch/elasticsearch-cloud-aws/2.0.0-SNAPSHOT/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/elasticsearch/elasticsearch-cloud-aws/2.0.0-SNAPSHOT/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/archive/v2.0.0-SNAPSHOT.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/archive/master.zip...
Downloading ...................................................DONE
Installed elasticsearch/elasticsearch-cloud-aws/2.0.0-SNAPSHOT into /usr/share/elasticsearch/plugins/cloud-aws
Usage:
    -u, --url     [plugin location]   : Set exact URL to download the plugin from
    -i, --install [plugin name]       : Downloads and installs listed plugins [*]
    -t, --timeout [duration]          : Timeout setting: 30s, 1m, 1h... (infinite by default)
    -r, --remove  [plugin name]       : Removes listed plugins
    -l, --list                        : List installed plugins
    -v, --verbose                     : Prints verbose messages
    -s, --silent                      : Run in silent mode
    -h, --help                        : Prints this help message

 [*] Plugin name could be:
     elasticsearch/plugin/version for official elasticsearch plugins (download from download.elasticsearch.org)
     groupId/artifactId/version   for community plugins (download from maven central or oss sonatype)
     username/repository          for site plugins (download from github master)

Message:
   Error while installing plugin, reason: IllegalArgumentException: Plugin installation assumed to be site plugin, but contains source code, aborting installation.

Thanks in advance for your thoughts on the matter.

Did this plugin move?

Running bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.11.0 results in:

-> Installing elasticsearch/elasticsearch-cloud-aws/1.11.0...
Trying https://github.com/downloads/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-1.11.0.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/zipball/v1.11.0...
Failed to install elasticsearch/elasticsearch-cloud-aws/1.11.0, reason: failed to download

If I hit those urls in a browser, one is a 'Not found', and one is a github 404 page.

Will the latest v1.16.0 work with Elasticsearch 0.90.7?

Hi there,

I am wondering if the current stable version of elasticsearch-cloud-aws plugin can work with Elasticsearch v0.90.7?

Because I found on the front page there is only ES v0.90.4 listed there.

Does anyone know about this ?

Thanks

Tags for versions after 1.4.0

[ec2-user@ip-10-64-2-113 elasticsearch-0.19.12]$ sudo bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.11.0
-> Installing elasticsearch/elasticsearch-cloud-aws/1.11.0...
Trying https://github.com/downloads/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-1.11.0.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/zipball/v1.11.0...
Failed to install elasticsearch/elasticsearch-cloud-aws/1.11.0, reason: failed to download

Looks like the tags are missing from the repo. Either update instructions or add the relevant tags to the repo.

Bad Snapshot Repo Example

When I use the example to create a snapshot repo I get an AWS exception that the name in the example is bad:

{"error":"RepositoryException[[my_s3_repository] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, java.lang.IllegalArgumentException: Bucket name should not contain '_'\n  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)\n  while locating org.elasticsearch.repositories.s3.S3Repository\n  while locating org.elasticsearch.repositories.Repository\n\n1 error]; nested: IllegalArgumentException[Bucket name should not contain '_']; ","status":500}

Obviously it's an example but making it have a valid name would be a +.

I would recommend 'my_backet_name' -> 'my-bucket-name'

Add plugin version in es-plugin.properties

With elastic/elasticsearch#2784, we can now add plugin version in es-plugin.properties file.

It will only be used with elasticsearch 1.0.0 and upper. No need to push it in 1.x branch.

minimum_master_nodes parameter

Hello,

Is it possible to define the minimum_master_nodes parameter (discovery.zen.minimum_master_nodes) while using the cloud plugin for discovery or this parameter is available only when zen is used for discovery?

Regards,
Nick

InvalidLocationConstraint when registering snapshot repository in us-east-1

This appears to work with any of the other regions, so I'm not sure what's up. I've seen in the S3 docs that us-east-1 doesn't require a location constraint as it's the default, but it looks like that's taken care of here: https://github.com/elasticsearch/elasticsearch-cloud-aws/blob/master/src/main/java/org/elasticsearch/repositories/s3/S3Repository.java#L91-L95

curl -XPUT 'http://.../_snapshot/repository' -d '{
  "type" : "s3",
  "settings" : {
    "bucket" : "bucket_name",
    "region" : "us-east-1"
  }
}'

{"error":"RepositoryException[[repository] failed to create repository]; nested: CreationException[Guice creation errors:

1) Error injecting constructor, com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: ..., AWS Error Code: InvalidLocationConstraint, AWS Error Message: The specified location-constraint is not valid, S3 Extended Request ID: ...
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error]; nested: AmazonS3Exception[Status Code: 400, AWS Service: Amazon S3, AWS Request ID: ..., AWS Error Code: InvalidLocationConstraint, AWS Error Message: The specified location-constraint is not valid]; ","status":500}

Deletes may not be honored by the S3 cloud backend

When deleting an index from ES using the HEAD plugin, I noticed the following log statement:

[com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser] Unexpected tag: DeleteResult

Upon some research I found this link:

https://forums.aws.amazon.com/thread.jspa?threadID=82588

Based on that thread, the problem seems to have been resolved from AWS's end so I was wondering if fixing this for the cloud-aws plugin is as simple as working the aws's SDK update into the latest build?

Starting a non-master, non-data instance first breaks cluster discovery

Hello,

I have ES 0.90.10, I couldn't test with the latest release, I honestly don't know if maybe it was fixed since.

On the same server, I have a master/data instance of ES and a second non-master. ( node.master: false )

When the master instance is started first, and the non-master afterwards, both are in the same configured cluster.

If I start the non-master first, it stops working.

I also tested with both the AWS EC2 plugin discovery and the unicast discovery.

That tells me, if the first node to start can't be master, even by starting a second instance that can, with the same cluster.name, the cluster can't be created.

Closed: Sorry, opened in the wrong repo.

Install not working. Failed to install elasticsearch/elasticsearch-cloud-aws/1.9.0, reason: failed to download

/usr/share/elasticsearch/bin/plugin -install elasticsearch/elasticsearch-cloud-aws
-> Installing elasticsearch/elasticsearch-cloud-aws...
Trying https://github.com/downloads/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-0.19.11.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/zipball/v0.19.11...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/zipball/master...
Failed to install elasticsearch/elasticsearch-cloud-aws, reason: failed to download

/usr/share/elasticsearch/bin/plugin -install elasticsearch/elasticsearch-cloud-aws/1.9.0
-> Installing elasticsearch/elasticsearch-cloud-aws/1.9.0...
Trying https://github.com/downloads/elasticsearch/elasticsearch-cloud-aws/elasticsearch-cloud-aws-1.9.0.zip...
Trying https://github.com/elasticsearch/elasticsearch-cloud-aws/zipball/v1.9.0...
Failed to install elasticsearch/elasticsearch-cloud-aws/1.9.0, reason: failed to download

Tempory S3 failure can block launching

I had a host fail to come up after the S3 pull-down failed. I could add retry logic inside of readBlob in AbstractS3BlobContainer or I could add retry logic inside of ES its self. Any preference?

elastic / elasticsearch-cloud-aws Goto Github PK

elasticsearch-cloud-aws's Introduction

License

elasticsearch-cloud-aws's People

Contributors

Stargazers

Watchers

Forkers

elasticsearch-cloud-aws's Issues

Recommend Projects

Recommend Topics

Recommend Org