exceptions on startup on ec2: exhibitor 1.4.4/1.4.3 and IAM roles about exhibitor HOT 24 CLOSED

Does the new release come up if you specify S3 credentials on the command line or do you get the same error?

Thx,
Jay

from exhibitor.

Same result

$ ~/dev/exhibitor/exhibitor-standalone/build/resources/main/buildscripts/standalone/gradle$ java -jar ./build/libs/gradle-1.4.4.jar -cs3 --s3credentials /home/ubuntu/s3credentials -s3config :conf-1.4.4 --port 8080
v1.4.4
INFO com.netflix.exhibitor.core.activity.ActivityLog Exhibitor started [main]
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:278)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:885)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:775)
at com.netflix.exhibitor.core.s3.S3ClientImpl.getObject(S3ClientImpl.java:102)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.getConfigObject(S3ConfigProvider.java:170)
at com.netflix.exhibitor.core.config.s3.S3ConfigProvider.loadConfig(S3ConfigProvider.java:99)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:83)
at com.netflix.exhibitor.core.config.ConfigManager.(ConfigManager.java:57)
at com.netflix.exhibitor.core.Exhibitor.(Exhibitor.java:128)
at com.netflix.exhibitor.application.ExhibitorMain.(ExhibitorMain.java:95)
at com.netflix.exhibitor.application.ExhibitorMain.main(ExhibitorMain.java:68)
Caused by: org.apache.http.ProtocolException: Received redirect response HTTP/1.1 301 Moved Permanently but no location header
at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:107)
at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1022)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
... 15 more
INFO com.amazonaws.http.AmazonHttpClient Unable to execute HTTP request: null [main]

from exhibitor.

Hmm, then my best guess is that something changed re: how S3 client handles redirects in AWS-SDK 1.3.11 to 1.3.22 which was upgraded in the exhibitor release. Is anything about your config pointing to a bucket in a different region or anything unusual like that?

from exhibitor.

I also notice that you don't have a bucket name. This is a guess - but maybe that has something to do with this?

from exhibitor.

@Randgalt that's just a mistake of me pasting. I fixed it. It's a real bucket but I've obscured the name. The same exact config works fine in 1.4.2

@zarfide nothing unusual like that.

It's a pretty simple set up. I've got an IAM role. I've edited my bucket policy so that that role has read/write perms to the bucket, and then I launched some instances with the role.

from exhibitor.

Hmm, well, looking at the code the ExhibitorCreator doesn't take Roles into consideration and just starts the S3 clients with "null" credentials if you don't specify anything on the command line. I can fix that, but in the mean time it doesn't explain why you can work at all without specifying credentials on the command line in older versions and the redirect error is a bit of a mystery.

from exhibitor.

I thought it might have had something to do with the SDK version, but I don't really see anything in the changelog http://aws.amazon.com/releasenotes/Java?browse=1

sorry, let me clarify, I launched the instances with an IAM profile (ie ec2-run-instance -p). just so we're on the same page, http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/UsingIAM.html#UsingIAMrolesWithAmazonEC2Instances

I believe support was added for this in the 1.3.11 sdk

Release: AWS SDK for Java 1.3.11
This release adds support for AWS IAM Instance Profiles, including new credentials provider implementations in the SDK and updates to Amazon EC2 and Auto Scaling for IAM Instance Profiles, and also adds new features in the Amazon S3, Amazon EC2, Auto Scaling, and Amazon RDS APIs.

So I assume in exhibitor 1.4.2, the sdk transparently looked up the current access keys. Why it's not longer working is a mystery to me as well.

from exhibitor.

Yeah, I didn't see anything in the release notes either, I was just eyeballing a diff of S3Client and noticed a few things (setObjectRedirection on line 835 of 1.3.22).

I understand what you're saying about launching with an IAM profile -- the standalone exhibitor code isn't plumbed up to be smart about using the IAM Role-based on-instance credentials right now (but should be... and I'll get to that).

But if you're getting the error even when manually specifying AWS Credentials, then I'm suspicious that it's something else.

I will try to reproduce your error tomorrow in a clean environment. If there's anything else you can think to mention about your environment don't hesitate to let me know.

Thx,
Jay

from exhibitor.

Scratch that, I think I found the smoking gun.

https://forums.aws.amazon.com/message.jspa?messageID=390486

Looks like we should upgrade to a newer SDK.

from exhibitor.

Can you poke in an upgrade AWS SDK and see if it resolves the problem?

Thx,
Jay

from exhibitor.

Good catch! that looks like it.

I will try to use the 1.3.26 sdk, but I'm a bit of a java newbie so it might take me a bit. I'll respond as soon as I have it working.

from exhibitor.

I updated to 1.3.27 and, while it didn't solve the problem, it did make the error more verbose

$ java -jar exhibitor-standalone.jar -cs3 -s3config :conf --port 8080
dev
INFO com.netflix.exhibitor.core.activity.ActivityLog Exhibitor started [main]
Exception in thread "main" Status Code: 301, AWS Service: Amazon S3, AWS Request ID: 84154EDB07C21790, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., S3 Extended Request ID: 53/9sh/Ib9Q9HIbq+CpakMKhWmI7XJoMulx4EsC2fjtDESbtwuBAZmPiL2ovW9V0

@zarfide as you guessed earlier, I'm not in the US Standard region like I assumed I was. However, I don't think that's the issue.

The code works fine on a non-us-standard bucket that does not have an IAM policy, and I can reproduce the error on a us-standard bucket that does have an IAM policy.

you mentioned

the standalone exhibitor code isn't plumbed up to be smart about using the IAM Role-based on-instance credentials right now (but should be... and I'll get to that).

however I have been able to get IAM roles to work on 1.4.2, so there must be a regression somewhere.

I'll be available tomorrow for more debugging.

from exhibitor.

Any update on this?

from exhibitor.

I haven't been able to identify the regression. I'm attempting to build this branch https://github.com/mwhooker/exhibitor/tree/iam to see if it resolves my issues. If that works it seems like the next step will be to create a new provider chain which checks for a property file first and then proceeds as normal.

Has anyone been able to reproduce?

from exhibitor.

We haven't reproduced this.

from exhibitor.

just tried again with a US-standard region and it works. I am not sure why it didn't work before. If we assume it's an issue with the region, is there a way to fix it?

from exhibitor.

1.4.4 works with a us-standard region, but my build from master with the 1.3.27 sdk does not.

from exhibitor.

My best guess is that the default SDK behavior with accessing cross region S3 buckets has evolved over the past several SDK releases.

If you're up and running with the current exhibitor release, rather than investigating this too much I would rather proceed with our (my?) previously planned Exhibitor updates to get smarter about IAM handling (rather than defaulting to whatever the SDK behavior is) and see if these issues go away.

from exhibitor.

@zarfide I think you're right about cross region s3 bucket behavior changing.

However, if I might hazard a guess, this has less to do with poor IAM handling and more to do with not using the correct endpoint for my region / not following redirects properly.

btw I really appreciate the support and will try whatever I can to help resolve this

from exhibitor.

"this has less to do with poor IAM handling and more to do with not using the correct endpoint for my region / not following redirects properly"

No argument. Not really my area of expertise unfortunately...

from exhibitor.

Not sure if my pr is how you want to solve this (the other option I can see would be to call getBucketLocation), but it does solve my problem.

    String endpoint = "https://s3-" + s3.getBucketLocation(bucketName) + ".amazonaws.com";
    s3.setEndpoint(endpoint);

from exhibitor.

FYI - we finally reproduced this as well. A fix will be published today.

from exhibitor.

Great! thanks again for all your help. sorry it took so long to identify the issue.

from exhibitor.

Thanks for your help on this.

from exhibitor.