Git Product home page Git Product logo

aws-signing-request-interceptor's Introduction

aws-signing-request-interceptor

Deprecated

This has been deprecated as there are a lot more standard ways of signing requests to ElasticSearch / OpenSearch hosted in AWS. If you're looking here you probably need to find a newer client.

aws-signing-request-interceptor

Request Interceptor for Apache Client that signs the request for AWS.

Originally created to support AWS' Elasticsearch Service using the Jest client.

Usage

You have to add the AWSSigningRequestInterceptor to the end of the Apache client request chain. Otherwise it won't have visibility of all of the headers being added to the request.

This depends on the AWS core SDK as it relies on an AWSCredentialsProvider to get the key, secret and optional session token. It's advised that you use dependencyManagement to lock in the version of aws-java-sdk-core that works for your project.

private static final String SERVICE = "es";
private static final String REGION = "eu-west-1";
...
final Supplier<LocalDateTime> clock = () -> LocalDateTime.now(ZoneOffset.UTC);
final AWSSigner awsSigner = new AWSSigner(awsCredentialsProvider, REGION, SERVICE, clock);
builder.addInterceptorLast(new AWSSigningRequestInterceptor(awsSigner));

To be able to add the AWSSigningRequestInterceptor to Jest, and thus be able to sign requests to the Elasticsearch Service, you need to override the configureHttpClient method in the JestClientFactory.

final AWSSigningRequestInterceptor requestInterceptor = new AWSSigningRequestInterceptor(awsSigner);
final JestClientFactory factory = new JestClientFactory() {
    @Override
    protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
        builder.addInterceptorLast(requestInterceptor);
        return builder;
    }
    @Override
    protected HttpAsyncClientBuilder configureHttpClient(HttpAsyncClientBuilder builder) {
        builder.addInterceptorLast(requestInterceptor);
        return builder;
    }
};

The project can be found in maven central:

<dependency>
    <groupId>vc.inreach.aws</groupId>
    <artifactId>aws-signing-request-interceptor</artifactId>
    <version>0.0.22</version>
</dependency>

Other Languages

If you're looking for a native Scala version of the AWSSigner then take a look at @ticofab's port: https://github.com/ticofab/aws-request-signer

aws-signing-request-interceptor's People

Contributors

camalot9 avatar censoreduser avatar dmnm avatar ericcitaire avatar homes2001 avatar louiszuckerman avatar ltagliamonte avatar marnix avatar quintonm avatar restfulhead avatar thesmith avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-signing-request-interceptor's Issues

Refresh index failed

There is not error when query and insert data to AWS Elasticsearch service. But When I refresh an index like:
Refresh refresh = new Refresh.Builder().addIndex(index).build(); JestResult jestResult = _jestClient.execute(refresh);
And get an error return.
CONTEXT=index=newlmes_idx_28_1456704000000, jsonResult={"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/newlmes_idx_28_1456704000000/_refresh\na=b\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:0\ncontent-type:application/json; charset=UTF-8\nhost:search-xxx.ap-northeast-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.4.1 (Java/1.7.0_71)\nx-amz-date:20160405T091625Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160405T091625Z\n20160405/ap-northeast-1/es/aws4_request\n87dc94bf640ae0c2e61071265eda687ee8c6dae9968b460687f836df717e798d'\n"}

Issue with "Connection"

Sometimes (don't ask me about it, sorry), the Signage might break due to the "Connection" header:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'PUT\n/user/user/aldrinleal2\n\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:107\ncontent-type:application/json; charset=UTF-8\nhost:search-wechase-dev-65phvknk3pxuhxrqzpr4hggmvq.us-east-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_101)\nx-amz-date:20160823T135045Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\n9192d051542ec4df30c816b8b59d9134d21f6d90878c4f45777b825c9bb9389f'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160823T135045Z\n20160823/us-east-1/es/aws4_request\n77575171872045c07f741778e0b18f20121695938077b02519201f81b8187064'\n"}

The shortest change to allow this would be to add this line:

    final LocalDateTime now = clock.get();
    final AWSCredentials credentials = credentialsProvider.getCredentials();
    final Map<String, Object> result = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
    result.putAll(headers);

    result.remove("Connection"); // THIS ONE

Hope it helps.

request signed incorrectly when * is included in index name - URI is double encoded

aws-signing-request-interceptor version 0.0.15
jest version 2.0.4

Search via JestClient with index name of log-message-*

15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "POST /log-message-*/log-message/_search HTTP/1.1[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJFBO2C537SOTMKQQ/20170126/us-west-2/es/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;user-agent;x-amz-date, Signature=fab5bd06f69d72bec5aa66d2e2da06123e8950d5a20f27ba0e7c52d0a45d0e86[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Length: 201[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Host: search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "x-amz-date: 20170126T212513Z[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "{[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "  "query": {[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "    "match_phrase": {[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "      "logFile": "test:test:test-hostname:rest-test-hostname.api.2017-01-26.21-25-02.bbc65439-2e21-4ee6-a4fd-90c508e85086.txt:2017-01-26T21:25:04.636Z"[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "    }[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "  }[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "}"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "HTTP/1.1 403 Forbidden[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Type: application/json[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "x-amzn-RequestId: ec88e0db-e40d-11e6-bdf7-0be252654144[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Length: 877[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Connection: keep-alive[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
15:25:13.599 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/log-message-%2A/log-message/_search\n\naccept-encoding:gzip,deflate\ncontent-length:201\ncontent-type:application/json; charset=UTF-8\nhost:search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_45)\nx-amz-date:20170126T212513Z\n\naccept-encoding;content-length;content-type;host;user-agent;x-amz-date\nb279691bba767476d3f034826b94c64dafa3d888c775eda7582e9cb9d8a8932d'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20170126T212513Z\n20170126/us-west-2/es/aws4_request\n9d44dbc4f0ca90dc8f719077c1582dc27d73a3a5459c341872bf7c352f5559ed'\n"}"

Same search with index name of log-message-x

15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "POST /log-message-x/log-message/_search HTTP/1.1[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJFBO2C537SOTMKQQ/20170126/us-west-2/es/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;user-agent;x-amz-date, Signature=fe965a263ae8164c0ea951a41b188dd127a05afd859caf3cb071508188501046[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Length: 201[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Host: search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "x-amz-date: 20170126T212753Z[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "{[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "  "query": {[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "    "match_phrase": {[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "      "logFile": "test:test:test-hostname:rest-test-hostname.api.2017-01-26.21-27-36.a1726ebb-1845-4b49-82af-791662650e0c.txt:2017-01-26T21:27:38.901Z"[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "    }[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "  }[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "}"
15:27:53.338 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "HTTP/1.1 404 Not Found[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Access-Control-Allow-Origin: *[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Length: 335[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Connection: keep-alive[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"log-message-x","index":"log-message-x"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"log-message-x","index":"log-message-x"},"status":404}"

AWS signing fails if the URI has special character *

I was trying to delete indexes with pattern *_test.
AWS signing with the following debugging info :
The request signature we calculated does not match the signature you provided
...
The Canonical String for this request should have been\n'DELETE\n/%2A_test\n\naccept-encoding:gzip
...

Update
I guess the bug is when encoding the CanonicalURI

Update
My bad, have not seen the discussion on #18

Closing this in favour of #18

Thanks,
Vinay

403 Forbidden while accessing AWS Elastic Search with IAM Role.

Hi @ericcitaire ,
I was using this library to connect to AWS ES using IAM role. I have added role to my application and then created JestClient using AWS ES endpoint, please refer below code to create JestClient.

public SElasticSearchClient(final String esEndpoint) { this.esEndpoint = esEndpoint; final Supplier<LocalDateTime> clock = () -> LocalDateTime.now(ZoneOffset.UTC); AWSCredentialsProviderChain awsCredentialsProvider = new DefaultAWSCredentialsProviderChain(); final AWSSigner awsSigner = new AWSSigner(awsCredentialsProvider, "us-east-1", "es", clock); final JestClientFactory factory = new JestClientFactory() { @Override protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) { builder.addInterceptorLast(new AWSSigningRequestInterceptor(awsSigner)); return builder; } }; this.jestClient = initJestClient(factory); }

When I deployed this code and run my application it says error 403 user doesn't have access.
Error Message:
{ "Message": "User: arn:aws:sts::*****:assumed-role/search.bulk_ingest/DiscoveryDeAggregator-test is not authorized to perform: es:ESHttpPost on resource: discovery-content2" }
Please let me know what I am missing in code to create JestClient.
Any help will really appreciated
Thanks,
Anish

v0.0.13 appears to break signing

Here is the DEBUG-level logging from an example:

2016-08-07 19:08:49 DEBUG headers:135 - http-outgoing-0 >> POST /message/post/_search?search_type=scan&size=100&scroll=5m HTTP/1.1
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Accept-Encoding: gzip,deflate
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAJU34EHB4XVGMOWVA/20160808/us-east-1/es/aws4_request, SignedHeaders=accept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date, Signature=64d5917fa3d7905fb9852b55f3e7cf061ae6e2a5804cc303845790c97a3aab44
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Connection: Keep-Alive
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Content-Length: 26
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Content-Type: application/json; charset=UTF-8
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Host: search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0)
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> x-amz-date: 20160808T000849Z
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "POST /message/post/_search?search_type=scan&size=100&scroll=5m HTTP/1.1[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJU34EHB4XVGMOWVA/20160808/us-east-1/es/aws4_request, SignedHeaders=accept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date, Signature=64d5917fa3d7905fb9852b55f3e7cf061ae6e2a5804cc303845790c97a3aab44[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Content-Length: 26[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Host: search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0)[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "x-amz-date: 20160808T000849Z[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:86 - http-outgoing-0 >> "{"query":{"match_all":{}}}"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Content-Type: application/json[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Server: Jetty(8.1.12.v20130726)[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "x-amzn-RequestId: 4836bf0c-5cfc-11e6-970e-79636a7c0458[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Content-Length: 918[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Connection: keep-alive[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:86 - http-outgoing-0 << "{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/message/post/_search\nscroll=5m&search_type=scan&size=100\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:26\ncontent-type:application/json; charset=UTF-8\nhost:search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0)\nx-amz-date:20160808T000849Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\nbaa6846b65b050d71831bb2e4cd6e6f1593902f6d82b16a6c1f9979d14cfcd12'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160808T000849Z\n20160808/us-east-1/es/aws4_request\ne1a96e4e161e1a13e43eb5f0f5b25144d9921b797ed66fd5f08e88f14a29ab9c'\n"}"
2016-08-07 19:08:49 DEBUG headers:124 - http-outgoing-0 << HTTP/1.1 403 Forbidden
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Content-Type: application/json
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Server: Jetty(8.1.12.v20130726)
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << x-amzn-RequestId: 4836bf0c-5cfc-11e6-970e-79636a7c0458
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Content-Length: 918
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Connection: keep-alive

I rolled back to v0.0.12 and it appears to work just fine. Based on the nature of the change in v0.0.13, I suspect the issue has to do with moving to Keep-Alive connections.

Thanks for the work to get to v0.0.12, by the way -- v0.0.11 was choking on an "=" sign in a scroll ID! ๐Ÿ˜„

Super useful project!

signing fails with asterisks in query param value

When a query parameter value contains an asterisks, the request signing fails and AWS returns an error:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The reason is that AWS expects more characters in the URL to be encoded than what is necessary at the minimum. The Java SDK URLEncoder does not encode the asterisks. Here is the corresponding code from the AWS SDK: https://github.com/aws/aws-sdk-java/blob/7e064a48bee151ef222e0edea6157c1e65c7904a/aws-java-sdk-core/src/main/java/com/amazonaws/util/SdkHttpUtils.java#L66

Double urlEncode in AWSSigner.java on line 120

Hi,

I'm not sure if it's the correct behavior to urlEncode 2 times for the Uri at line number 120 in AWSSigner.java?

SdkHttpUtils.urlEncode(SdkHttpUtils.urlEncode(uri, true), true) + RETURN +

I've downloaded it and fix it to encode only one time to be able to work with AWS Signing 4

Any suggestion?

Cheers,
Arnon K.

Aws Signing issues in PutMapping Method

Hi

I raised an issue in StackOverflow we follow your project for AWS Signing and we have issues in PutMapping and Create Records on AWS Signing.

We are creating index, put mapping and insert record and rollback by deleting index in exceptions case. we faced an 403 forbidden error in PUTMapping and insertrecords This 403 forbidden error not raised for Index create and Delete. it looks like weird behaviour.

We are executing this Java Program as AWS Lambda. We provide Full Permission to ES in AWS Lambda IAM ROLE.

We are using aws-java-sdk (1.11.33), vc.inreach.aws (0.0.15), Jest Client (2.0.2) and AWS ES 2.3.

Attached our Code Part

HttpClientConfig clientConfig = new HttpClientConfig.Builder(esEndPoint).multiThreaded(true).build();
       JestClientFactory factory = new JestClientFactory();
       factory.setHttpClientConfig(clientConfig);
       JestClient jestClient = factory.getObject();
       System.out.println("Final Es Endpoint " + esEndPoint + "/"+indexName);

 //Check Index Exist in Elastic Search
       boolean indexExists = jestClient.execute(new IndicesExists.Builder(indexName).build()).isSucceeded();
       if (indexExists)
        esUtils.insertESData(jestClient, esRecords, indexName, docType);
       else
       {
        boolean isCreated = esUtils.createESIndex(jestClient, indexName, docType);
        if(isCreated)
         esUtils.insertESData(jestClient, esRecords,indexName,docType);
        else
         System.out.println("Record Insertion failure ");
       }

**Esutils Class**
public class ESUtils {
 
  public boolean createESIndex(JestClient jestClient, String indexName, String docType) throws Exception
  {
   System.out.println("ESUtils createESIndex " + indexName + "-" + docType);
   String date_prop1 = "{\"type\" : \"date\", \"format\" : \"" + Constants.SYSTEM_DATE_TIME_FORMAT1 + "\"}";
   String date_prop2 = "{\"type\" : \"date\", \"format\" : \"" + Constants.SYSTEM_DATE_TIME_FORMAT2 + 
     "||" + Constants.SYSTEM_DATE_TIME_FORMAT3 + "\"}";
   String timestamp_prop1 = "{\"type\" : \"date\", \"format\" : \"epoch_millis\"}";
   String timestamp_prop2 = "{\"type\" : \"date\", \"format\" : \"hour_minute_second_millis\"}";
   String map_properties = "{ \"date\" : " + date_prop1 + 
     ", \"messageDate\" : " + date_prop2 + 
     ", \"timestamp\" : " + timestamp_prop1 + " } ";
     /*", \"messageTime\" : " + timestamp_prop2 + */
   
   boolean esIndexFlag;  
   try {
       JestResult createResult = jestClient.execute(new CreateIndex.Builder(indexName).build());
       if(!createResult.isSucceeded())
       {
        System.out.printf("ESUtils createESIndex : JestResult error message %s and response code %s",  
          createResult.getErrorMessage(), createResult.getResponseCode());
        esIndexFlag = false;
       }
       else if(createResult.isSucceeded())
       {
        System.out.println("Create index"+createResult.getResponseCode());
        PutMapping putMapping = new PutMapping.Builder(
          indexName, docType, "{ \"" + docType + "\" : { \"properties\" : " + map_properties+"} }").build();
        JestResult map_result  = jestClient.execute(putMapping);
        
        if(map_result.isSucceeded())
         System.out.println("ESUtils createESIndex : Mapping Created");
        else
        {
         System.out.println("ESUtils createESIndex : Mapping not Created" + map_result.getErrorMessage());
         JestResult delete_result = jestClient.execute(new DeleteIndex.Builder(indexName).build());
         if(delete_result.isSucceeded())
          System.out.println("ESUtils createESIndex : Index Deleted");
         else
          System.out.println("ESUtils createESIndex : Index not Deleted"+delete_result.getErrorMessage());
        }
        esIndexFlag = true;
       }
       else
    {
        esIndexFlag = false;
        System.out.println("ESUtils createESIndex : Index not created");
    }
      }
      catch(Exception esExp)
      {
       esIndexFlag = false;
       System.out.println("ESUtils createESIndex : Index create Exception :" + esExp);
      }
      return esIndexFlag;
  }

 public boolean insertESData(JestClient jestClient, Map<String, Object> esRecords, String indexName, String docType){
   try {
    Index index = new Index.Builder(esRecords).index(indexName).type(docType).build();
    JestResult result = jestClient.execute(index);
    if(result.isSucceeded())
    {
     System.out.println("ESUtils insertESData Success Message : " + result.getJsonString());
     return true;
    }
    else
    {
     System.out.println("ESUtils insertESData Response code and Error message : " 
       + result.getResponseCode() + " : " + result.getErrorMessage());
     return false;
    }
   } catch (Exception exp) {
    System.out.println("ESUtils insertESData Exception :" + exp);
    return false;
   }
 }
}     

The Error we are getting for PutMapping and InsertRecords are 403 forbidden error.

Error try to connect ElasticSearch aws 6.2 with jest client

Hi

I worked this library in elasticsearch versions 2.3 and 5.5 and run corretly. But I try to pass ElasticSearch 6.2 and when I executed http petition get mapping (GET schneider/user/_mapping).

I recived next error: "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details"

Is possible change security parameters in this version ES ? because in previous versions run correlty,
Thanks

Jest client configuration

Is the below correct? The documentation is a bit vague...

What is the service value?

    private static final String SERVICE = "es";

    private String clusterUri;

    @Bean
    public JestClient jestClient() throws Exception {

        final Supplier<LocalDateTime> clock = new Supplier<LocalDateTime>() {
            public LocalDateTime get() {
                return LocalDateTime.now();
            }
        };


        DefaultAWSCredentialsProviderChain defaultAWSCredentialsProviderChain = new DefaultAWSCredentialsProviderChain();
        Region region = Region.getRegion(Regions.fromName("ap-southeast-2"));
        final AWSSigner awsSigner = new AWSSigner(defaultAWSCredentialsProviderChain, region.getName(),
                SERVICE, clock);

        final AWSSigningRequestInterceptor requestInterceptor = new AWSSigningRequestInterceptor(awsSigner);
        final JestClientFactory factory = new JestClientFactory() {
            @Override
            protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
                builder.addInterceptorLast(requestInterceptor);
                return builder;
            }

            @Override
            protected HttpAsyncClientBuilder configureHttpClient(HttpAsyncClientBuilder builder) {
                builder.addInterceptorLast(requestInterceptor);
                return builder;
            }
        };
        factory.setHttpClientConfig(new HttpClientConfig
                .Builder(clusterUri)
                .multiThreaded(true)
                .build());
        return factory.getObject();

    }

Cast Exception

Hi nice work!
i'm using your library to check and initialise an index on a ES and i'm having the following cast exception:

Caused by: java.lang.ClassCastException: org.apache.http.client.methods.HttpHead cannot be cast to org.apache.http.HttpEntityEnclosingRequest
    at vc.inreach.aws.request.AWSSigningRequestInterceptor.body(AWSSigningRequestInterceptor.java:59)
    at vc.inreach.aws.request.AWSSigningRequestInterceptor.process(AWSSigningRequestInterceptor.java:32)
    at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:48)
    at com.draios.services.ElasticSearchService.initSchema(ElasticSearchService.java:99)
    at com.draios.services.ElasticSearchService.<init>(ElasticSearchService.java:77)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

here is the code:

private void initSchema(JestClient esClient, String esIndex, String esEndpoint) throws IOException {
        boolean indexExists = esClient.execute(new IndicesExists.Builder(esIndex).build()).isSucceeded();
        if (indexExists) {
            logger.info("Index {} is already configured on {}", esIndex,esEndpoint);
        }else {
            logger.info("Index {} is not configured on {} creating", esIndex,esEndpoint);
            esClient.execute(new CreateIndex.Builder(esIndex).build());
        }
    }

using my a local elasticsearch node as endpoint it is working properly.

Upgrade dependencies + Security Patches

All the dependencies of this project has security vulnerabilities to some degree.
We've been able to dependency manage most of these but guava is not one I think we can manage.
Can you upgrade the dependencies to the latest and re-release?
Our security scanner recommends us migrate to guava 31.0.1-jre which to follow semantic versioning would require a major version bump of your libary.

Unit Tests Fail

https://github.com/inreachventures/aws-signing-request-interceptor/blob/master/src/test/java/vc/inreach/aws/request/test/SkdSignerUtil.java#L40-L48

The comment here is incorrect:

       Note: Double uri encoding is off simple before the signature does not match the expected signature of the test cases
       if it is enabled.  This was a bit unexpected because AWSElasticsearchClient (AWS SDK Class) enabled double URI encoding
       in the signer by default.  I can only assume that double encoding is needed when accessing the service but not when accessing
       elasticsearch.

S3 expects single encoding. ElasticSearch expects double encoding.
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

Changing the AWS4Signer 'doubleUrlEncoding' constructor parameter to true in SkdSignerUtil.java (sic.) makes the unit tests pass, as the documents would suggest.

PR #28 (where the change was introduced) also notes that the double-encoding is, in-fact, correct. It appears that the change was not backed out of the unit test code.

Adding port to ES URL breaks signature

Adding :80 to the host in the ES server URL breaks signing. Amazon returns an error showing what they used to compute the signature and the port is missing, so the request fails.

This took me a while to pin down but glad it turned out to be something easy to fix (on my end at least).

Thanks for this helpful interceptor!

scan and scroll query failed

When I write a scroll query like this:
SearchScroll searchScroll = new SearchScroll.Builder(_scrollId, _scrollExpirationTime).build();

AWSSigningRequestInterceptor throw an exception:
Caused by: java.lang.IllegalArgumentException: Chunk [scroll_id=c2NhbjsxOzc3Mjo5WGljUUFNeVJGcVdDSzBjaUVQcDJ3OzE7dG90YWxfaGl0czo1NTg0Ow==] is not a valid entry at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at com.google.common.base.Splitter$MapSplitter.split(Splitter.java:512) at com.logicmonitor.lmes.elasticsearch.aws.AWSSigningRequestInterceptor.params(AWSSigningRequestInterceptor.java:)

I use JestClient 0.1.6

Does you library support streaming?

I have created repository with bug reproducing.

This repo contains a maven project with 2 java classes NonWorkingExampleStreamVersion, WorkingExampleNonStreamVersion. Both of them has main method. Both take file from s3 and try to store it in aws elasticsearch. But:

WorkingExampleNonStreamVersion - load whole file content into byte stream, creates base64 from it and then send file to elasticsearch
NonWorkingExampleStreamVersion - do the same thins as previous but instead of downloading whole file it wrap stream and uploads bytes to elaticsearch only when those byte arrives from s3. This version is does not work due to:

     "message": "The request signature we calculated does not match the signature you provided. Check > your AWS Secret Access Key and signing method. Consult the service documentation for details.

More details you can see on reaadme.md file. Feel free to ask me any questions.

Signature fails if query string has empty value

If a query String contains "key=" without value, the signature fails because the "=" is URL-encoded.
Solution: In AWSSigningRequestInterceptor change line 64 from
if (index > 0 && pair.length() > index + 1) {
to
if (index > 0 && pair.length() > index) {

GET content-length header generated incorrectly for signature

It looks like 32f858a broke signing GETs.

I now get a 403 response with the following message (slightly sanitized):

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'GET
/tetris_test/split_records/3A1

accept-encoding:gzip,deflate
connection:close
content-length:
host:SANITIZED.us-west-2.es.amazonaws.com
user-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_91)
x-amz-date:20160510T174638Z

accept-encoding;connection;content-length;host;user-agent;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20160510T174638Z
20160510/us-west-2/es/aws4_request
a7d878202cc3ba4b24808058eb5f37c1d9b30c8eda9b12201bfc047201d48e23'

When I look at the Canonical String that was actually used for signing, it looks like this (note the "0" for content-length):

GET
/tetris_test/split_records/3A1

accept-encoding:gzip,deflate
connection:close
content-length:0
host:SANITIZED.us-west-2.es.amazonaws.com
user-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_91)
x-amz-date:20160510T174638Z

accept-encoding;connection;content-length;host;user-agent;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Port back to Java 7

I work in a limited environment where the latest Java I can use is 7 (actually, it's a C program calling into Java). Is it possible to port this lib back to Java 7?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.