An extension to the Amazon SQS client that enables sending and receiving messages up to 2GB via Amazon S3.

License: Apache License 2.0

Java 100.00%

amazon-sqs-java-extended-client-lib's Introduction

Amazon SQS Extended Client Library for Java

The Amazon SQS Extended Client Library for Java enables you to manage Amazon SQS message payloads with Amazon S3. This is especially useful for storing and retrieving messages with a message payload size greater than the current SQS limit of 256 KB, up to a maximum of 2 GB. Specifically, you can use this library to:

Specify whether message payloads are always stored in Amazon S3 or only when a message's size exceeds 256 KB.
Send a message that references a single message object stored in an Amazon S3 bucket.
Get the corresponding message object from an Amazon S3 bucket.
Delete the corresponding message object from an Amazon S3 bucket.

You can download release builds through the releases section of this project.

For more information on using the amazon-sqs-java-extended-client-lib, see our getting started guide here.

Getting Started

Sign up for AWS -- Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see AWS Account and Credentials in the AWS SDK for Java Developer Guide.
Sign up for Amazon SQS -- Go to the Amazon SQS console to sign up for the service.
Minimum requirements -- To use the sample application, you'll need Java 7 (or later) and Maven 3. For more information about the requirements, see the Getting Started section of the Amazon SQS Developer Guide.
Download -- Download the latest preview release or pick it up from Maven:

Version 2.x

  <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>amazon-sqs-java-extended-client-lib</artifactId>
    <version>2.1.1</version>
    <type>jar</type>
  </dependency>

Version 1.x

  <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>amazon-sqs-java-extended-client-lib</artifactId>
    <version>1.2.5</version>
    <type>jar</type>
  </dependency>

Further information - Read the API documentation.

Feedback

Give us feedback here.
If you'd like to contribute a new feature or bug fix, we'd love to see Github pull requests from you.

amazon-sqs-java-extended-client-lib's People

Contributors

Stargazers

Watchers

Forkers

cfinerty jarwol glidester navinvishy nexstra yodasantu bullhorn jamestyrrell qinnnyul krithikaraghavan enriquecoge9 peterosudio sudheer2845 trinhtomsk copoo foodessentials dsw2127 ken-sheppard choeflake nithinvenugopal virginpulsepublic shs174 ss37 jorgemontes jesse-heath wilq87 damiox umeshh smadasu kevinjdinicola greyfairer solo810 dharmeshspatel4u maityh alexanderfedin jmcazaux varunmehta rajhiren jkuehl2 aravindmjrao aditya-triconinfotech williamweiwei nick-anderssohn svoeller99 acalvin-smartsheet ruperrrt ashleyfrieze adeepak vinodh1989 omkarjoshi awsportela manickam91 sumitrsystems hellostephenli cb-manideep barnettj ronitabuzo-ey mredolatti adieldarsfdc jamesanto msailes adam-aws dicondur tomcat33031 mngo87 chockstone jagadeeshvenkatesh corporateadedayo prower-turnitin philliprower unni-p skbuser nangal potatospud harsha-changejar hvardhan617 juandavidg890121 jurabek shotmk bright martin-massicotte tryflux saurabh-tuc kfadillah thread-sleep shih richardsmithsfdc vbuitoni hasskell ziyanli-amazon srinivasaraogorle bibhu107 evangilo mia-li-z arus2023 tingtingtao1112 scottgerring jagdev-aws pomadchin

amazon-sqs-java-extended-client-lib's Issues

Doesn't work with large batches

Messages are looked at individually as to whether they exceed the maximum, but a SendMessageBatchRequest might exceed 256kb in aggregate even if none of the individual messages do. It is somewhat confusing given this library appears to support batching.

Deleting from FIFO SQS extended after Visibility time got over

We have found an issue when we try to confirm our messages have been processed using a FIFO SQS extended.
When the delete happens it is basically deleting iniatially from the S3 bucket and after that's done it tries to remove from the queue, basically an AmazonSQSException is raised saying the ReceiptHandler is expired, and as a result the message stays in the queue, however the payload is correctly deleted from the S3 bucket.
As a result the message is repicked by the queue and when it does try to read it again an AmazonS3Exception is raised saying there is no payload related to the message from the queue.
I think the delete operation should be somehow atomic as the message in the queue means nothing without the real content from S3.

We have experimented around changing the order of the operations, so removing first from the queue and later on the S3 bucket and it works better, but from our point of view this should be provided by the API itself.

Let me know if you need any further information.

Question on Spring AWS QueueMessagingTemplate and Extended Client

Is there a plan to update the extended client to support the Spring AWS QueueMessagingTemplate? Currently, QueueMessagingTemplate only accepts the async client (AmazonSQSAsync), while the extended client only supports the standard client (AmazonSQS). This update would be beneficial for us since we rely on QueueMessagingTemplate for sending and receiving messages.

Deleting SQS Message when S3 file not found

We found an issue when receiving a message from SQS with S3 pointer but the file's has been deleted from S3 this library only throws an exception. However, we would like to delete the SQS message when this issue happen.
Getting SQS message/handler will help client to decide how to process the message in this case.

Wrong attributes requested from messages

I'm using latest version (2.0.0).
I have messages in my queue and payloads in S3 but it fail to read "messages attributes". It request "attributes".
And without "message attributes", it's like my payload is not in S3.

Code to change :
com/amazon/sqs/javamessaging/AmazonSQSExtendedClient.java:325
receiveMessageRequestBuilder.attributeNamesWithStrings(messageAttributeNames);

Thanks

JavaScript Version of this Library

Is there a plan for a JavaScript version of this library?

It would be really useful to have one, not only for JavaScript, but for other languages with AWS SDK support.

Thank you.

Not compatible with aws-java-sdk-bundle

aws-java-sdk-bundle is one of the modules in the AWS SDK repository. Its purpose is to shade all of the SDK's third-party dependencies such as Jackson or org.apache.commons.logging.

In the AmazonSQSExtendedClient class there is a method getTextFromS3(). Within that method, there is a call to IOUtils.closeQuietly(is, LOG);. LOG is of type org.apache.commons.logging.Log. The problem is, if you use the shaded version of the SDK, then IOUtils.closeQuietly is expecting com.amazonaws.thirdparty.apache.logging.Log instead, which causes it to throw a java.lang.NoSuchMethodError.

I believe this problem could be solved by converting this library into multiple modules and offer a shading module with the exact same rules as aws-java-sdk-bundle.

MessageS3Pointer should be public

I am trying to retrieve the s3 object url, but the MessageS3Pointer class is not public.
Is there any reason why it cannot be public?

Allow Custom Message Size Threshold/Include Message Attributes' size in threshold

Currently, DEFAULT_MESSAGE_SIZE_THRESHOLD = 256KB. This doesn't help if SQS messages have MessageAttributes that are not included in the size computed for the message.

One of the following would help:

Have a method that allows custom message size threshold in ExtendedClientConfiguration class
(or) While computing message size, include message attributes along with message content to calculate the actualy size of SQS message.

Reference: "All parts of the message attribute, including name, type, and value, are included in the message size restriction, which is currently 256 KB (262,144 bytes)." https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-attributes.html

Cross language incompatibility : java namespace in sqs json body causing deserialization issue on dotnet

Hi!

First of all, thank you for your amazing work done.

The java library is not compatible with the dotnet library and others since the java namespace is located inside a dynamic array raol/amazon-sqs-net-extended-client-lib#32.

In golang, the library tends to support the latest version but still, previous java library with other namespace is not compatible when the golang client sends a sqs message to a java consumer. Here is an attempt which is not compatible with older java versions: https://github.com/jwcorle/sqs-extended-client-go/blob/main/client.go#L201

I find that putting such namespace is annoying, how about using only the pointer with the bucketname and the s3key like in dotnet and golang

Implement Payload Offloading Java Common Library For AWS

Implement the new Common Library for AWS messaging in the SQSExtendedClient

How to identify file from S3 with message in the queue.

Hi, I am planning to use this library but I can't find any documentation on how we can find the related file in the bucket.

My use case is if some message fails to be processed or drops to the dead letter queue I would like to see and look through the contents of the file manually. Is it possible to achieve this with this library?

Cross account support

Would be great to have a bit more support for when the S3 resource is in a different account. As it stands, if we were to put the object to another account's S3 bucket, we need to grant the bucket owner access to it.

We could extend the ExtendedClientConfiguration class to provide configuration for canned acl like this barnettj@0ca7b16.

Or we could even just make the putObject function protected and will give the consumers flexibility to extend and override the putObject method and apply whatever configuration they want to the request barnettj@3b25975.

Exeption on AmazonSQSExtendedClient.deleteMessage

Just testing the sample App and got an error:

Exception in thread "main" java.lang.AbstractMethodError: com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.deleteMessage(Lcom/amazonaws/services/sqs/model/DeleteMessageRequest;)Lcom/amazonaws/services/sqs/model/DeleteMessageResult;
    at com.di.maven.SQSExtendedClient.main(SQSExtendedClient.java:105)
    at com.di.maven.App.main(App.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Line: 105
sqsExtended.deleteMessage(new DeleteMessageRequest(myQueueUrl, messageReceiptHandle));

Maven's dependecy:

`dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.11.14</version>
    </dependency>
    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>amazon-sqs-java-extended-client-lib</artifactId>
      <version>1.0.0</version>
    </dependency>

Any solutions?

Using the library with lambda as event source mapping

We want to use the library in lambda handler which reads from SQS via event source mapping.
The issue is, SQS sends to the lambda SQSEvent which already includes the messages body.
We need to have a public API to extract the actually payload and takes SQSEvent as input instead of ReceiveMessageRequest.

aws-sqs protocol is not supported

Hi,

I have a big message that I want to send to sqs, so I 'm trying to use amazon-sqs-java-extended-client-lib. I'm following the aws example, but I'm getting an error.

This is the code



public class ExceptionProcessor {

    final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();


    final ExtendedClientConfiguration extendedClientConfig =
            new ExtendedClientConfiguration()
                    .withLargePayloadSupportEnabled(s3, "price-engine-exception");

    final AmazonSQS sqsExtended =
            new AmazonSQSExtendedClient(AmazonSQSClientBuilder
                    .defaultClient(), extendedClientConfig);

    public void handleOrder(@Headers Map<?, ?> in, @Body String payload, @OutHeaders Map<String, Object> out) {

        // Send the message.
        final SendMessageRequest myMessageRequest =
                new SendMessageRequest("aws-sqs://dev-prices-to-price-engine-dlq?amazonSQSClient=#sqsClient", payload);
        sqsExtended.sendMessage(myMessageRequest);
    }
}

this is the full error log.

com.amazonaws.SdkClientException: Unable to execute HTTP request: null
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2207) ~[aws-java-sdk-sqs-1.11.687.jar:na]
	at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2174) ~[aws-java-sdk-sqs-1.11.687.jar:na]
	at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2163) ~[aws-java-sdk-sqs-1.11.687.jar:na]
	at com.amazonaws.services.sqs.AmazonSQSClient.executeSendMessage(AmazonSQSClient.java:1762) ~[aws-java-sdk-sqs-1.11.687.jar:na]
	at com.amazonaws.services.sqs.AmazonSQSClient.sendMessage(AmazonSQSClient.java:1734) ~[aws-java-sdk-sqs-1.11.687.jar:na]
	at com.amazon.sqs.javamessaging.AmazonSQSExtendedClientBase.sendMessage(AmazonSQSExtendedClientBase.java:115) ~[amazon-sqs-java-extended-client-lib-1.0.2.jar:na]
	at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.sendMessage(AmazonSQSExtendedClient.java:176) ~[amazon-sqs-java-extended-client-lib-1.0.2.jar:na]
	at com.boohoo.esbdefused.priceengine.routes.ExceptionProcessor.handleOrder(ExceptionProcessor.java:34) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:481) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:300) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:273) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:198) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:53) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:82) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:1063) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:474) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:76) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.aggregate.AggregateProcessor$1.run(AggregateProcessor.java:791) ~[camel-core-2.24.1.jar:2.24.1]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at org.apache.camel.util.concurrent.SynchronousExecutorService.execute(SynchronousExecutorService.java:62) ~[camel-core-2.24.1.jar:2.24.1]
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[na:na]
	at org.apache.camel.processor.aggregate.AggregateProcessor.onSubmitCompletion(AggregateProcessor.java:783) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.aggregate.AggregateProcessor.access$1900(AggregateProcessor.java:88) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:1160) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.onEviction(AggregateProcessor.java:1117) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:229) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.processor.aggregate.AggregateProcessor$AggregationTimeoutMap.purge(AggregateProcessor.java:1131) ~[camel-core-2.24.1.jar:2.24.1]
	at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:179) ~[camel-core-2.24.1.jar:2.24.1]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.apache.http.client.ClientProtocolException: null
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.9.jar:4.5.9]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.9.jar:4.5.9]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.9.jar:4.5.9]
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) ~[aws-java-sdk-core-1.11.687.jar:na]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) ~[aws-java-sdk-core-1.11.687.jar:na]
	... 58 common frames omitted
Caused by: org.apache.http.HttpException: aws-sqs protocol is not supported
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:89) ~[httpclient-4.5.9.jar:4.5.9]
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) ~[httpclient-4.5.9.jar:4.5.9]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.9.jar:4.5.9]

I can see that the message was sent to S3, but I don't understand why I am seeing this error.

What am I missing?

nodejs version

Hi All! Is there a nodejs version of this library?

Maven Central Jar Compiled with Java 1.8?

When I switch my application to use the latest 1.0.1 version of this jar and pull it from Maven Central, I get the following exception:

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/amazon/sqs/javamessaging/ExtendedClientConfiguration : Unsupported major.minor version 52.0, compiling:(/tmp/form-init8767020834767061108.clj:1:73)
        at clojure.lang.Compiler.load(Compiler.java:7391)
        at clojure.lang.Compiler.loadFile(Compiler.java:7317)
        at clojure.main$load_script.invokeStatic(main.clj:275)
        at clojure.main$init_opt.invokeStatic(main.clj:277)
        at clojure.main$init_opt.invoke(main.clj:277)
        at clojure.main$initialize.invokeStatic(main.clj:308)
        at clojure.main$null_opt.invokeStatic(main.clj:342)
        at clojure.main$null_opt.invoke(main.clj:339)
        at clojure.main$main.invokeStatic(main.clj:421)
        at clojure.main$main.doInvoke(main.clj:384)
        at clojure.lang.RestFn.invoke(RestFn.java:421)
        at clojure.lang.Var.invoke(Var.java:383)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.Var.applyTo(Var.java:700)
        at clojure.main.main(main.java:37)
Caused by: java.lang.UnsupportedClassVersionError: com/amazon/sqs/javamessaging/ExtendedClientConfiguration : Unsupported major.minor version 52.0

I see in the pom.xml the compiler plugin is set to use Java 1.7 - is it possible though that the jar pushed to Maven Central was built with the wrong JDK?

For what it's worth, if I clone this repo and build it using my 1.7 JDK, I stop experiencing the error.

Thanks!

api deleteMessage(DeleteMessageRequest deleteMessageRequest) not deleting message from S3 even when doesCleanupS3Payload = true

api is calling internally isS3ReceiptHandle(receiptHandle) which will always return false as receiptHandle does not contain
receiptHandle.contains(SQSExtendedClientConstants.S3_BUCKET_NAME_MARKER)
&& receiptHandle.contains(SQSExtendedClientConstants.S3_KEY_MARKER);

payloadStore.deleteOriginalPayload(messagePointer) is called only when isS3ReceiptHandle returns true.

Sounds like bug. If not bug, please let me know how receiptHandle will contain requiredText while sending message from extendedClient

not compatible with java sdk2

Is it planned to migrate to aws java sdk2?

FIFO Support

I changed our existing queue to wrapper it with this extended support. We are seeing this now:
The input receipt handle is invalid. (Service: AmazonSQS; Status Code: 404; Error Code: ReceiptHandleIsInvalid; Request ID: ba9421e9-a9d2-56ba-8e17-70ff7190f05a)

This is on consuming the queue, we are able to put a message on the queue, and the body is being stored on the S3 bucket.

AmazonSQSExtendedClient: Failed to get the S3 object which contains the message payload. Message was not received.

Team, somehow I tried to understand the flow.. but am not sure it is right or wrong.

I am using sqs extended client to support large payload on my app.

when ever payloads available in queue working super but when the queue is empty am able to see SQSMessageConsumerPrefetch warning log after few seconds am getting AmazonSQSExtendedClient error log and my console/log file is fulled with these errors only.

Please handle/ignore it.

`2019-07-23 12:13:47.161 WARN 21778 --- [refetchThread-2] c.a.s.j.SQSMessageConsumerPrefetch : Encountered exception during receive in ConsumerPrefetch thread

javax.jms.JMSException: AmazonServiceException: receiveMessage. RequestId: null
HTTPStatusCode: 0 AmazonErrorCode: null
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.handleException(AmazonSQSMessagingClientWrapper.java:432) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.receiveMessage(AmazonSQSMessagingClientWrapper.java:353) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.getMessages(SQSMessageConsumerPrefetch.java:262) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.run(SQSMessageConsumerPrefetch.java:221) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: com.amazonaws.AmazonServiceException: Failed to get the S3 object which contains the message payload. Message was not received. (Service: null; Status Code: 0; Error Code: null; Request ID: null)
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.getTextFromS3(AmazonSQSExtendedClient.java:1178) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.receiveMessage(AmazonSQSExtendedClient.java:351) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.receiveMessage(AmazonSQSMessagingClientWrapper.java:351) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
... 5 common frames omitted
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: null; S3 Extended Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1660) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1324) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1074) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1427) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.getTextFromS3(AmazonSQSExtendedClient.java:1174) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
... 7 common frames omitted

2019-07-23 12:14:17.186 ERROR 21778 --- [refetchThread-2] c.a.s.j.AmazonSQSExtendedClient : Failed to get the S3 object which contains the message payload. Message was not received.

com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: null; S3 Extended Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1660) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1324) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1074) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1427) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.getTextFromS3(AmazonSQSExtendedClient.java:1174) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.receiveMessage(AmazonSQSExtendedClient.java:351) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.receiveMessage(AmazonSQSMessagingClientWrapper.java:351) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.getMessages(SQSMessageConsumerPrefetch.java:262) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.run(SQSMessageConsumerPrefetch.java:221) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2019-07-23 12:14:17.186 ERROR 21778 --- [refetchThread-2] c.a.s.j.AmazonSQSMessagingClientWrapper : AmazonServiceException: receiveMessage. RequestId: null
HTTPStatusCode: 0 AmazonErrorCode: null

com.amazonaws.AmazonServiceException: Failed to get the S3 object which contains the message payload. Message was not received. (Service: null; Status Code: 0; Error Code: null; Request ID: null)
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.getTextFromS3(AmazonSQSExtendedClient.java:1178) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.receiveMessage(AmazonSQSExtendedClient.java:351) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.receiveMessage(AmazonSQSMessagingClientWrapper.java:351) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.getMessages(SQSMessageConsumerPrefetch.java:262) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at com.amazon.sqs.javamessaging.SQSMessageConsumerPrefetch.run(SQSMessageConsumerPrefetch.java:221) ~[amazon-sqs-java-messaging-lib-1.0.5.jar:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: null; S3 Extended Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1660) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1324) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1074) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[aws-java-sdk-core-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1427) ~[aws-java-sdk-s3-1.11.415.jar:na]
at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.getTextFromS3(AmazonSQSExtendedClient.java:1174) ~[amazon-sqs-java-extended-client-lib-1.0.1.jar:na]
... 7 common frames omitted`

[Question] Why is the SQS message deleted after the S3 object is deleted, and not vice versa?

Python version of this Library

Hi Guys

Is there a python version of this library? I seem to only see dot net and java.

Question

Does amazon-sqs-java-extended-client-lib support async SQS client with AWS SQS version 2? If yes, where can I find the documentation?

Support for SQS Lambda source event

AWS now supports using SQS as a Lambda event source. https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/ Events containing messages created with the extended client are hard to handle. Would be great to have a mechanism of deserializing the message and fetching the document from S3 if the message contains a reference.
Thanks.

Question & suggestion

Is there any plan to bring the extended client in line with the standard client and produce an async version (or is this not possible due to the interactions with s3)?

It would be helpful if I could configure the client not to delete the s3 component of the message on the message delete action. This would then leave me an audit log of all the messages I'd received and I could then use the life-cycle attribute of the bucket to clear out the older messages that I no longer wish to retain.

Thanks,

Tim

Why 2G limit ?

This is great, I just found this looking up SQS pricing -- I wrote a simplier version of this years ago.
Question: What is the reason for the 2G limit ? I have a use case where files can vary from 100bytes to 6+G ... Of course I can split them up, but since I currently use S3 and upload/download them with no problem wondering what it would take to break the 2G limit ?

( forking right now ... )

-David

Test for issue submission

This is a test message for issue submission via https://github.com/awslabs/amazon-sqs-java-extended-client-lib/issues.

Set ObjectMetaData || BucketKeyEnabled || Cost Savings

Hi,

It's not an issue with with library , but a general query .

We have enabled Bucket Key to save on cost for making KMS API calls , which would help in reducing the cost.

Does this library supports in checking the AWS S3 ObjectMetaData BucketKeyEnabled field while the Object is put into a Bucket?

If not, is there an API available in ExtendedClientConfiguration class, which can be leveraged to set the value?

Thanks in advance.

Milind

sqsExtended.receiveMessage(request).getSdkHttpMetadata() is returned as null

SDK <groupId>com.amazonaws</groupId> <artifactId>amazon-sqs-java-extended-client-lib</artifactId> <version>1.2.4</version>
We have a message in a SQS, when fetching the message from the queue using the receiveMessage method, we are checking for the presence of HTTP Meta Data. SDK is returning null for the sqsExtended.receiveMessage(request).getSdkHttpMetadata().

No upgrade path from 1.0.x to 1.1.0 due to change in serialized pointer

There is no way to incrementally upgrade from 1.0.2 to 1.1.0 because the "pointer" payload has changed from:

["com.amazon.sqs.javamessaging.MessageS3Pointer", {"s3BucketName": "...", "s3Key": "..."}]

...to:

["software.amazon.payloadoffloading.PayloadS3Pointer", {"s3BucketName": "...", "s3Key": "..."}]

The former no longer exists in 1.1.0, and the latter never existed in 1.0.x. That means I can't upgrade my sender to 1.1.0 because it will send PayloadS3Pointer, which the receiver will not recognize, and I can't upgrade my receiver to 1.1.0 because senders are still sending MessageS3Pointer, which will no longer be recognized. The only option is to upgrade all senders and receivers at the same time, which is a non-starter for our services. What is the recommended upgrade path for users of amazon-sqs-java-extended-client-lib?

There are various tricks with @JsonTypeInfo(use=Id.NAME, include=As.WRAPPER_ARRAY), @JsonTypeName, and ObjectMapper.registerSubtypes that could be used to control which names can be read/written using Jackson, but it might be easiest to just read/write raw JSON instead. At this point, you probably don't want to just use those mechanisms to revert to the old type since that would similarly break anyone now using PayloadS3Pointer, so perhaps it would be best to add ExtendedClientConfiguration.useLegacyPointerType that causes the client to write the old MessageS3Pointer type in the JSON, and then update JsonDataConverter to support either the old or new type. Projects using 1.0.x would still be broken when upgrading to 1.1.0, but they can at least remain compatible by opting into the compatibility boolean.

SNS equivalent?

Is there an equivalent library that would handle large messages to SNS in the same way? Any plans?

This library is not well maintained. Can it be hosted by somebody else, or given more attention please?

Firstly, thanks for being great awslabs, but this library does not seem to be well maintained, and i get that these libraries don't exist for python, and JavaScript so I am thankful that it exists in Java.

In saying that, I think this PR:
#58
fixes the second most popular issue:
#35
The first most popular issue being a request for python. I think it should have been merged months ago. It has been open for almost a year.

Can this library please be given more attention, or can we have another maintainer, or it is possible to just get an aws sdk v2 build? Going to aws sdk v2 is a lot of work

Consuming SQSEvent from lambda

When consuming large payload event from SQS in aws lambda, should we build logic to get the payload from S3 and process?

Retention of S3 object once message consumed by queue.

One part of the outstanding pull request #6 addresses this.

If a queue is one of many subscribers to a topic, then we want all the queues to have a 'fair go' at processing a message before the max retention period of the topic lapses.

With the current setup, the deletion of a message from the queue using the extended lib deletes the underlying S3 bucket. This is fine when the queue is the only consumer of the message (and implicitly the S3 bucket) but with a topic and queue setup the first consumer will stop any other subscriber from getting the contents of the S3 bucket.

We can currently work around this by writing custom code to override the behaviour of the client when deleting a message, but this seems flaky given we don't control the library itself.
Would be great to get a configuration option that leaves the message in S3 when a consumer sends a delete message request . This way we could let the topic drive when an S3 bucket is finally cleaned up.

Cannot process an SQS message sent from extended client in a lambda triggered from SQS

If an extended client was used to send a message to SNS, and then an SQS queue is used to trigger a lambda on receipt then it's not possible to use the extended client functionality to get the original message (i.e. download the payload).

The code that actually handles an incoming message should be a public function that can be used in other environments where we have an SQS message already which we need to get the full payload.

SQS Extended Client Lib not compatible with Core SDK 1.11.xx

We are using SQS Extended Client Lib. Based on a recent announcement on IAM Roles for Tasks we are going to upgrade AWS SDK to the latest 1.11 version. But AmazonSQS interface in the latest SDK has changed, e.g. setQueueAttributes returns SetQueueAttributesResult instead of void while this lib is implementing the old AmazonSQS interface. So we are not able to use amazon-sqs-java-extended-client-lib 1.0 with Core SDK 1.11.xx.

This is blocking the upgrade to the latest AWS Java SDK.

The library jar picked up from maven repository is having issue.

The latest source when compiled and build from https://github.com/awslabs/amazon-sqs-java-extended-client-lib is working fine for deleting messages from SQS backed by S3 for large payloads.
However the same is not working when the jars are picked from maven repository.
Please do resolve it and push the latest one to maven.

Issue faced when the jar is picked from maven: (when attempting to delete SQS messages)

Exception in thread "main" java.lang.AbstractMethodError: com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.deleteMessage(Lcom/amazonaws/services/sqs/model/DeleteMessageRequest;)Lcom/amazonaws/services/sqs/model/DeleteMessageResult;
    at org.nfjs.aws.sqs.SQSExtendedClient.deleteMessage(SQSExtendedClient.java:149)
    at org.nfjs.aws.sqs.SQSExtendedClient.pollForMessages(SQSExtendedClient.java:139)
    at org.nfjs.aws.sqs.SQSExtendedClient.receiveMessage(SQSExtendedClient.java:108)
    at org.nfjs.aws.sqs.SQSExtendedClient.main(SQSExtendedClient.java:63)

Client adds a message attribute name to ReceiveMessageRequest for every call

AmazonSQSExtendedClient.receiveMessage(ReceiveMessageRequest) always appends SQSExtendedClientConstants.RESERVED_ATTRIBUTE_NAME to the list of message attribute names of the request passed in. Therefore if the request object is re-used for multiple invocations, this request can grow and grow. Eventually, if allowed to grow over time, SQS will deny the request with a 413 Request Entity Too Large. This seems to happen when the header gets to be somewhere around 1MB.

com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 413; Error Code: 413 Request Entity Too Large; Request ID: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
        at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1792)
        at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1768)
        at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1374)
        at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1350)
        at com.amazon.sqs.javamessaging.AmazonSQSExtendedClientBase.receiveMessage(AmazonSQSExtendedClientBase.java:217)
        at com.amazon.sqs.javamessaging.AmazonSQSExtendedClient.receiveMessage(AmazonSQSExtendedClient.java:349)

Example of message attribute names over four invocations:

MessageAttributeNames: [SQSLargePayloadSize]
MessageAttributeNames: [SQSLargePayloadSize, SQSLargePayloadSize]
MessageAttributeNames: [SQSLargePayloadSize, SQSLargePayloadSize, SQSLargePayloadSize]
MessageAttributeNames: [SQSLargePayloadSize, SQSLargePayloadSize, SQSLargePayloadSize, SQSLargePayloadSize]

S3 Object encryption

Great library!

I noticed the messages stored in S3 do not have encryption turned on. For the best HIPAA compliance, I think they should be? If this is useful, I could work on it; we'd need to add to the API the ability to set the S3 region.

OWASP Vulnerability

Good Morning,

We're really interested in using this useful dependency to handle large messaging be we are unable to get it to pass our OWASP vulnerability scan (that we cannot mitigate through exclusions or newer versions) that prevents us from using it. I'm reluctant to suppress this warning as it would need to be propagated into suppression files for all service that would support large messaging.

Are there any plans to address this vulnerability in a new release? I've included the relevant section from the vulnerability report.

Many Thanks,

Chris


License:

Apache License, Version 2.0: https://aws.amazon.com/apache2.0
MD5: 97b38556515d9f53d61a3111a4538f54
SHA1: 76ab7349ef513485895d80051bafe88a1d1d8f2d 
Referenced In Project/Scope: messaging-utility:compile
Evidence
Identifiers
cpe: cpe:/a:amazon_aws_project:amazon_aws:1.0.2   Confidence:Low   suppress
maven: com.amazonaws:amazon-sqs-java-extended-client-lib:1.0.2 ✓   Confidence:Highest
Published Vulnerabilities
CVE-2015-3373  suppress

Severity: Medium 
CVSS Score: 5.0 (AV:N/AC:L/Au:N/C:P/I:N/A:N) 
CWE: CWE-200 Information Exposure

The Amazon AWS module before 7.x-1.3 for Drupal uses the base URL and AWS access key to generate the access token, which makes it easier for remote attackers to guess the token value and create backups via a crafted URL.

BID - 74277
CONFIRM - http://cgit.drupalcode.org/aws_amazon/commit/?id=9377a26
CONFIRM - https://www.drupal.org/node/2415457
MISC - https://www.drupal.org/node/2415873
MLIST - [oss-security] 20150129 Re: CVEs for Drupal contributed modules - January 2015
Vulnerable Software & Versions:

cpe:/a:amazon_aws_project:amazon_aws:7.x-1.2::~~~drupal~~ and all previous versions```

AmazonSQSExtendedClient does not have an implementation

Hi.
First of all I want to thank you for this solution.
I have an implementation that works fine, but when I updated AWSSDK.Core, AWSSDK.S3, AWSSDK.SimpleNotificationService and AWSSDK.SQS the client stopped working.

Now I have got the error.

System.TypeLoadException: 'Method 'get_Paginators' in type 'Amazon.SQS.ExtendedClient.AmazonSQSExtendedClient' from assembly 'Amazon.SQS.ExtendedClient, Version=1.2.1.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.'

try            
{
    _queueUrl = config.Url;
    _s3Client = new AmazonS3Client(new BasicAWSCredentials(config.AccessKey, config.SecretKey), Amazon.RegionEndpoint.GetBySystemName(config.Region));
    _sqsClient = new AmazonSQSClient(config.AccessKey, config.SecretKey, Amazon.RegionEndpoint.GetBySystemName(config.Region));
    _extendedClient = new AmazonSQSExtendedClient(_sqsClient, new ExtendedClientConfiguration().WithLargePayloadSupportEnabled(_s3Client, config.BucketName));
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Constructor: SqsService");
        throw;
    }
}

I'll thank you again if help me!

sendMessageBatch method sends the original messages instead of references to s3 to SNS

sendMessageBatch is sending the original messages instead of references to s3 bucket.
Files are uploaded to s3 too though.

Looks like we are building batchEntries in AmazonSQSExtendedClient:618 - AmazonSQSExtendedClient:628, but we forget to update the builder with new entries before submitting the request to underlying sqsClient instance.

List<SendMessageBatchRequestEntry> batchEntries = new ArrayList<>(sendMessageBatchRequest.entries().size());

        for (SendMessageBatchRequestEntry entry : sendMessageBatchRequest.entries()) {
            //Check message attributes for ExtendedClient related constraints
            checkMessageAttributes(entry.messageAttributes());

            if (clientConfiguration.isAlwaysThroughS3() || isLarge(entry)) {
                entry = storeMessageInS3(entry);
            }
            batchEntries.add(entry);
        }

        return super.sendMessageBatch(sendMessageBatchRequest);

Probably we want to do smth like:

return super.sendMessageBatch(sendMessageBatchRequest.toBuilder().entries(batchEntries).build());

Support setting an explicit prefix for the S3 store

Hello,
Can you please extend support for S3 by letting the user specify a prefix for the large objects to be put in S3. This means we don't have to dedicate a bucket explicitly for one service - especially when we're overlaying IAM policies for the access to the object store.

Current implementation is:
ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, s3BucketName);

Would be nice if we could do:
ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration()
.withLargePayloadSupportEnabled(s3, s3BucketName, s3PrefixPattern);

Where pattern may contain:

A complete string

It would be nice if this also supported expansion of some predefined variables, such as sprintf values:
%{y} - four digit year
%{m} - two digit month
%{d} - two digit day of month
%{aws::accountumber} - aws account number (in the case of a cross-account bucket being used)
%{aws::rolename} - IAM role name

Hence we can make prefix a string like:
"LARGE_MESSAGES/%{y}/%{m}/%{d}/%{aws::rolename}/"

I can then give the appropriate role the ability to CreateObject at that location, and any roles that read from it the ability to GetObject and DeleteObject (but not create or list).

No message, only "S3 object read"-log line

Hey! 👋

This library seems perfect for our use case, but I am currently struggling a bit on using it.

After following the setup, we do not receive any large message. Small messages are handled as they should normally, but when an S3-payload is received, all that happens is that a log line with "S3 object read, Bucket name: [...], Object key: [...]" is printed. No message is ever in the returned messages from .getMessages()

Is there an obvious mistake that I might have made?

Unable to find a region via the region provider chain.

This issue appears when the aws credentials are not configured on the environment.
Please indicate in the read.me that the aws credentials has to be set first to avoid this issue.

Invalid ReceiptHandle on ChangeMessageVisibilityRequest

Just noticed that I can't successfully call 'changeMessageVisibility' on the AmazonSQSExtendedClient class.

This is because the 'AmazonSQSExtendedClient' does not override the 'changeMessageVisibility' method of the 'AmazonSQSExtendedClientBase' and work it's magic on the ReceiptHandle to strip the "-..s3Key..-" prefix (which does happen for the 'deleteMessage' method).

Python version

A Python version of the of this boto3 extension should be provided. Thank you.

Supporting sqs temporary queues

Hi,
I created AmazonSQSRequester based on this guide and I want to wrap the generated AmazonSqs with AmazonSQSExtendedClient.
The issue is that AmazonSQSExtendedClient doesn't contains the function sendMessageAndGetResponse.
What is the alternative?

awslabs / amazon-sqs-java-extended-client-lib Goto Github PK