[feature request] s3 multipart resumable uploads about amplify-js HOT 19 CLOSED

I was having trouble uploading a file that was larger than 5MB and realized that Amplify is not exposing the ETag header in the CORS configuration when it creates the S3 bucket. (Explained here: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/cors.html)

I have not thoroughly tested this solution but I was able to upload a 7MB file and the network traffic indicated a successful multipart upload. Previously, a 7MB file upload always failed. If you need to upload large files in your application, simply:

1. Find your S3 bucket (created by Amplify)
2. Go To Permissions -> CORS Configuration
3. Add <ExposeHeader>ETag</ExposeHeader>
4. Make sure that ETag is included in "AllowedHeader" (a wildcard * by default)

After making this backend fix, the client side code appears to handle a multi-part file upload correctly.

-- Disclaimer: I'm not sure whether Amplify will overwrite this file, or if altering it somehow messes with Amplify's CloudFormation process --

My current S3 CORS Configuration

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <ID>S3CORSRuleId1</ID>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

To the Amplify team, I would do a PR but I don't believe that the initial S3 config is in the client-side library (or another open-source repo). Please let me know if I'm incorrect.

from amplify-js.

@mlabieniec @nidsharm

We are absolutely looking for this functionality. What's the estimated timeline for the 0.4 release?

Thank you for all of your hard work on Amplify!

from amplify-js.

I'd like to add a plus one to multipart uploads with features of

• resumable
• pausable
• subscribe to upload progress
• use S3 transfer accelerattion

Thank you for all the work on this 👍

from amplify-js.

@mlabieniec
Has there been any progress on this issue?

I am also wondering how you would cancel a put request
Thank you for all the work on this 👍

from amplify-js.

Not stale+this bot is annoying

from amplify-js.

Hi, any idea on when this might be done?
thanks

from amplify-js.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from amplify-js.

This is a very old issue. We use evaporatejs by now and I don't have time to review another solution.
In any case, this issue is about recovering page refreshes and other types or interruptions during upload.

from amplify-js.

@clarle we have a large number of feature requests and are working on grooming all of them. I'm working on re-organizing the milestones etc. as we are now using a new tracking tool. Will keep this issue updated as we move ahead, this is definitely in the backlog.

from amplify-js.

With #8935 merged, this is now supported. You may check out https://docs.amplify.aws/lib/storage/upload/q/platform/js/#pause-and-resume-uploads on how to use the feature!

from amplify-js.

@amitport I am seeing a series of PRs that were merged into this e.g feat(amplify-category-storage): add s3 multipart upload to cfn #579. What else were you looking for as this seems to be the original FR is complete. Please let us know

from amplify-js.

Is this available for React Native projects now?

from amplify-js.

@amitport it would be great if we could use evaporatejs as well but looking at the github documentation, the configuration they require needs to have a signed URL and an "aws key" - both of which we do not know how to generate and obtain in our Amplify angular code. Any pointers as to how to integrate evaporatejs with Amplify? Thanks!

from amplify-js.

@dorontal, I didn't use Amplify.

wrapping evaporatejs in Angular is the easy part, getting "aws key" is a bit more tricky depending on your exact use-case (you don't need Amplify for that). I suggest that you start by making raw evaporatejs work and only then add Angular integration.
(in any case, any support question should go to evaporatejs or aws-sdk, not here)

from amplify-js.

@amitport apologies: I misundestood 'aws_key' to mean some encryption key... no, it means the S3 key, i.e. the "filename" that's within the bucket in S3, not encryption key - that's why I had asked the wrong question above. Writing this here just to clarify, thanks. EvaporateJS indeed seems like a good library to use for now until Amplify gets pause/resume built-in.

from amplify-js.

3. ETag

Thanks @ajhool - I needed this to get by the error: "AWSS3ProviderManagedUpload - error happened while finishing the upload. Cancelling the multipart upload Error: Request failed with status code 400"

After around 5 MB, it seems like Amplify uses multipart uploads which fail.

from amplify-js.

@beaur asked this before but got no response. So, I am asking it again.
Is it available for the React Native projects?
The biggest issue with a Storage.put in React Native is that I need to read the whole file into memory.
The documentation says the same https://docs.amplify.aws/lib/storage/upload/q/platform/js#react-native-uploads.
But this approach doesn't work for big files, the application just crashes.
Looks like someone had the same issue and created this plugin for Storage https://github.com/walvekarnikhil/amplify-s3-chunk-upload, but it doesn't work.
The idea is really good to read files chunk by chunk, Evaporate also gives this functionality.
So, using the adapted Evaporate version for my needs for now, but it looks more like a hack.

from amplify-js.

Thanks! closing this.

from amplify-js.

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.

from amplify-js.