Copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes about localstack HOT 7 CLOSED

Actually it works with AWS S3 when i try with AWS s3; log comes as

{"message":"Copied test.jpg to s3:/test-file-management-storage/service_test_1/test.jpg"}

It worked with localstack after adding encryption to the locastack thanks.

from localstack.

Hello @alwismt and thanks for your report!

We have multiple tests in place to cover this behavior, validated with AWS. It's possible we missed something, but we would need a proper sample to reproduce the issue.

Is it possible that your real AWS bucket has some encryption in place? There are some special case where AWS will allow you to copy in place without change, only if your bucket has encryption configured.

Maybe your bucket is not properly configured in LocalStack, and that is why you're seeing different behavior.

From what we see in AWS, these are the things allowing copy in place:

• different storage class from the source object
• OR different encryption setting from the source object
• OR setting metadata-directive to REPLACE
• OR the destination bucket has encryption configured manually
• OR the object is being restored from an archives storage class

If you don't have any of these conditions, AWS will reject the request. Could you share more about your use case?

Also awslocal s3api put-bucket-metadata --bucket test-file-management-storage --metadata-directive REPLACE command does not exist?
You would need to do awslocal s3api copy-object --bucket test-file-management-storage --key <your-key-name> --copy-source test-file-management-storage/<your-key-name> --metadata-directive REPLACE

Where did you get the command above?

from localstack.

I'm using docker composer and mount localstack.sh. For copy object we use same bucket and same key it works fine with AWS when with localstack throw an error

version: '3.9'
services:
  localstack:
    image: localstack/localstack:1.2.0
    environment:
      - DEBUG=1
    ports:
      - "4566:4566"
    volumes:
      - ./restore/localstack/script/init.sh:/etc/localstack/init/ready.d/localstack.sh
      - ./restore/localstack/test-filestore:/etc/localstack/test-filestore
      

init.sh

#!/bin/bash

awslocal s3api create-bucket --bucket test-file-management-storage;
awslocal s3api put-bucket-versioning --bucket test-file-management-storage --versioning-configuration Status=Enabled

and then use golang S3 aws package , for now other api are working fine s3 versioning working fine but when try to copy and object to same location the above mentioned error happens.

language golang
package github.com/aws/aws-sdk-go/service/s3

copyInput := &s3.CopyObjectInput{
		CopySource: &copySource,
		Bucket:     toS3Bucket,
		Key:        toS3Key,
	}

	// Copy the object
	output, err := svc.CopyObject(copyInput)

I'm little confused how to add --metadata-directive REPLACE this in init.sh cause I cant add it when copy object request send since I'm using S3 package and lot more other copies

from localstack.

I'll look into the versioning, maybe the issue could be there. I'll come back to you soon with more data, just to confirm. Thanks!

from localstack.

FYI: Even without versioning enable this happens. I tried with removing versioning yet same issue happening.

InvalidRequest: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.

also tried with localstack/localstack:latest it also same error

from localstack.

Hello again @alwismt.

I've just checked with AWS, and what you are sharing here does not work against AWS.

I've created a new bucket in AWS, configured bucket versioning, put an object and then try to copy in place and AWS is raising an exception.

This is not a bug in LocalStack, LocalStack is in parity with AWS here and is behaving the same way. You must have a different configuration in AWS for your bucket.

You can try this against real AWS and see that it fails too:

$ aws s3api create-bucket --bucket testcopyinplace0703
{
    "Location": "/testcopyinplace0703"
}
$ aws s3api put-bucket-versioning --bucket testcopyinplace0703 --versioning-configuration Status=Enabled
$ aws s3api get-bucket-versioning --bucket testcopyinplace0703                                       
{
    "Status": "Enabled"
}
# replace README.md by whatever file you have locally
$ aws s3api put-object --bucket testcopyinplace0703 --key testkey --body README.md
{
    "ETag": "\"1248e2d7cb2091daf813eef871b80d8f\"",
    "ServerSideEncryption": "AES256",
    "VersionId": "x8K_o4enDQBhrIuKRzdQz62DHBxo5DEg"
}
$ aws s3api copy-object --bucket testcopyinplace0703 --key testkey --copy-source testcopyinplace0703/testkey

An error occurred (InvalidRequest) when calling the CopyObject operation: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.

As you can see, AWS is raising the exact same error.

It's possible that your AWS bucket has different encryption configured.

To solve your issue, you will need to either set some encryption on your bucket, something like:
awslocal s3api put-bucket-encryption --bucket my-bucket --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'

This will make your LocalStack bucket have different encryption that the default, and will allow copy in place. I don't know how your bucket is configured in AWS.

Or you can set the metadata directive in your golang project.

copyInput := &s3.CopyObjectInput{
		CopySource: &copySource,
		Bucket:     toS3Bucket,
		Key:        toS3Key,
                MetadataDirective: aws.String("REPLACE"),
	}

	// Copy the object
	output, err := svc.CopyObject(copyInput)

Again, this is not an issue in LocalStack. Hope that can help you solve your use-case.

from localstack.

Thanks for confirming it works! I suppose your real bucket in AWS has some kind of encryption configured, which is why it is working there. Thanks!

from localstack.