azcopy sync fails with "AuthenticationErrorDetail: se is mandatory", however the se is definitely included about azure-storage-azcopy HOT 41 CLOSED

After more testing I eventually got SAS tokens from AZ CLI to work.

EXPIRE=$(date -u -d "1 day" '+%Y-%m-%dT%H:%M:%SZ')
START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ')
ACCOUNT="some_account"
KEY="some_key"
CONTAINER="some_container"
BLOB="test.jpg"
SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --start "$START" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/\"//g')

azcopy copy "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" "$HOME/test.jpg"

from azure-storage-azcopy.

Part of the bug is also the documentation of the az tool. It specifies:

    --expiry              : Specifies the UTC datetime (Y-m-d'T'H:M'Z') at which the SAS becomes
                            invalid. Do not use if a stored access policy is referenced with --id
                            that specifies this value.

But that format is not that strict. If I use Y-m-d'T'H:M:S'Z' instead (%Y-%m-%dT%H:%M:%SZ in strftime) it works correctly.

from azure-storage-azcopy.

The Azure Node.js SDK also generates the SAS token with time format yyyy-MM-ddTHH:mm:ss.fffffffZ. And azcopy copy doesn't work. However, azcopy make works well. They are using different method to parse the time?

from azure-storage-azcopy.

As fate would have it, I just figured it out with help from another github post on the issue (sorry I can't find it now to credit the author). It was as simple as including a literal in the date format for '00Z':

NOW=`date +"%Y-%m-%dT%H:%M:00Z"`
EXPIRY=`date -d "$NOW + 10 years" +"%Y-%m-%dT%H:%M:00Z"`
SASa=`az storage container generate-sas --name $srcCon --start $NOW --expiry $EXPIRY --permissions rl --account-name $srcAcc --account-key $srcKey
--output tsv`

I do not know that this is exactly what will work for others but after many hours of hair-pulling it's worked for me.

@zezha-msft thanks for the link just now. Doesn't seem to load, but thanks!

from azure-storage-azcopy.

I'm having a similar issue. The REST API allows SAS tokens to be generated that work in other tools (e.g. Storage Explorer), but result in "Authentication Failed" when using AzCopy.

To replicate, generate some SAS tokens via the "Try It" option here: https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listaccountsas

This caught me out as I'm generating a SAS token from an ARM template using the listAccountSas function, and passing it into a container as an environment variable which is later used by AzCopy.

Here's the payload I'm using to generate a SAS token:

{
        "signedServices": "b",
        "signedPermission": "w",
        "signedProtocol": "https",
        "signedStart": "2019-01-01T00:00Z",
        "signedExpiry": "2029-01-01T00:00Z",
        "signedResourceTypes": "o",
        "keyToSign": "key1"
}

And here's an example of how I am trying to use it (real resources, but now nuked):

azcopy cp "./test.xml" "https://k5h7iywh3tnvy.blob.core.windows.net/reports/test.xml?sv=2015-04-05&ss=b&srt=o&sp=w&st=2019-01-01T00%3A00%3A00.0000000Z&se=2029-01-01T00%3A00%3A00.0000000Z&spr=https&sig=%2BadXJbjjuULC9g8ryNzoQKccX0c6h3gVgAr7ESd8Bu0%3D"

Things I've noticed:

• The SAS token DOES work when you manually make the PUT using a tool like Postman, so the SAS token is valid.
• In Postman, you can decode all the URL entries (%3A is a colon in the time, for example) without it affecting the signature. The SAS token still works.
• You CANNOT drop the case of the T or Z in the date format. In the AzCopy failure output, timestamps are reported with lowercase T and Z, but it's not clear whether that's just a facet of the console output rather than an actual representation of the PUT operation.
• The st and se values in the SAS token have the values: "2019-01-01T11:11:11.0000000Z". The sub-second precision is important: if you try to edit/remove the second component (even though they are all zeroes), then you invalidate the signature causing "AuthenticationFailed - Signature fields not well formed".

So, while the SAS token APIs (accessible via REST or AZ CLI) are quite happy to generate tokens with different date formats, I suspect that AzCopy isn't handling them verbatim and is breaking the signature as a result.

from azure-storage-azcopy.

I spent way more time digging into this than I expected last night. I was able to produce a working local version of azcopy by making changes purely to Blob Storage Go SDK. It looks like azcopy has forks of some of the files I was changing, but me making my edits there had seemingly no effect. Unsure if it's dead code, or hidden behind a switch.

I'm going to try to prepare a PR for the Go SDK this week, to help speed things along. It'll be my first change to a Go codebase! 💪

from azure-storage-azcopy.

So, it turns out the tokens generated are not complete!
The SAS token timestamps are wrong and this endpoint expects the following format:

?sv=2017-07-29&ss=b&srt=sco&sp=rwdlacup&se=2039-01-01T08:00:00Z&st=2018-01-01T08:00:00Z&spr=https&sig=<redacted>

Notice:
&se=2028-01-01
&se=2039-01-01T08:00:00Z

Unfortunately the current version of this tool does not know how to parse the URL correctly and figure out what's wrong.

from azure-storage-azcopy.

Apologies for being a little slow on the uptake here, but exactly what are the work-arounds? I've tried modifying the expiry time stamp in every way I can imagine with no luck. The sas I generate works fine in az commands, so I don't think that's the problem. Is there a know good format for expiry to use in, say, az storage container generate-sas', that would provide a valid (for azCopy) sig? Thanks!

from azure-storage-azcopy.

@mifydnu glad to hear you figured it out! We are really sorry for the inconvenience. We'll try to schedule this work item as soon as possible.

from azure-storage-azcopy.

This is also a problem when using SAS tokens with containers and storage policies (no explicit times in the signature). For example, this does not work on OSX, but does work on Windows using v10.3.1 on both.

./azcopy list https://myaccount.blob.core.windows.net/my-container?si=policy&sv=2019-02-02&sr=c&sig=redacted

from azure-storage-azcopy.

I've been banging my head for a while to no avail, happy I stumbled on this GH Issue, I'm having the same problem. Eagerly awaiting the bug fix. Can confirm the workaround of specifying the format as the Linux command date -u -d "1 day" '+%FT%TZ' to the parameter --expiry in az storage container generate-sas works for me too, I'm using ADLS Gen2.

from azure-storage-azcopy.

Has anyone managed to get SAS tokens generated from AZ CLI to work with 'azcopy copy'

I've tried generating tokens with all of the mentioned datetime formats mentioned here as well as the format from the AZ CLI docs and azcopy spits them all back.

Examples tried:

EXPIRE=`date -u -d "1 day" '+%FT%TZ'`

EXPIRE=`date -v+30M '+%Y-%m-%dT%H:%MZ'`

NOW=`date +"%Y-%m-%dT%H:%M:00Z"`
EXPIRE=`date -d "$NOW + 10 years" +"%Y-%m-%dT%H:%M:00Z"`

Using this AZ command to generate the SAS:

az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt

For reference I've compared a valid SAS token generated via the Azure Portal and a token generated from AZ CLI with %3A changed to : and they are more or less identical in formatting so this is very confusing as to what is going on with azcopy.

The only difference I can see is firstly the parameters of the SAS token are in a different order (Shouldn't matter though) e.g portal generated token starts with sv= where as Az CLI token starts with st= and secondly the sig= is always a few characters longer when using portal generated tokens.

NOTE: These are not valid tokens I've changed the values in them so don't worry about redacting
Portal Generated Token:
sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-09-06T07:00:52Z&st=2019-09-05T23:00:52Z&spr=https&sig=Xl4i%2Bsy%2Bg4ab7LstIguIlfXdOnhS6Slkdxe1CH0ptYE%3D

Az CLI Generated Token (With re-arranging):
sv=2018-03-28&ss=bqtf&srt=sco&sp=rwdlacup&se=2021-01-09T21:35:42Z&st=2019-01-07T21:35:42Z&spr=https&sig=GkJC3PiP6NHa/QsNoIYMI/miHbLVdsIOCPinRJePKKg%3D

Az CLI Generated Token (Without re-arranging):
st=2019-01-07T21:35:42Z&se=2021-01-09T21:35:42Z&sp=rwdlacup&spr=https&sv=2018-03-28&ss=bqtf&srt=sco&sig=GkJC3PiP6NHa/QsNoIYMI/miHbLVdsIOCPinRJePKKg%3D

I opened a ticket with the AZ CLI team as well regarding this issue.
Azure/azure-cli#11190

from azure-storage-azcopy.

@lflemingweb I don't think there's any way to get the REST API to produce a working sas. The problem is that even if you pass it a datetime that works with azcopy, the API will return a sas that includes decimals in the seconds field, which fails.

from azure-storage-azcopy.

I am having the same problem with azure-storage-deploy and can't figure out what the correct format is.

I have tried

&se=2028-01-01
&se=2039-01-01T08:00:00Z

Still, there is no hope!

The actual PUT URL is missing the se and st query parameters.

PUT https://XXXX.blob.core.windows.net/$web/webpack-runtime-887b0243a9c4b83c6243.js?sig=-REDACTED-&sp=rwdlacu&spr=https&srt=o&ss=b&sv=2017-07-29&timeout=901

It works locally with AzCopy 10.5.0 just fine.

Ist there any workaround in the meantime?

from azure-storage-azcopy.

I needed to use azcopy in Azure DevOps Pipelines but stuggled a lot to get it to work.

After trying out most suggestions, the only thing that worked for me was what @lmeyerov suggested, however, I don't need to use the --account-key parameter.

For others that find this issue, I did it with:

  - task: Bash@3
    displayName: Install azcopy
    inputs:
      targetType: 'inline'
      script: |
        mkdir $(Agent.ToolsDirectory)/azcopy && cd "$_"
        wget -O azcopy_v10.tar.gz https://aka.ms/downloadazcopy-v10-linux
        tar -xf azcopy_v10.tar.gz --strip-components=1
  - task: AzureCLI@2
    displayName: Download using azcopy
    inputs:
      azureSubscription: my-vmssagents-service-connection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        export STORE_NAME="data"
        export CONTAINER_NAME="data"

        NOW=`date +"%Y-%m-%dT%H:%M:00Z"` \
        EXPIRY=`date -d "$NOW + 1 day" +"%Y-%m-%dT%H:%M:00Z"` \
        && export SAS_TOKEN=$( az storage container generate-sas \
            --account-name $STORE_NAME \
            --name $CONTAINER_NAME \
            --start $NOW \
            --expiry $EXPIRY \
            --permissions acdlrw \
            --output tsv )

        $(Agent.ToolsDirectory)/azcopy/azcopy copy \
            "https://${STORE_NAME}.blob.core.windows.net/${CONTAINER_NAME}/${{ parameters.folder }}/?${SAS_TOKEN}" \
            "." --recursive --include-pattern "*c_*b.nc;left.nc;right.nc" # <-- my specific pattern

from azure-storage-azcopy.

How do I tell azcopy NOT to drop se value from SAS?
The SAS has it, but clearly missing in PUT statement below.
And very same question about st value

Time:2021-01-22T18:14:09.2117938Z, Details:
   AuthenticationErrorDetail: se is mandatory. Cannot be empty
   Code: AuthenticationFailed
   PUT https://-REDACTED-.blob.core.windows.net/-REDACTED-?blockid=nmm1ngvjogqtmzeyzs03yzq4ltu4yzitzdvjzmm4yjqymtbl&comp=block&sig=-REDACTED-&sp=rwdlacup&srt=sco&ss=b&sv=2018-03-28&timeout=901

from azure-storage-azcopy.

Ok, sorted it out:
Oh, I was running it from .cmd file on windows, thus lost all single % characters in SAS...
Need to double them inside .cmd file, obviously.
Totally my bad... :(

az storage account generate-sas
produce url encoded SAS
"st=2021-01-01T00 %3A 00 %3A 00Z&se=2022-01-01T00 %3A 00 %3A 00Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=bla-bla-bla %3D"

and azcopy expects decoded one:
"st=2021-01-01T00 : 00 : 00Z&se=2022-01-01T00 : 00 : 00Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=bla-bla-bla = "

• I put extra spaces around encoded chars above

azcopy shows funny errors to cheer you up and make you think!
Ingenious!

from azure-storage-azcopy.

Also confirmed on Alpine Linux, reproduction case:

Reproduction case with Docker:

docker run -it -v ${PWD}:/tmp hyperized/azcopy sync /tmp/content "https://<redacted>.blob.core.windows.net/<redacted>/content?sv=2017-07-29&ss=b&srt=sco&sp=rwdlacup&se=2028-01-01&st=2018-01-01&spr=https&sig=<redacted>" --recursive=true

Job 70e0f51f-593f-df42-6381-793d75977496 has started

Log file is located at: /root/.azcopy/70e0f51f-593f-df42-6381-793d75977496.log
0 File Scanned at Source, 0 Files Scanned at Destination
error performing the sync between source and destination. Failed with error error starting the sync between source /tmp/content and destination https://<redacted>.blob.core.windows.net/<redacted>/content. Failed with error cannot list blobs for download. Failed with error -> github.com/Azure/azure-storage-blob-go/azblob.NewResponseError, /go/src/github.com/Azure/azure-storage-blob-go/azblob/zz_generated_response_error.go:28
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:dbc15991-e01e-011b-4a9a-8c53ba000000
Time:2018-12-05T13:01:12.3625934Z, Details:
   AuthenticationErrorDetail: se is mandatory. Cannot be empty
   Code: AuthenticationFailed
   GET https://<redacted>.blob.core.windows.net/<redacted>?comp=list&prefix=content%2F&restype=container&sig=REDACTED&sp=rwdlacup&spr=https&srt=sco&ss=b&sv=2017-07-29&timeout=901
   User-Agent: [AzCopy/10.0.4-Preview Azure-Storage/0.3 (go1.11.2; linux)]
   X-Ms-Client-Request-Id: [b99617b9-fd78-4eff-5869-77f14121fd99]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [407]
   Content-Type: [application/xml]
   Date: [Wed, 05 Dec 2018 13:01:11 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [dbc15991-e01e-011b-4a9a-8c53ba000000]

from azure-storage-azcopy.

Good catch on the cause there, however, that's not totally satisfactory because when I did my initial test, I created the SAS token directly using the az tool like this:

az storage container generate-sas \
    --account-name production8858 \
    --account-key 'redacted' \
    --name wad-iis-logfiles \
    --permissions rl \
    --expiry 2030-01-01

However, I grant you I think "az" is a separate project from this one, so if it's generating invalid URLs, maybe the issue should be against that project, although improving azcopy's URL parsing would be nice as well, I think we need the devs to chime in.

from azure-storage-azcopy.

Hi @angrygreenfrogs @hyperized, I apologize for the delayed response.

I have created an issue on the CLI repo and will follow up with them to resolve this problem.

from azure-storage-azcopy.

@zezha-msft can we perhaps reopen this as the error given is not exactly helpful and actually incorrect? Can you maybe verify the individual argument fields in the tool before sending it out to the endpoint?

from azure-storage-azcopy.

@hyperized sure! Let me take a closer look and get back to you.

from azure-storage-azcopy.

Hi @angrygreenfrogs @hyperized, thanks for your patience.

After some investigation and help from the CLI Team, I've confirmed that the problem is actually in our Go Blob SDK.

Despite what the doc says, an expiration time like "2030-01-01" is actually accepted by the Storage service. It is the Go SDK not parsing the time properly. I've logged an item in our backlog to fix this issue. Thanks!

from azure-storage-azcopy.

@zezha-msft Thank you, that's great to hear and appreciate the response

from azure-storage-azcopy.

Hi @silvester747, thanks for the feedback!

You are indeed right that the Storage service is not very strict in terms of the time format. But I think the CLI's documentation is fine, since a suggested format should be given so that it's easier for users to understand.

The bug is in the Go Blob SDK, while fails to parse the variation of formats.

from azure-storage-azcopy.

@nathankitchen, thanks for reaching out!

Yes, this is a known issue, we'll try to schedule the work item as soon as possible.

In the mean time, please adjust the time formats to work around it. We are sorry for the inconvenience!

from azure-storage-azcopy.

@zezha-msft Are there any updates on this issue?

from azure-storage-azcopy.

@raidlman No updates just yet, sorry. Once release 10.3 is out (soon) we'll take a look at outstanding work and check priorities.

from azure-storage-azcopy.

@raidlman sorry we haven't picked up this item yet, since there are workarounds. We'll try to schedule it as soon as the more prioritized items are done.

from azure-storage-azcopy.

@zezha-msft can you note the workarounds here pls?

from azure-storage-azcopy.

Hi @mifydnu, thanks for reaching out.

As noted earlier in this thread, the documented format works fine, e.g. 2019-10-16T21:50:18Z.

Example SAS: https://zemaintest.blob.core.windows.net/bla?st=2019-10-16T21%3A50%3A18Z&se=2019-10-17T21%3A50%3A18Z&sp=racwdl&sv=2018-03-28&sr=c&sig=REDACTED

from azure-storage-azcopy.

After more testing I eventually got SAS tokens from AZ CLI to work.
EXPIRE=$(date -u -d "1 day" '+%Y-%m-%dT%H:%M:%SZ')
START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ')
ACCOUNT="some_account"
KEY="some_key"
CONTAINER="some_container"
BLOB="test.jpg"
SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --start "$START" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/"//g')

azcopy copy "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" "$HOME/test.jpg"

This eventually worked for me as well. Azcopy seems to need the seconds in start and expiry in order for it to work, however, this is unfortunately not documented anywhere.

from azure-storage-azcopy.

It's been over a year since this bug was found, has there been any progress in fixing it?

from azure-storage-azcopy.

No, sorry. We have been getting a lot of high priority feature requests from a lot of different sources, and have not been able to work on this one yet. I agree that's not ideal. We have not forgotten about it.

from azure-storage-azcopy.

Can you please confirm how to apply the date format workaround when calling the listAccountSas rest api directly (through postman for example), and not from the Azure CLI?

I have tried the format below in my request, which returns a token that works with azcopy make, but not azcopy copy (same issue as above):
"signedExpiry":"2020-02-22T00:00:00Z"

from azure-storage-azcopy.

I don't know the answer, sorrry. @adreed-msft Any ideas for lflemmingweb's question above?

from azure-storage-azcopy.

@AtOMiCNebula, I had an email about this same topic from @adreed-msft this morning? Are you already in touch with her about this?

from azure-storage-azcopy.

I hadn't been...sounds like a coincidence! 😇

from azure-storage-azcopy.

This has been affecting the Service Fabric diagnostic support case for a while as mentioned in this bug on ASE which consumes azcopy:
microsoft/AzureStorageExplorer#2457

We use Azure Storage Explorer to download logs from storage accounts via SAS and since azcopy does not decode the SAS key correctly we are having to use an older version of ASE which doesn't appear to use azcopy:
V1.10.1
9/19/2019 (Build 20190919.2)
https://github.com/microsoft/AzureStorageExplorer/releases/tag/V1.10.1

Transfer of 'fabriclogs-bec4a403-0026-4589-... by Alfredo Santamaria Gomez
Alfredo Santamaria Gomez11:17 AM
Transfer of 'fabriclogs-bec4a403-0026-4589-ad37-ac8bfc53fa08/_fcdv2seed_0/System/extension/' to 'C:\Users\alsantam\Documents\sfrp\managedClusters\vmExtension' failed: 0 items transferred, error: failed to perform copy command due to error: cannot start job due to error: cannot list blobs. Failed with error -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) ===== Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:c894b0cb-c01e-0084-19c0-4849e2000000 Time:2020-06-22T18:11:00.0002391Z, Details: AuthenticationErrorDetail: Signature did not match. String to sign used was sflogsfcdeventsv21686 rl bt sco 2020-06-23T02:02:16Z 2015-04-05 Code: AuthenticationFailed GET https://sflogsfcdeventsv21686.blob.core.windows.net/fabriclogs-bec4a403-0026-4589-ad37-ac8bfc53fa08?comp=list&include=metadata&prefix=_fcdv2seed_0%2Fsystem%2Fextension%2F&restype=container&se=2020-06-23t02%3A02%3A16z&sig=-REDACTED-&sp=rl&srt=sco&ss=bt&sv=2015-04-05&timeout=901 User-Agent: [Microsoft Azure Storage Explorer, 1.13.0, win32, AzCopy/10.4.2 Azure-Storage/0.7 (go1.13; Windows_NT)] X-Ms-Client-Request-Id: [8df3d11b-1a30-44c6-5e00-f10c939c1c51] X-Ms-Version: [2019-02-02] -------------------------------------------------------------------------------- RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Content-Length: [491] Content-Type: [application/xml] Date: [Mon, 22 Jun 2020 18:10:59 GMT] Server: [Microsoft-HTTPAPI/2.0] X-Ms-Error-Code: [AuthenticationFailed] X-Ms-Request-Id: [c894b0cb-c01e-0084-19c0-4849e2000000] . (used SAS, discovery not completed)

from azure-storage-azcopy.

+1 to fixing docs . In particular, for CLI users, an end-to-end example showing the happy path. For us, it was a combo of (a) timezone issue and (b) auth mode (needing to use account-key instead of auth-mode login).

export RESOURCE_GROUP="zzz"
export LOCATION="zzz"
export STORE_NAME="zzz"
export CONTAINER_NAME="zzz"

az storage account create \
    --name "$STORE_NAME" \
    --resource-group "$RESOURCE_GROUP" \
    --location "$LOCATION" \
    --sku Standard_ZRS \
    --encryption-services blob

az storage container create \
    --account-name "$STORE_NAME" \
    --name "$CONTAINER_NAME" \
    --auth-mode login

export ACCOUNT_KEY=$( \
    az storage account keys list \
        -g $RESOURCE_GROUP -n $STORE_NAME \
        | jq -r .[0].value )

NOW=`date +"%Y-%m-%dT%H:%M:00Z"` \
EXPIRY=`date -d "$NOW + 1 day" +"%Y-%m-%dT%H:%M:00Z"` \
&& export SAS_TOKEN=$( az storage container generate-sas \
    --account-name $STORE_NAME \
    --name $CONTAINER_NAME \
    --start $NOW \
    --expiry $EXPIRY \
    --permissions acdlrw \
    --account-key $ACCOUNT_KEY \
    --output tsv )

azcopy copy \
    myfile \
    "https://${STORE_NAME}.blob.core.windows.net/${CONTAINER_NAME}/myfile?${SAS_TOKEN}"

from azure-storage-azcopy.

The original time parsing issue is resolved. Please open new issues if you are still experiencing problems with your SAS tokens, but first please double check that you are passing the token in correctly.

from azure-storage-azcopy.