When azcopying blobs between 2 storage containers, blob metadata key values are capitalized. about azure-storage-azcopy HOT 40 CLOSED

Copying the data is part of our disaster recovery plan, so this is actually a much bigger issue then I think it is getting attention for.

Since the .net storage client uses a dictionary to index the metadata it is by nature case sensitive. So this tool changing the casing of the keys, that essentially is breaking the supported apis. I will file a support ticket in azure to escalate this issue from the support side.

from azure-storage-azcopy.

Thank you for your response @JohnRusk . I am looking for a better tool for copying blob to blob.
I have encountered problem with the metadata where case is changing. I tried AzCopy V8 but problem with that is, it does not allow me to exclude folders/files. That is the reason I was hoping a fix to V10 as soon as possible. I understand that V10 is dependent on the Go SDK internally, but the metadata changing case is really a big problem for me.

from azure-storage-azcopy.

I had the exact same issue on the following metadata:

• encryptiondata
• unencrypted_content_length

After reading this thread, I understand the issue resides in the use of AzCopy. So the fix for me was the download of the latest version of Microsoft Azure Storage Explorer that does not use AzCopy for blobs transfer: V1.10.1.
Now, I can transfer files or directories without any problem. However, blob transfer is really slow...

I hope this could help someone, until this ticket's resolution.

from azure-storage-azcopy.

i understand the frustration, I'll follow up on this issue offline.

@zezha-msft you remove support for Shared Key, you remove support for Tables, you introduce this issue on tool azcopy, I think the best for Microsoft is you to get fired!!!

from azure-storage-azcopy.

Hi there,

When we tried to copy Event Hub blob storage checkpoints from storage to another one.
We detect that azcopy does not keep the case of the custom HTTP header name which creates a bug when we try to use the copied version in C# using the Event Hub library version 3.0.0.0 on .NET CORE.

We had this error:

Microsoft.WindowsAzure.Storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. (Most recent call last)
at Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__33.MoveNext() in Microsoft.Azure.EventHubs.Processor.AzureStorageCheckpointLeaseManager+<AcquireLeaseCoreAsync>d__33 line 1133
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in System.Runtime.ExceptionServices.ExceptionDispatchInfo line 17
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task) in System.Runtime.CompilerServices.TaskAwaiter line 39
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) in System.Runtime.CompilerServices.TaskAwaiter line 46
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult() in System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter line 11
at Microsoft.Azure.EventHubs.Processor.PartitionManager+<>c__DisplayClass12_2+<<RunLoopAsync>b__2>d.MoveNext() in Microsoft.Azure.EventHubs.Processor.PartitionManager+<>c__DisplayClass12_2+<<RunLoopAsync>b__2>d line 585

The cause is due to lost of the case of the HTTP header OWNINGHOST
Original:
Copied:

The HTTP header OWNINGHOST seems to be important on the server-side of the Event Hub and the case needs to be kept. https://github.com/Azure/azure-sdk-for-net/blob/809f48630e06b7672b4f3475f814cd46bfd97b33/sdk/eventhub/Microsoft.Azure.EventHubs.Processor/src/AzureStorageCheckpointLeaseManager.cs#L18

We lost a week to found that, so please keep the case during the copy to avoid any side effects.

Thanks,
Johnny

from azure-storage-azcopy.

looks like golang/go#37834 has just got the fix merged? Any good news for an azcopy fix and even Azure storage explorer? Many thanks!

from azure-storage-azcopy.

@antonio-gabriele I get mad at many of the technical decisions, but you do not need to take it out on the person who is communicating with us. There are many people on the team , @zezha-msft is just the person who has to deliver the bad news to us.

from azure-storage-azcopy.

I've opened a fresh issue with the Go team here: golang/go#37834

from azure-storage-azcopy.

@philMarius there's no short-term fix unfortunately. The long-term fix is on the service side, they will need to avoid putting metadata keys as HTTP header names, which are not supposed to be case sensitive.

from azure-storage-azcopy.

i understand the frustration, I'll follow up on this issue offline.

from azure-storage-azcopy.

Hi @sharkymcdongles, thanks for reaching out!

I reproduced this problem and have identified that the Blob SDK is causing this behavior.

In general though, we advise not to rely on the case of metadata keys, since they are implemented as HTTP headers, which are supposed to be case-insensitive. However, the Storage Service is case-preserving, for now.

I'll get back to you shortly. Thanks for bringing this issue to our attention!

from azure-storage-azcopy.

I had the same issue. I had copied over 20 million blob from account to account only to realize all the metadata is corrupted. The Blob SDK for go should work like any other lang. We use the Blob java SDK and the case isn't converted.

from azure-storage-azcopy.

Hi @cy33hc, I apologize for the inconvenience. Unfortunately the HTTP client in Golang canonicalizes the headers when they are read off the wire, so we are having a really hard time fixing this issue.

I've enlisted the help of some service folks to see if there are any way to work around this, since HTTP2, which forces headers to be lowercase, will eventually create the same problem. I'll get back to you when I have more information.

from azure-storage-azcopy.

Is there an eta of when this will be fixed? Also how to you copy from blob to blob with account key as the mitigation?

from azure-storage-azcopy.

@cy33hc I haven't quite understood what you mean about "copy with account key as mitigation". From your original comments when logging this issue it sounds like you have a way to preserve the headers in some situations, but I haven't quite understood what that situation is.

from azure-storage-azcopy.

@zezha-msft is there an update on this issue?

from azure-storage-azcopy.

Hi @erichexter, thanks for following up on this issue.

Unfortunately there's no easy solution for us right now. We'd have to make a change on the service end to pass the metadata differently.

from azure-storage-azcopy.

Hi @erichexter, I assure you this issue is definitely on our radar. Solving this problem is not trivial though.

Unfortunately, the service (the source of truth, not the C# SDK) implemented metadata using HTTP headers. This means that by definition, the metadata keys are case-insensitive. However, the service has been case-preserving thus far, e.g. the same casing is preserved, and this resulted in users relying on the casing (some tools were not built correctly as well).

We are trying to figure out a way to solve this problem at its source. I'll update this thread as soon as there's more info.

from azure-storage-azcopy.

I still don't get why the service cannot simply be case preserving. But if it cannot do that implementing something simple and reliable like all lowercase would definitely be easier to cope with than this strange windows-like capitalisation. What more, there used to be a case preserving route in azcopy when using access key authentication but it has been removed leaving only the SAS route which suffers this bug. If it was possible with keys before why is it not possible with SAS? We currently need to use an older azcopy version to continue, so this is actually a regression.

from azure-storage-azcopy.

@fpa-mls What's the version number of the one where this worked for you?

BTW, it's not a service issue, unfortunately. In this service-to-service copying scenario, the data flows directly from source to destination but the service API requires that the headers flow through the client application (in this case, AzCopy). As you probably know, AzCopy v10 is written in Go. The designers of Go have taken an opinionated stance on this issue, in which they insist that its is correct for their SDK to change the case of the headers as they arrive. Technically, according to HTTP standards etc, they are correct, but it has undesirable side effects in cases like this. The reason it's so difficult for us to work around is that it's at a very low level in the Go HTTP stack, and the behavior is not configurable.

from azure-storage-azcopy.

from azure-storage-azcopy.

That is correct, this applies to custom metadata. I believe the only difference is that, for standard ones like ContentMD5, folks normally access those through an SDK that takes care of case insensitivity for them. i.e. No-one relies on case preservation at service level for those, because they access them through strongly-typed properties in their SDK.

I believe that AzCopy 8.1 also supports access key auth (at least the docs say it does).

from azure-storage-azcopy.

Is there any plan on fixing this in upcoming release?

from azure-storage-azcopy.

We don't have a date for when it might be fixed, due to the difficulty of fixing it within the way the Go SDK works

from azure-storage-azcopy.

Is there an update on this? I just hit the same issue and it costed me hours to identify the source problem. I have not built a workaround, yet, just wanted to check what the state of this issue is.

from azure-storage-azcopy.

Sorry, no change as yet. (It's a hard problem, because its in the underlying framework, not in AzCopy itself).

from azure-storage-azcopy.

Yes, we need a solution for this. It is quite a bad limitation.

from azure-storage-azcopy.

We're aware of the issue, and looking for solutions Johnny. Unfortunately, the behavior is hard-coded into the runtime library of Go, the language that AzCopy v10 is written in, which is why its so difficult for us to find a solution. I don't have any other updates at this time, I'm sorry.

from azure-storage-azcopy.

Thanks for your response! Good luck guys!

from azure-storage-azcopy.

I had the same issue. I had copied over 20 million blob from account to account only to realize all the metadata is corrupted. The Blob SDK for go should work like any other lang. We use the Blob java SDK and the case isn't converted.

@JohnRusk @sharkymcdongles I was asking the question to the submitter of this issue. Under the section for mitigation, he/she mentions using account key as a mitigation. I don't know how that was achieved too.

from azure-storage-azcopy.

I don't know how using account key would help either. My only guess, about what that may have possibly meant, is that they went back to an older version of AzCopy (since versions 8 and earlier work differently. Unlike v10, they don't suffer from the header capitalization issue and they do support account key authentication).

from azure-storage-azcopy.

Notes:

1. Looks like there might be some promising options recently raised in the GoLang thread (see link in comment above).
2. If those GoLang changes allow us to fix this issue, it's been suggested that we should retain AzCopy's current behavior as the default, and only preserve metadata case if a command line or environment variable flag is set. That approach makes sense in terms of a future-facing view because HTTP 2 won't allow case preservation, and so the internet is moving in the direction of not preserving case. But meantime, users who are aversely affected by this issue, can set the flag to preserve case and unblock their HTTP 1 data migrations.

from azure-storage-azcopy.

you can use az cli
az storage blob metadata update
as a workaround

This preserves case sensitivity

from azure-storage-azcopy.

Hi there, just ran into this issue myself and it took me all morning to find. What's most odd is that storage explorer was reporting the correct lowercase metadata but when I dived into the Python SDK I found that the first letter for all values were capitalised.

Has this been fixed yet?

from azure-storage-azcopy.

Unfortunately not, this is a limitation of the httpclient in Golang.

from azure-storage-azcopy.

So no fix will be issued for this at all? This is making using azcopy for a lot of our processes less viable

from azure-storage-azcopy.

Any update on this? It looks like we maybe made progress 3 months ago.

from azure-storage-azcopy.

Does anyone know where to get a copy of azcopy 8.1.0 or azcopy 7.3.0-netcore. I cannot find a download or package that works.

from azure-storage-azcopy.

i understand the frustration, I'll follow up on this issue offline.

@zezha-msft you remove support for Shared Key, you remove support for Tables, you introduce this issue on tool azcopy, I think the best for Microsoft is you to get fired!!!

To be honest the people who should be fired are rhe people who decide to use Azure as a cloud provider because it is very obvious Azure has no idea what they are doing and anyone that chooses to use Azure by extension knows even less. Azure should pay people to use it because it is garbage. We moved every single workload to GCP now and have had 0 issues ever since. Azure is a barren wasteland of garbage and bad decisions. The only worthwhile product they offer is their Active Directory setup.

from azure-storage-azcopy.

Key capitalization is inherent by Go SDK and nothing much in control of AzCopy. With latest SDK integration this shall not hit. Closing this for now. Feel free to reopen or create a new report if this is still hit.

from azure-storage-azcopy.