Comments (42)
Do we have ETA for adding support of syncing between Local and Azure Files?
from azure-storage-azcopy.
We will do this for out 10.9 release. @adreed-msft to make it happen.
from azure-storage-azcopy.
Hi @oscarwest, I apologize for the inconvenience.
We had to do some major refactoring, so this feature is not implemented yet. But it will much easier to implement once we get to it.
from azure-storage-azcopy.
@zezha-msft nicest guy ever... I hope they pay you well
from azure-storage-azcopy.
We run some stuff in AKS which mounts from Fileshares, in our CD pipe we'd love to have this sync capability so we never get diffs. It's not necessarily time sensitive but would make our life a lot easier :)
from azure-storage-azcopy.
Yes that's right, sure i'll create an issue over there so you guys can sync
from azure-storage-azcopy.
That sounds fine.
from azure-storage-azcopy.
Parity between robocopy and azcopy would make cloud migrations much simpler. +1 for local to file sync.
from azure-storage-azcopy.
@Kapanther Currently we don't support sync between Local and Azure File. We support sync only between Local and Azure Blob. But we plan to implement sync between Local and Azure File eventually.
from azure-storage-azcopy.
No Support for sync between Local and Azure Files.
from azure-storage-azcopy.
@prjain-msft thanks for confirming. Tested against blob storage and it works fine. (a bit slow, but possibly my awful australian internet)
from azure-storage-azcopy.
I've been waiting on this for a couple of months now, looking forward to it being implemented.
from azure-storage-azcopy.
@zezha-msft Sounds good :)
from azure-storage-azcopy.
Is this still planned? @zezha-msft
from azure-storage-azcopy.
It's on our approved backlog of work, but we don't have any planned date to share yet. Is your need for it time sensitive, with a particular deadline? If so please let us know here or via email. Can't promise anything of course, but all information is helpful in planning.
from azure-storage-azcopy.
If we could integrate this with Azure DevOps it would be even greater, right now we're deploying from Octopus Deploy with the binary
from azure-storage-azcopy.
Any suggestions on what such integration, with DevOps, would look like?
from azure-storage-azcopy.
Maybe expanding on something like this?
https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureFileCopyV1/README.md
from azure-storage-azcopy.
So that one is using AzCopy v8, right? And you'd be looking for:
- The same thing, but with v10... AND
- With "sync" supported, from Local to Azure Files?
Is that correct? If so, would you mind popping over to the azure-pipelines-tasks GitHub project, and logging an issue there to request v10 support? You can include a link back to this issue, with a comment that its Local to Files sync that you most want.
from azure-storage-azcopy.
:-)
from azure-storage-azcopy.
We also need this feature. Is there a workaround to have the same behavior with the azcopy copy command?
from azure-storage-azcopy.
@Symbianx The copy command already works from local disk to Azure Files. Is that what you were asking about, or are you looking for something else?
from azure-storage-azcopy.
We also need this feature. Is there a workaround to have the same behavior with the azcopy copy command?
A temporary solution I used is mount Azure File to a local machine, and use robotcopy to sync(To be exact, it's mirroring).
Robocopy S:\FolderPath\ T:\FolderPath\ /MIR /FFT /Z /XA:H /W:5 /MT:8
from azure-storage-azcopy.
@JohnRusk From what I understood, the azcopy copy
doesn't remove the files that no longer exist (like rsync --delete
).
Our use case is pushing application backups to the Azure File Share from a container running in kubernetes, we also need to remove the files that don't exist locally anymore (hence the need to sync).
In one of our environments we are mounting the file share inside the container and we rsync
the directories, however this forces us to run the container in privileged mode which is not ideal but works.
On the other environment the only way to access the internet is through an http_proxy, but unfortunately we can't mount the file system through the proxy.
from azure-storage-azcopy.
@Symbianx. Yes, it does sound like local->Files sync would be ideal for you there. Sorry its not ready yet!
Maybe you could to a workaround where the copying is done by AzCopy but the deletion is done differently. How are you authenticating? With a SAS url/token? If so, it's probably possible (with a little work) to call the rest API directly from script, to list the contents of the file share. Then, still in script, you could compare the local file listing with the one retrieved via the rest API, and then delete the ones that need to be deleted one by one (probably again with the REST API).
In theory you could do the same with other authentication types, but they are not as easy to script as SAS.
That's the best workaround I can think of right now. (And I'm not sure its terribly good... but its the best I can think of at the moment).
Note that you can't compare last modified dates with copy. You either re-copy everything, or else sett overwrite=false and don't copy anything that already exists (last modified dates are not checked, only existence is checked).
from azure-storage-azcopy.
@JohnRusk Thanks for the suggestion, we are using SAS.
For now, we ended up cleaning the entire directory with azcopy remove --recursive
and uploading all of the files over again.
This is not ideal but since the files are relatively small I think this is a nice compromise until sync supports Local->Files.
Do you think there will be a downside that we may not be aware of?
from azure-storage-azcopy.
Just tossing in a vote of support for this functionality!
from azure-storage-azcopy.
Since Azure Files Fileshare with On premises AD ACL support has gone GA, I'm guessing a lot more will start missing this feature :) Big fileshares need the incremental sync otherwise we need a few days of down time for the share, which is never going to happen.
from azure-storage-azcopy.
Any progress on prioritisation for this?
from azure-storage-azcopy.
This is a must have functionality. We have to sync GBs of small files of which only a few change every week. Now we have to copy everything every time.
from azure-storage-azcopy.
from azure-storage-azcopy.
Hey Guys, its can help me alot.Is there some update about azcopy sync with Azure FileShare ?
from azure-storage-azcopy.
cc @amishra-dev
from azure-storage-azcopy.
Hi Guys, do we have any new status here?
from azure-storage-azcopy.
Amishra-dev;
Any chance of getting a hight priority assigned to this? This feature has been in request since 2018. Getting bi-direction sync is a featuer much-needed in this feature set.
from azure-storage-azcopy.
Hi @wsandmanw, we've already committed to a scope for 10.8, so 10.9 is where we are adding the high priority items such as this feature.
from azure-storage-azcopy.
Thanks for the update. Any update on when 10.9 will be released ?
from azure-storage-azcopy.
+1
from azure-storage-azcopy.
@zezha-msft any word?
from azure-storage-azcopy.
Thanks for following up, we shuffled the numbers a bit and the original 10.9 is now 10.10. We are targeting to land it in early Q2.
from azure-storage-azcopy.
ok...thanks for the update...
from azure-storage-azcopy.
Hey @Kapanther,
We've addressed this issue in the recent release of AzCopy 10.10.0. Please use the same and do reach out to us for more information.
from azure-storage-azcopy.
Related Issues (20)
- azcopy jobs list command is broken HOT 9
- AzCopy sync with hash-meta-dir hides original files HOT 6
- azcopy download shuffles lines on windows but works on linux HOT 5
- AzCopy fails to write xattr HOT 9
- Azcopy blob download of case sensitive files to local case sensitive fs fails on mac os. HOT 5
- Azcopy fails on copy operation from onprem server to azure blob storage for some files HOT 2
- Which permissions are required by `--overwrite=false`? HOT 4
- Documentation does not indicate the maximum file size handled by Azcopyv10 HOT 1
- Error while copying data from ADLS to ADLS in azure HOT 5
- Login timeout HOT 4
- RESPONSE Status: 500 Operation could not be completed within the specified time HOT 6
- Panic error in using user-managed identity HOT 1
- AzCopy Azure government - Copy finish sucessfully but some operations return 401 Server failed to authenticate the request HOT 3
- AZCOPY_AUTO_LOGIN_TYPE="MSI" fails due to host port number interfering with trusted domain suffix detection HOT 1
- AZCOPY_AUTO_LOGIN_TYPE="AZCLI" fails with "Tenant shouldn't be specified for managed identity account" HOT 4
- Is it possible to skip scanning of the excluded folders? HOT 1
- How to disable auto update HOT 1
- AZCopy finishes successfully, although an AuthorizationFailure and a Cancelled job status were reported. HOT 2
- AADSTS90055: The server has terminated the request due to excessive request rate. Please wait a few seconds and try again HOT 1
- Missing from Debian 12 apt packages HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-storage-azcopy.