Git Product home page Git Product logo

blob-copy-test's Introduction

Blob copy test

Warning

This is a test project to identify possible problems with cross-region data copying, and to find solutions/workarounds. This is not secure piece of software and running it exposes sensitive information deliberately for troubleshooting purposes.

Viewer discretion is advised.

The solution will deploy two, peered virtual networks in different locations (westeurope and swedencentral) both containing a blob storage and a file share storage so 4 storage accounts in total. None of the storages allow public internet access, but use private endpoints instead. A very simple app service with a Swagger UI is provided to test copying blobs from different accounts to different file shares.

Prerequisites

Tested in Windows with Azure CLI version 2.46.0.

Scripts to install/deploy

PowerShell scripts provided to install/deploy the project:

Script Description
DeployToAzure.ps1 Full install/deployment. Runs all the scripts below in the specified order.
New-BicepDeployment.ps1 Creates a resource group and provisions Azure resources specified in the main Bicep template (/bicep/main.bicep).
New-Blobs.ps1 Uploads the test blobs (to copy later) to storage accounts.
New-CodeDeployment.ps1 Deploys the app code (with Swagger) to the app service.

All the scripts are (more or less) idempotent. Note that in most cases the scripts will not stop executing when an error occurs. The "sub" scripts can be run independently to save time if any of the phases should fail.

Deploying the solution

  1. Login with Azure CLI:

    az login

    Verify that you have the right subscription selected with command az account show. Change the subscription with command az account set --subscription <subscription ID> if necessary.

  2. Run the main script:

    .\DeployToAzure.ps1 -ResourceNameMeronym <two alphanumeric, lowercase characters>

    The resource name meronym is required for unique resource names should this solution ever be deployed by more than one person.

    Note

    Errors can happen and if they do, rather than starting all over, you can continue from a point closer to the failure point by running the "sub" scripts separately:

    .\New-BicepDeployment.ps1 -ResourceNameMeronym <two alphanumeric, lowercase characters>
    .\New-Blobs.ps1 -ResourceNameMeronym <two alphanumeric, lowercase characters>
    .\New-CodeDeployment.ps1 -ResourceNameMeronym <two alphanumeric, lowercase characters>

    If you deleted a previous deployment, due to the soft delete enabled for the Key Vault, you may need to recover it with command:

    az keyvault recover `
        --location westeurope `
        --name kv-copytest<meronym>-dev `
        --resource-group rg-copytest<meronym>-dev
  3. If all goes well, head to the Azure portal and find the app service and launch it in the browser

  4. Copy some blobs using the Swagger UI (or a tool that can do POST calls of your choice)!

Observed copy results

App service location Source blob location Destination file share location Copy result
westeurope westeurope westeurope OK
westeurope westeurope swedencentral OK
westeurope swedencentral westeurope FAILS with error CannotVerifyCopySource (*
westeurope swedencentral swedencentral OK

Note *) Workaround

Add IP ranges of Azure storage accounts in westeurope to the firewall rules of the blob storage account in swedencentral ("stctb<meronym>devswedencentral") to allow access.

You can find the ranges at Azure IP Ranges site. Enter "Storage.WestEurope" to the search box.

blob-copy-test's People

Contributors

tompaana avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.