all: add Azure support about go-cloud HOT 25 CLOSED

should this be implemented as a wrapper for https://github.com/Azure/azure-sdk-for-go ?

from go-cloud.

Yes, this should likely be using the Azure Go SDK, much like the GCP and AWS implementations use their respective SDKs.

from go-cloud.

@zombiezen @joshgav, great.
I'll start to work on this then.

from go-cloud.

Does anybody work on this?
Can I take it?

from go-cloud.

I was able to use the conformance test for Azure and results looks promising.

Results:
PASS
ok github.com/google/go-cloud/blob/azureblob 1.846s
Success: Tests passed.

I am gearing up to send my first pull request after some refactoring. Please look over if you want.

from go-cloud.

@vangent I am still interested in helping with Azure support (if you need the help).

from go-cloud.

Certainly, I will resume work by updating the original PR.

from go-cloud.

@VPashkov You should check over at Azure/azure-sdk-for-go#2296. Folks may already be working on this.

from go-cloud.

@VPashkov we (Azure Go team) aren't able to work on this quite yet but would be happy to support you! Please do open issues in our repo or use Azure/azure-sdk-for-go#2296 for questions, and would be great if you could @-mention me @jhendrixMSFT @marstr @vladbarosan in PRs, we'd love to review :) Thank you!

from go-cloud.

Disclaimer: Although I work for Microsoft, my comments represent my own personal interest in this project.

I can help with the implementation as well. Just out of curiosity, the TerraForm Azure Provider does a great job wrapping the Azure Management SDK for Go, see config.go

https://github.com/terraform-providers/terraform-provider-azurerm/blob/master/azurerm/config.go

For Azure Storage (Blob as an example) requires distinct steps:

1. Authenticate with ServicePrincipal
2. Provision a Resource Group (requires a Resource Group Name or Random Generated)
3. Provision a Storage Account (requires a Storage Account Name or Random Generated)
4. Retrieve the Access Keys to the Storage Account
5. Create a Container (Based on Access Levels)
6. Create / Upload Blob

Should the implementation automatically provision Resource Name, Storage Accounts or will this be supplied by the caller (exists)?

@VPashkov – lets connect so we can coordinate our efforts.

from go-cloud.

Awesome, sounds good!

@vsaroopchand Take a look at the AWS and GCP implementations as well as design.md. Implementations should assume that provisioning has happened. Feel free to send questions to our mailing list.

from go-cloud.

I have a question on Reader.go (https://github.com/google/go-cloud/blob/master/blob/blob.go); there is no operation to return the read bytes? How should the retrieved bytes be returned?

from go-cloud.

Hi @vsaroopchand,

Reader.Read reads into the slice of bytes it is passed, p. There's an explanation of io.Reader and examples here:
https://tour.golang.org/methods/21

HTH!

BTW, PR #311 is providing a conformance test for Blob implementations which you should be able to use to test yours.

from go-cloud.

I committed a proposal for Azure Blobs (Block Blob) in my fork

Here are some observations:

1. This implementation wraps Block Blob; there are other types of blobs, see documentation
2. I added a Settings type “AzureBlobSettings” for the various settings needed to read & write. It is definitely more than gcs and s3 hence the wrapper
3. Caller must supply the autorest.Authorizer using one of the numerous ways to perform authorization. The sample uses a Service Principal for simplicity.
4. Access key can be supplied otherwise it will be fetched from the Storage Account.
5. Moniker ‘bucket’ is dropped for ‘container’ to keep with Azure naming and documentation

6. Integration tests will be forthcoming 
   

Questions:

1. Do you intend to support multiple file writes per one writer instance?
2. Do you intend to support listing all files within a container?

from go-cloud.

A driver.Writer will only operate on the blob that was specified at creation, but it may receive multiple calls to Write as per the io.Writer interface.

Listing all blobs in a bucket is not yet supported. The feature is tracked in #241.

from go-cloud.

RE integration tests, it might be worth waiting for #311, which provides a conformance test for blob.Bucket. Assuming Azure uses HTTP, you should be able to use go-vcr to record/replay the tests similar to what is being done for S3 and GCS (see for example blob/s3blob/s3blob_test.go in the PR).

from go-cloud.

Ok, I can wait on #311 . I am also rethinking about the use of a Service Principal vs. ConnectionString for Authentication. I feel the developer/client should manage AccessKey/ConnectionStrings. Using ConnectionString/SasToken removes the need for the Settings Wrapper and greatly simplifies the bucket initialization logic. I implemented both options for now.

from go-cloud.

Just an FYI, I changed my proposal to use the new storage package . The original implementation which used github.com/Azure/azure-sdk-for-go/storage was moved to sub-folder /azureblob2 for reference and will be deleted at some point.

from go-cloud.

I sent a pull request #427 ; however, having some issues with CLA & the various email addresses. Please advise, thanks.

from go-cloud.

@vangent:

1.5 months ago:

Expect to hear more about next steps from us in the next few weeks.

(#427 (comment))

Any progress on this?

from go-cloud.

We are making progress on this but don't have an ETA for Azure support.

@philippgille, are you wanting to use Go Cloud with Azure?

from go-cloud.

Not currently, but maybe in the future. go-cloud is on my radar for use in multi-cloud deployments, but doesn't seem to fit the bill as long as Azure isn't supported. When looking at multi-cloud strategies of enterprises, they always include the "big three": AWS, Azure and GCP.

from go-cloud.

@philippgille -- agreed. Follow this issue for updates.

from go-cloud.

Hi @vsaroopchand ! At this point we are ready for your contribution, so we'd be delighted if you are ready & willing to refresh your earlier PR and re-send it.

Please let us know if you have any questions or comments!

from go-cloud.

Opened the following issues to simplify ServiceBus PubSub driver

Add Exists test on Queue, Topic & Subscription

Add support for SendBatch on Topic

Add support for Bulk Dispositions on Subscription

from go-cloud.