Git Product home page Git Product logo

Comments (43)

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024 2

@hongooi73 I'm having the same issue with the recent version 3.6.0 using the same script that runs without issue in 3.5.2. I did some testing and found out that the issue happens when the endpoint is in this format "https://storageaccount.**dfs**.core.windows.net/", but when using "https://storageaccount.**blob**.core.windows.net/", it works fine - knowing that our storage account is ADLSGen2.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024 1

https://community.rstudio.com/t/issues-with-azurestor-and-rstudio-workbench/126884

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024 1

The blob endpoint is the older one. The ADLS2 endpoint is a more up-to-date design and generally better, although the blob API does have some features that aren't (yet) in ADLS2. In particular, directory listing is much faster and less error-prone when using ADLS2.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024 1

@MichaelHannaEQT have you been able to resolve this yet?

@hongooi73 We had to revert back to 3.5.2 until we hear back from RStudio. Using the blob.core.windows.net works fine, but we won't switch it to keep it consistent in the organization.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Can you provide some more details? Are you running the same code on both your local machine and the server?

What version of R does Workbench provide? Provide a sessionInfo() output for both of these.

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

Trying to list a container using the Rstudio server in AML

Looks like this is the same error.

az <- create_azure_login(auth_type = "device_code", tenant="microsoft.onmicrosoft.com", graph_host = NULL)
Creating Azure Resource Manager login for tenant 'microsoft.onmicrosoft.com'
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code EBA7ELQPR to authenticate. 
Waiting for device code in browser...
Press Esc/Ctrl + C to abort
Authentication complete.
> az
<Azure Resource Manager client>
<Authentication>
 Azure Active Directory v2.0 token for scope https://management.azure.com/.default openid offline_access
   Tenant: microsoft.onmicrosoft.com
   App ID: 04b07795-8ddb-461a-bbee-02f9e1bf7b46
   Authentication method: device_code
   Token valid from: 2022-01-18 19:57:32 UTC  to: 2022-01-18 20:58:51 UTC
   MD5 hash of inputs: fc73ec846b35dd579c1c23c5d80cdd4d

Authenticating using a SaS sig for reading, listing, etc.

> endpt <- AzureStor::storage_endpoint('https://kea.blob.core.windows.net/variance', sas='https://kea.blob.core.windows.net/variance?sp=racwdl&st=2022-01-18T20:05:48Z&se=2022-03-30T03:05:48Z&spr=https&sv=2020-08-04&sr=c&sig=[deleted]')
> endpt
Azure blob storage endpoint
URL: https://kea.blob.core.windows.net/variance
Access key: <none supplied>
Azure Active Directory token: <none supplied>
Account shared access signature: <hidden>
Storage API version: 2020-10-02

Creating the container endpoint

> cn_pt <- AzureStor::storage_container(endpoint = endpt, name='variance')
> cn_pt
Azure blob container 'variance'
URL: https://kea.blob.core.windows.net/variance
Access key: <none supplied>
Azure Active Directory access token: <none supplied>
Account shared access signature: <hidden>
Storage API version: 2020-10-02

Attempts to use the list_storage_files method

> AzureStor::list_storage_files(cn_pt)
Error in process_storage_response(response, match.arg(http_status_handler),  : 
  Bad Request (HTTP 400). Failed to complete Storage Services operation. Message:
InvalidUri
The requested URI does not represent any resource on the server.
RequestId:0a85b587-901e-000f-09aa-0ca81f000000
Time:2022-01-18T20:29:41.8160689Z.
> AzureStor::list_storage_files(cn_pt, '/variance')
Error in process_storage_response(response, match.arg(http_status_handler),  : 
  Bad Request (HTTP 400). Failed to complete Storage Services operation. Message:
InvalidUri
The requested URI does not represent any resource on the server.
RequestId:dea05999-001e-0032-7ead-0cde04000000
Time:2022-01-18T20:57:25.0930662Z.

Here's an example that appears to authenticate, but returns null info (Could be a problem with the az login timing out)

> endpt <- AzureStor::storage_endpoint('https://kea.blob.core.windows.net/variance', sas='sp=rcl&st=2022-01-18T20:05:48Z&se=2022-03-30T03:05:48Z&spr=https&sv=2020-08-04&sr=c&sig=[deleted]')
> cn_pt <- AzureStor::storage_container(endpoint = endpt, name='')
> AzureStor::list_storage_files(cn_pt, '/variance')
data frame with 0 columns and 0 rows

Here's a solution that works:

AzureStor::list_storage_files(cn_pt, '/')
1                                                 joined_data_v0_2021-11-01...2021-12-02.csv  1584882966 FALSE BlockBlob [...]

Environment

sessionInfo("AzureStor")
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.6 LTS

Matrix products: default
BLAS/LAPACK: /opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin/libmkl_rt.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8      
 [8] LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
character(0)

other attached packages:
[1] AzureStor_3.6.0

loaded via a namespace (and not attached):
 [1] httr_1.4.2          BiocManager_1.30.16 compiler_4.1.2      R6_2.5.1            AzureRMR_2.4.3      graphics_4.1.2      tools_4.1.2         utils_4.1.2        
 [9] curl_4.3.2          rappdirs_0.3.3      grDevices_4.1.2     jose_1.2.0          stats_4.1.2         datasets_4.1.2      xml2_1.3.2          methods_4.1.2      
[17] jsonlite_1.7.2      AzureGraph_1.3.2    AzureAuth_1.3.3     openssl_1.4.5       base_4.1.2          askpass_1.1    

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Hey there @jmagosta! Still in Boston?

I can't replicate this on my own storage account:

az <- AzureRMR::get_azure_login("aicatr")
sub <- az$list_subscriptions()[[1]]
rg <- sub$get_resource_group("stor")
hongstor3 <- rg$get_storage_account("hongstor3")

bl <- hongstor3$get_blob_endpoint()
newcont <- storage_container(bl, "newcont")
list_storage_files(newcont)
# ...

sas <- "?sv=2020-08-04&..."
bl_endp <- blob_endpoint("https://hongstor3.blob.core.windows.net", sas=sas)
newcont <- storage_container(bl_endp, "newcont")
list_storage_files(newcont)
# ...

Does it work when you try accessing storage from outside AML?

Also, you don't need to run get_azure_login or create_azure_login if you're not actually using Azure Resource Manager. In this case you're working purely in the data plane, and ARM is never touched at all.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

I'm still not able to replicate this, although I can see how various errors can happen.

Make sure that the type of storage that you're trying to access matches the class of the storage endpoint object. In particular, if you do blob_endpoint("https://acctname.dfs.core.windows.net", *) where the URL contains "dfs" and not "blob", this will cause the 400 error "Value for one of the query parameters specified in the request URI is invalid."

Similarly if you do adls_endpoint("https://acctname.blob.core.windows.net", *) where the URL has "blob" and not "dfs", this will cause a 404 error "The specified container does not exist.".

Please post an exact, minimal sample of code that reproduces the problem, along with the exact error message that you're getting.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

One possibly workaround is to run options(azure_storage_api_version="2020-04-08") which is the API version from the previous AzureStor release. This will uncover whether changes in API version are causing any problems.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Oh, one more thing:

Here's an example that appears to authenticate, but returns null info (Could be a problem with the az login timing out)

Try removing the leading slash in the directory name (variance instead of /variance). I've tweaked the code to better handle this, but there may still be issues.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

One possibly workaround is to run options(azure_storage_api_version="2020-04-08") which is the API version from the previous AzureStor release. This will uncover whether changes in API version are causing any problems.

@hongooi73 Using options(azure_storage_api_version = "2020-04-08") gave the exact same error.
Error in list_adls_files(container, ...) : Bad Request (HTTP 400). Failed to complete Storage Services operation. Message: Value for one of the query parameters specified in the request URI is invalid. RequestId:39c415fe-... Time:2022-01-19T16:30:43.3998962Z.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Error in list_adls_files(container, ...)

Make sure you're not calling list_adls_files on a blob container object. That's what stands out from that error message.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

I'm still not able to replicate this, although I can see how various errors can happen.

Make sure that the type of storage that you're trying to access matches the class of the storage endpoint object. In particular, if you do blob_endpoint("https://acctname.dfs.core.windows.net", *) where the URL contains "dfs" and not "blob", this will cause the 400 error "Value for one of the query parameters specified in the request URI is invalid."

Similarly if you do adls_endpoint("https://acctname.blob.core.windows.net", *) where the URL has "blob" and not "dfs", this will cause a 404 error "The specified container does not exist.".

Please post an exact, minimal sample of code that reproduces the problem, along with the exact error message that you're getting.

Here is a code sample masking the organization data. Thanks in advance for the help!

az_token <-
  AzureRMR::get_azure_token(
    resource = "https://mystorage.dfs.core.windows.net/",
    tenant = tenant,
    app = app_id,
    password = app_secret,
    auth_type = "client_credentials", 
    use_cache = FALSE
  )

blob_endpoint <- AzureStor::storage_endpoint(endpoint = "https://mystorage.dfs.core.windows.net/", token = az_token)

container <- AzureStor::storage_container(blob_endpoint, cont_name)

adls_files <- AzureStor::list_storage_files(container,
                                            dir = "/folder/subfolder1/subfolder2/",
                                            info = "all",
                                            recursive = TRUE)

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

list_storage_files

Even though I'm using list_storage_files, the error prints Error in list_adls_files(container, ...)

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

It still works perfectly over here.

tok <- AzureAuth::get_azure_token("https://storage.azure.com/.default", ...)
ad_endp <- storage_endpoint("https://hongstor3.dfs.core.windows.net", token=tok)
cont <- storage_container(ad_endp, "contname")
list_storage_files(cont, "/tests/testthat", recursive=TRUE, info="all")

                                     name  size isdir        lastModified permissions
1                  tests/testthat/setup.R  1715 FALSE 2021-12-10 13:45:24   rw-r-----   
2        tests/testthat/test01_resource.R  2053 FALSE 2021-12-10 13:45:24   rw-r-----   
3            tests/testthat/test02_blob.R 11902 FALSE 2021-12-10 13:45:24   rw-r-----   
4        tests/testthat/test02a_blobext.R  4944 FALSE 2021-12-10 13:45:24   rw-r-----   
5       tests/testthat/test02b_blobdirs.R  3019 FALSE 2021-12-10 13:45:24   rw-r-----   
6     tests/testthat/test02c_blobappend.R  2516 FALSE 2021-12-10 13:45:24   rw-r-----
...

What is the exact error you're getting? And post your sessionInfo() along with whether you're running this locally or on a remote server.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

Here is the error. I'm running this from RSutdio-Workbench. When I revert to the AzureStor V3.5.2, it works.

Error in list_adls_files(container, ...) : 
  Bad Request (HTTP 400). Failed to complete Storage Services operation. Message:
Value for one of the query parameters specified in the request URI is invalid.
RequestId:38f08eeb-.........
Time:2022-01-19T17:34:40.2611651Z.
> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7                     pillar_1.6.2                   compiler_4.1.1                 RColorBrewer_1.1-2             AzureRMR_2.4.2                 tokenizers_0.2.1              
 [7] statquotes_0.2.2               tools_4.1.1                    bit_4.0.4                      jsonlite_1.7.2                 lifecycle_1.0.0                tibble_3.1.4                  
[13] lattice_0.20-44                pkgconfig_2.0.3                rlang_0.4.11                   Matrix_1.3-4                   AzureAuth_1.3.2                DBI_1.1.1                     
[19] curl_4.3.2                     yaml_2.2.1                     httr_1.4.2                     AzureStor_3.6.0                janeaustenr_0.1.5              dplyr_1.0.7                   
[25] stringr_1.4.0                  generics_0.1.0                 vctrs_0.3.8                     rappdirs_0.3.3                 bit64_4.0.5                   
[31] grid_4.1.1                     tidyselect_1.1.1               wordcloud_2.6                  glue_1.4.2                     R6_2.5.1                       fansi_0.5.0                   
[37] purrr_0.3.4                    magrittr_2.0.1                 SnowballC_0.7.0                ellipsis_0.3.2                 assertthat_0.2.1               tidytext_0.3.1                
[43] arrow_5.0.0                    config_0.3.1                   AzureGraph_1.3.1               utf8_1.2.2                     stringi_1.7.4                  crayon_1.4.1   

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Can you try running it in a local R session? I suspect that RStudio Workbench is screwing around with the URL that's being sent to the storage server. And you ARE authenticating with the AAD token, correct?

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

I'm using AAD token V1. It works on my local machine with V3.6.0, however, I rarely use it and the environment (R version and the installed packages) may be different.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Ok, if it works on your local machine but not on RStudio Workbench, that implies there's something funny going on with the latter. If there was a problem with the package, then it wouldn't work locally as well as on Workbench.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

You have any thoughts why would the previous version of AzureStor work on RStudio-Workbench, but not 3.6? We have contacts with RStudio partner but I need to understand more in order to raise the issue.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Can you check if it works if you authenticate with a SAS?

Also, does it work if you use the blob endpoint instead of ADLS2? It's accessing the same storage, but via a different API.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

Can you check if it works if you authenticate with a SAS?

Also, does it work if you use the blob endpoint instead of ADLS2? It's accessing the same storage, but via a different API.

I haven't tried to authenticate with SAS before. I'll have to do some research on that.

Using the blob endpoint seems to work fine, is there really any difference between using the dfs vs blob endpoints?

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

Try removing the leading slash in the directory name (variance instead of /variance).

I noticed there's a dependence between the URL passed to storage_endpoint() and the name passed to storage_container(). If I include the container in the storage_endpoint URL, then I need to just pass an empty string for the storage container name, and vice versa. Then on top of that I need to specify a path for list_storage_files() Seems there needs to be a more intuitive way to specify what is really a combined account - container - blob path among the three methods.

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

Hey @hongooi73 - actually still in Mountain View. I get to visit Boston if I'm good. :)
Maybe we should split this issue -- I'm loosing track of the complaint.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

The endpoint is the domain name (+scheme) for your storage account: https://acctname.svc.core.windows.net. You only ever create an AzureStor storage_endpoint object with this domain name.

The container is the first element of the path: containername. This is a blob container, a file share, or and ADLS filesystem.

Inside a container, you have a path to a file: path/to/file.

The addressing syntax lets you combine all of these into a single URL: https://acctname.svc.core.windows.net/containername/path/to/file.

As a convenience, storage_container lets you specify a container using a number of methods:

  • As a combination of endpoint object and container name:

    endp <- storage_endpoint("https://acctname.svc.core.windows.net", *)
    cont <- storage_container(endp, "containername")
    
  • As a string with both the domain name and container name, along with the authentication args:

    endp <- storage_container("https://acctname.svc.core.windows.net/containername", *)
    

If this is all too complicated to understand, just stick to the first method. It's what I use.

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

The storage documentation is quite consistent in calling the endpoint the domain name portion only. This extends to, eg, the portal, under the "Endpoints" tab:

image

There is nothing in the AzureStor documentation or elsewhere that would suggest including the container name inside the endpoint.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

Can you check if it works if you authenticate with a SAS?

Also, does it work if you use the blob endpoint instead of ADLS2? It's accessing the same storage, but via a different API.

@hongooi73 Same error using SAS.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

Hey @hongooi73 - actually still in Mountain View. I get to visit Boston if I'm good. :) Maybe we should split this issue -- I'm loosing track of the complaint.

@jmagosta Sorry for hijacking your issue, the error is identical and I didn't want to create a duplicate issue.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

@MichaelHannaEQT this is the issue for tracking RStudio Server/Workbench problems, no worries.

@jmagosta were you having this same problem on AML? Or was it something different?

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

The storage documentation is quite consistent in calling the endpoint the domain name portion only.

This being the case, then the code shown here should throw a warning when an extended endpoint is passed, and not allow a null name field when specifying the container, if only to avoid confusion for the user: (Right now this pattern actually works ..)

  the_endpoint <- AzureStor::storage_endpoint(
      'https://the_storage_account_name.blob.core.windows.net/the_container_name', ...)
  the_container <- AzureStor::storage_container(the_endpoint, name='')

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

@MichaelHannaEQT no problem! @hongooi73 As for having the same problem in AML - since this is an AzureStor library issue, I don't know what the equivalent AML method would be. (Probably authentication by managed identity, I imagine). The end-goal here is to avoid SaS or account key secrets from littering the code.

Here's an example that works in my subscription for getting a blob endpoint, but fails in a subscription where I have Contributor access to the storage account but not to the resource group that is throwing the error. RG privileges would not seem necessary to be able to authenticate to the storage---which do have access to otherwise.

subscription_id <-'XXXXXXX-XXXX-XXXX-XXXX-XXXXXX'
sub1 <- az$get_subscription(subscription_id)
rg <- sub1$get_resource_group('Albatross')

# AuthorizationFailed
# The client '[email protected]' with object id '041d7dbc-7024-4f37-8a45-c0913ccb8575' does not have authorization to perform
# action 'Microsoft.Resources/subscriptions/resourcegroups/read' over scope
# '/subscriptions/XXXXXXX-XXXX-XXXX-XXXX-XXXXXX/resourcegroups/Albatross' or the scope is invalid

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

That's a different problem entirely. You're not authenticating to storage, you're trying to access the Azure resource group that contains your storage account.

from azurestor.

jmagosta avatar jmagosta commented on July 18, 2024

Yes - but why should I need to have permissions on the resource group simply to be able to authenticate to storage? The whole point is to have an alternative to using SaS secrets on storage one has reader roles.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

@jmagosta your problem is unrelated to this issue, but briefly: you are NOT authenticating to storage. AzureRMR is an interface to Azure Resource Manager which is the management plane for Azure resources. Authenticating there doesn't give you access to the data plane, which is what you need to actually read your files and whatnot.

If you want to access the data plane via AAD, see the vignettes or the readme for this repo.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

@MichaelHannaEQT have you been able to resolve this yet?

from azurestor.

mjkanji avatar mjkanji commented on July 18, 2024

I'm also experiencing the same problem. On my local machine (RStudio Desktop), it works fine, but when using RStudio Server on an Azure ML Compute Instance, I'm getting the same error.

Reverting to version 3.5.2 fixes it.

Similarly, using org.**blob**.core.windows.net instead of org.**dfs**.core.windows.net returns a valid response, though it of course recursively shows all the contents of the sub-directories instead of just the top level.

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

I have a tentative fix in the path-fix branch. Can you try that and see if it changes anything?

from azurestor.

mjkanji avatar mjkanji commented on July 18, 2024

I have a tentative fix in the path-fix branch. Can you try that and see if it changes anything?

Just tried with the path-fix branch and it works! 🎊 🎆

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

@mjkanji excellent. Can you post the exact code that broke with 3.6.0 and works with the fix?

I can see the change I made in 3.6.0, but I can't see why it should fail....

from azurestor.

mjkanji avatar mjkanji commented on July 18, 2024

Hi @hongooi73, please see below:

> pak::pkg_install("AzureStor") # v3.6.0 from CRAN is already installed so installation is skippedLoading metadata database ... doneNo downloads are needed1 pkg + 19 deps: kept 18 [3.6s]

# Restarting R session to reload packages
Restarting R session...

> list_files = function() {
+   org_name = "..."
+   some_container = "..."
+   some_dir = "..."
+   
+   endpoint <- paste0("https://", org_name, ".dfs.core.windows.net")
+   
+   token <- AzureAuth::get_azure_token("https://storage.azure.com",
+                                       tenant=Sys.getenv("TENANT_ID"),
+                                       app=Sys.getenv("APP_ID"),
+                                       password=Sys.getenv("CLIENT_SECRET"))
+   
+   endp_token = AzureStor::storage_endpoint(endpoint, token = token)
+   
+   container_name <- some_container
+   cont <- AzureStor::storage_container(endp_token, container_name)
+   
+   AzureStor::list_storage_files(container = cont, some_dir)
+ }
> list_files()
Using client_credentials flow
Loading cached token
 Error in list_adls_files(container, ...) : 
  Bad Request (HTTP 400). Failed to complete Storage Services operation. Message:
Value for one of the query parameters specified in the request URI is invalid.
RequestId: ...
Time: ... 

# Now installing AzureStor using the path-fix branch
> pak::pkg_install("Azure/AzureStor@path-fix")
! Using bundled GitHub PAT. Please add your own PAT using `gitcreds::gitcreds_set()`.Loading metadata database ... doneWill update 1 package.Will download 1 package with unknown size.
+ AzureStor 3.6.03.6.0.9000 [bld][cmp][dl] (GitHub: 1796dd3)

! AzureStor is loaded in the current R session, you probably need to restart R
after the installation.

? Do you want to continue (Y/n) YGetting 1 pkg with unknown sizeCached copy of AzureStor 3.6.0.9000 (source) is the latest buildNo downloads needed, all packages are cachedPackaging AzureStor 3.6.0.9000Packaged AzureStor 3.6.0.9000 (557ms)                              
ℹ Building AzureStor 3.6.0.9000Built AzureStor 3.6.0.9000 (2.8s)                                  
✓ Installed AzureStor 3.6.0.9000 (github::Azure/AzureStor@1796dd3) (28ms)
✓ 1 pkg + 19 deps: kept 17, upd 1 [11.2s]                              

# Restarting R session to reload packages
Restarting R session...

> list_files = function() {
+   org_name = "..."
+   some_container = "..."
+   some_dir = "..."
+   
+   endpoint <- paste0("https://", org_name, ".dfs.core.windows.net")
+   
+   token <- AzureAuth::get_azure_token("https://storage.azure.com",
+                                       tenant=Sys.getenv("TENANT_ID"),
+                                       app=Sys.getenv("APP_ID"),
+                                       password=Sys.getenv("CLIENT_SECRET"))
+   
+   endp_token = AzureStor::storage_endpoint(endpoint, token = token)
+   
+   container_name <- some_container
+   cont <- AzureStor::storage_container(endp_token, container_name)
+   
+   AzureStor::list_storage_files(container = cont, some_dir)
+ }
> list_files()
Using client_credentials flow
Loading cached token
  name size isdir lastModified permissions etag
1  ...  ...   ...          ...         ...  ...

For reference, I'm using RStudio on an Azure ML compute instance and the sessionInfo is:

R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.6 LTS

Matrix products: default
BLAS/LAPACK: /opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64_lin/libmkl_rt.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] forcats_0.5.1    stringr_1.4.0    dplyr_1.0.7.9000 purrr_0.3.4      readr_2.1.1     
[6] tidyr_1.1.4      tibble_3.1.5     ggplot2_3.3.5    tidyverse_1.3.1 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.7           cellranger_1.1.0     pillar_1.6.4         compiler_4.1.2      
 [5] dbplyr_2.1.1         AzureRMR_2.4.3       tools_4.1.2          lubridate_1.8.0     
 [9] jsonlite_1.7.3       lifecycle_1.0.1      gtable_0.3.0         pkgconfig_2.0.3     
[13] rlang_1.0.0          reprex_2.0.1         AzureAuth_1.3.3      rstudioapi_0.13     
[17] cli_3.1.1            DBI_1.1.2            curl_4.3.2           haven_2.4.3         
[21] xml2_1.3.3           withr_2.4.3          httr_1.4.2           AzureStor_3.6.0.9000
[25] fs_1.5.2             hms_1.1.1            generics_0.1.1       vctrs_0.3.8         
[29] askpass_1.1          rappdirs_0.3.3       grid_4.1.2           tidyselect_1.1.1    
[33] glue_1.6.0           R6_2.5.1             fansi_1.0.2          readxl_1.3.1        
[37] tzdb_0.2.0           modelr_0.1.8         magrittr_2.0.1       backports_1.4.1     
[41] scales_1.1.1         ellipsis_0.3.2       rvest_1.0.2          assertthat_0.2.1    
[45] colorspace_2.0-2     utf8_1.2.2           AzureGraph_1.3.2     stringi_1.7.6       
[49] munsell_0.5.0        openssl_1.4.5        broom_0.7.10         crayon_1.4.2  

from azurestor.

hongooi73 avatar hongooi73 commented on July 18, 2024

Looks like another Linux/Windows file.path hassle. I can repro this in R running in a WSL2 instance on my laptop.

from azurestor.

Mike-Hanna avatar Mike-Hanna commented on July 18, 2024

@hongooi73 The update you made fixed the issue. Thank you for your help!

from azurestor.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.