Support virtual-hosted–style only S3-compatible remote? about dvc HOT 7 OPEN

DVC should work properly with the virtual-host style endpointurl addressing. The error indicates that you don't have the right permissions to access that bucket. Are you able to use the AWS CLI to access that byte-plus bucket?

from dvc.

It looks like you have to pass addressing_style: 'virtual' to botocore to enable this?

boto/boto3#2477

from dvc.

In botocore it defaults to auto, where if you are setting an endpointurl it uses virtual style and then falls back to path
https://github.com/boto/botocore/blob/e7c5b6ab22174797db551f44053a0b2245430649/botocore/utils.py#L2604-L2614

from dvc.

Thank you, @skshetry and @pmrowla.
When forcing boto3 to use {'addressing_style': 'virtual'}, I can access byte-plus bucket:

when not, it raises error:

What should I do to make dvc work？

from dvc.

The error given with s3fs/DVC is a permission error when accessing a specific file. It does not look like s3fs is not failing to create the client session (InvalidPathAccess is not the raised exception when you use DVC). This would normally mean that the issue is specifically with the credentials you are setting for your DVC remote (and not that the issue is due to an incorrect addressing style)

@0ut0fcontrol can you verify that you are able to access your bucket via the AWS CLI?

from dvc.

@pmrowla Yes, I can access my bucket via aws cli:

And, if I insert config_kwargs['s3'] = {'addressing_style': 'virtual'} in here, dvc will work fine.
However, I don't know how to pass this kwarg from dvc cli into s3fs.

• add {'addressing_style': 'virtual'}
  
  

• Comment out {'addressing_style': 'virtual'}, dvc not work again.
  
  

from dvc.

This looks to me like it may be an aiobotocore or botocore bug since their behavior to default to virtual host addressing and fall back to path addressing may not be working correctly?

But we can add support for setting this explicitly in dvc-s3 dvc-s3 (we set config timeouts in the same way). It will also need to get added to the DVC remote config schema
https://github.com/iterative/dvc-s3/blob/43f70226160f1a5c9ffdad4092d41a8bab7ec19b/dvc_s3/__init__.py#L176-L179

from dvc.