gdbtek / aws-tools Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
The base URL of s3.amazonaws.com is assigned and cannot be changed by the user. However, when accessing via that base URL, the following message can appear when using the link:
PermanentRedirect
The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
...
Adding a Regions option (-r) will eliminate this issue.
When creating an URL for eu-central-1 i receive the following error
<Error>
<Code>InvalidRequest</Code>
<Message>
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
</Message>
...
</Error>
How can i fix this?
The standard AWS environmental variables are AWS_ACCESS_KEY and AWS_SECRET_KEY (used by lots of 3rd party libs/tools). Would be nice to use those instead of forcing users to copy/paste ever time.
Hi,
I have a private S3 bucket and created an URI using this tool as below:
./sign_s3_url.bash --bucket 'eae-data-test' --aws-access-key-id 'my-key' --aws-secret-access-key 'secret-key' --file-path 'hdf/customer' --region 'us-east-1' --minute-expire 10001
Above script created a link similar as below:
https://s3.amazonaws.com/eae-data-test/hdf/customer?AWSAccessKeyId=AK6&Expires=1533802548&Signature=MJ%2E%3D
I tried using this link in pandas as below:
import pandas as pd
df=pd.read_sql('SELECT * FROM SQL-SERVER-TABLE', sql-server-pypyodbc-connection)
uri='https://s3.amazonaws.com/eae-data-test/hdf/customer?AWSAccessKeyId=AK6&Expires=1533802548&Signature=MJ%2E%3D'
df.to_hdf(uri,key='df')
Above 'to_hdf' throws below error:
File "/eae/env/dts/lib/python3.6/site-packages/pandas/core/generic.py", line 1471, in to_hdf
return pytables.to_hdf(path_or_buf, key, self, **kwargs)
File "/eae/env/dts/lib/python3.6/site-packages/pandas/io/pytables.py", line 280, in to_hdf
complib=complib) as store:
File "/eae/env/dts/lib/python3.6/site-packages/pandas/io/pytables.py", line 467, in init
self.open(mode=mode, **kwargs)
File "/eae/env/dts/lib/python3.6/site-packages/pandas/io/pytables.py", line 580, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/eae/env/dts/lib/python3.6/site-packages/tables/file.py", line 320, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/eae/env/dts/lib/python3.6/site-packages/tables/file.py", line 784, in init
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 371, in tables.hdf5extension.File._g_new
File "/eae/env/dts/lib/python3.6/site-packages/tables/utils.py", line 185, in check_file_access
check_file_access(filename, 'w')
File "/eae/env/dts/lib/python3.6/site-packages/tables/utils.py", line 175, in check_file_access
raise IOError("%s
does not exist" % (parentname,))
OSError: https://s3.amazonaws.com/eae-data-test/hdf
does not exist
Could you please help me with this?
Hi,
This code looks useful, however...
Please can you check in the util.bash dependency, else steer me to where it lives.
Cheers,
R
It would be nice if eu-central-1 (frankfurt) would be allowed as region parameter
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.