Comments (6)
Hi @yaxxie
When creating an omniduct PrestoClient, you can pass additional connection_options as keyword arguments (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L38). These then get passed to pyhive's presto client. In this case I think you want protocol='https'
. (https://github.com/dropbox/PyHive/blob/19d7c2b8fb8b19dadeb2339d4d3f42c8f09204c1/pyhive/presto.py#L93)
I'll close this but feel free to re-open if that doesn't work for you.
from omniduct.
Hi, it did not work for me. I can see from the source code that while the kwargs get passed to the connect
call, they're not being passed to the SQLAlchemy engine: https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L76
from omniduct.
@danfrankj are you able to re-open? I'm not able to reopen this.
from omniduct.
@yaxxie Ah I see, the sql_alchemy engine is only used for schema browsing and saving dataframes ("push"ing). Are you able to query using connection_options
(https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L75) ? This should only use the pyhive client and does not rely on the sqlalchemy engine.
from omniduct.
@danfrankj sorry for wasting your time -- I just realised that this is to do with a known issue with presto returning http URLs when using HTTPs terminated at a load balancer.
The sqlalchemy engine would still need the protocol passed through if SSL were enabled and we wanted to use the dataframe retrieval function, right? So perhaps that needs to be an enhancement request/PR?
from omniduct.
@yaxxie that is an annoying issue and hopefully the presto team is working on it.
for omniduct the connection we use to issue queries is pyhive's presto client (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L73) which you can already configure to use https and retrieve dataframes.
from omniduct.
Related Issues (20)
- v1.1.4 breaks SSHClient HOT 3
- It is not safe to use "input" in python2 HOT 1
- DatabaseClient stream() and stream_to_file()
- If table exists, then append and replace do not work when adding a dataframe to a table HOT 2
- ssh connect error HOT 1
- Druid Connection HOT 1
- Access problem in hiveserver2 HOT 5
- More examples HOT 6
- Hive DataFrame columns and push errors HOT 4
- Some issues for cassandra protocol HOT 3
- tests? HOT 1
- missing checks in _mkdir method in LocalFsClient HOT 10
- implementation of mkdir and remove for s3 filesystem HOT 1
- Implement Azure Data Lake Client HOT 2
- REST implement put and post the data ? HOT 2
- Hive dataframe_to_table failing on non-remote execution
- logo HOT 11
- Nonstandard YAML config format results in crash HOT 10
- Handling connection failover HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from omniduct.