Connecting to presto through https about omniduct HOT 6 CLOSED

Hi @yaxxie

When creating an omniduct PrestoClient, you can pass additional connection_options as keyword arguments (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L38). These then get passed to pyhive's presto client. In this case I think you want protocol='https'. (https://github.com/dropbox/PyHive/blob/19d7c2b8fb8b19dadeb2339d4d3f42c8f09204c1/pyhive/presto.py#L93)

I'll close this but feel free to re-open if that doesn't work for you.

from omniduct.

Hi, it did not work for me. I can see from the source code that while the kwargs get passed to the connect call, they're not being passed to the SQLAlchemy engine: https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L76

from omniduct.

@danfrankj are you able to re-open? I'm not able to reopen this.

from omniduct.

@yaxxie Ah I see, the sql_alchemy engine is only used for schema browsing and saving dataframes ("push"ing). Are you able to query using connection_options (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L75) ? This should only use the pyhive client and does not rely on the sqlalchemy engine.

from omniduct.

@danfrankj sorry for wasting your time -- I just realised that this is to do with a known issue with presto returning http URLs when using HTTPs terminated at a load balancer.
The sqlalchemy engine would still need the protocol passed through if SSL were enabled and we wanted to use the dataframe retrieval function, right? So perhaps that needs to be an enhancement request/PR?

from omniduct.

@yaxxie that is an annoying issue and hopefully the presto team is working on it.

for omniduct the connection we use to issue queries is pyhive's presto client (https://github.com/airbnb/omniduct/blob/master/omniduct/databases/presto.py#L73) which you can already configure to use https and retrieve dataframes.

from omniduct.