Comments (10)
@naoyak Hmmm... looking at this more closely, this failure was "designed", in that it didn't seem sensible to me at the time to have inconsistently nested definitions, which could create ambiguous definitions. In this case, why not just suppress the database
nesting, and instead have:
data_environments:
env1:
presto:
protocol: presto
host: localhost
port: 9999
catalog: silver
schema: default
hive:
protocol: hiveserver2
host: localhost
port: 3333
schema: default
driver: pyhive
metastore:
host: localhost
port: 5555
If metastore
is not a duct, then just create a new nesting for omniduct
or services
, and pass only the subset of configurations to Omniduct, like:
data_environments:
env1:
services:
databases: ....
metastore:
....
from omniduct.
I'm going to reopen this for discussion - if we can provide a simple patch which handles "empty" nesting levels gracefully, would it be acceptable to re-enable the handling? What do you think @matthewwardrop?
Again for context, we have some production libraries which currently pin an old version of Omniduct in order to deal with a specific legacy YAML config on the servers. cc @OneRaynyDay
from omniduct.
@naoyak @OneRaynyDay I've included a fix for this in Omniduct 1.1.9 (929a39a). It's a more generous parser that traverses the configuration looking for any potential Duct configurations, and so allows configurations like the one you provided. It's also backward compatible with any configurations for Omniduct <= 1.1.8. I'll close this issue out; but feel free to reopen if this doesn't solve your issues.
from omniduct.
Roger that... I'll look into this tonight. Thanks for the report!
from omniduct.
Thanks @matthewwardrop! For context, I think this is happening because the envs defined in the YAML config have a mix of Duct
s (Hive, Presto) and unrelated stuff which doesn't follow the convention expected by Omniduct. I suppose this is something that could be handled gracefully, e.g. by ignoring sections which don't look like what's expected.
from omniduct.
@matthewwardrop thanks for checking it out! Yeah, you're probably right about this. It seems to be as simple as either tweaking the YAML format or tweaking the read-in logic, maybe even a one-liner. It's on the radar internally and should be fixed there.
from omniduct.
Thanks for reporting in @naoyak.
from omniduct.
By "empty" do you mean levels which do not have a "protocol" field? I think I'd even be willing to change the parser to just check if there is a "protocol" field at any level, and if so, interpret that as a Duct configuration. Would that work for you?
from omniduct.
@matthewwardrop I think that would work! If Omniduct simply ignores entries w/o a protocol
entry instead of throwing the exception, it would likely accommodate our usage.
@OneRaynyDay can you confirm?
from omniduct.
Thanks @matthewwardrop! This looks great, we'll go with 1.1.9 going forward and hopefully this issue will be resolved for good on our end.
from omniduct.
Related Issues (20)
- v1.1.4 breaks SSHClient HOT 3
- It is not safe to use "input" in python2 HOT 1
- DatabaseClient stream() and stream_to_file()
- If table exists, then append and replace do not work when adding a dataframe to a table HOT 2
- ssh connect error HOT 1
- Druid Connection HOT 1
- Access problem in hiveserver2 HOT 5
- Connecting to presto through https HOT 6
- More examples HOT 6
- Hive DataFrame columns and push errors HOT 4
- Some issues for cassandra protocol HOT 3
- tests? HOT 1
- missing checks in _mkdir method in LocalFsClient HOT 10
- implementation of mkdir and remove for s3 filesystem HOT 1
- Implement Azure Data Lake Client HOT 2
- REST implement put and post the data ? HOT 2
- Hive dataframe_to_table failing on non-remote execution
- logo HOT 11
- Handling connection failover HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from omniduct.