Nonstandard YAML config format results in crash about omniduct HOT 10 CLOSED

@naoyak Hmmm... looking at this more closely, this failure was "designed", in that it didn't seem sensible to me at the time to have inconsistently nested definitions, which could create ambiguous definitions. In this case, why not just suppress the database nesting, and instead have:

data_environments:
    env1:
        presto:
            protocol: presto
            host: localhost
            port: 9999
            catalog: silver
            schema: default
        hive:
            protocol: hiveserver2
            host: localhost
            port: 3333
            schema: default
            driver: pyhive
        metastore:
            host: localhost
            port: 5555

If metastore is not a duct, then just create a new nesting for omniduct or services, and pass only the subset of configurations to Omniduct, like:

data_environments:
    env1:
        services:
             databases: ....
        metastore:
             ....

from omniduct.

I'm going to reopen this for discussion - if we can provide a simple patch which handles "empty" nesting levels gracefully, would it be acceptable to re-enable the handling? What do you think @matthewwardrop?

Again for context, we have some production libraries which currently pin an old version of Omniduct in order to deal with a specific legacy YAML config on the servers. cc @OneRaynyDay

from omniduct.

@naoyak @OneRaynyDay I've included a fix for this in Omniduct 1.1.9 (929a39a). It's a more generous parser that traverses the configuration looking for any potential Duct configurations, and so allows configurations like the one you provided. It's also backward compatible with any configurations for Omniduct <= 1.1.8. I'll close this issue out; but feel free to reopen if this doesn't solve your issues.

from omniduct.

Roger that... I'll look into this tonight. Thanks for the report!

from omniduct.

Thanks @matthewwardrop! For context, I think this is happening because the envs defined in the YAML config have a mix of Ducts (Hive, Presto) and unrelated stuff which doesn't follow the convention expected by Omniduct. I suppose this is something that could be handled gracefully, e.g. by ignoring sections which don't look like what's expected.

from omniduct.

@matthewwardrop thanks for checking it out! Yeah, you're probably right about this. It seems to be as simple as either tweaking the YAML format or tweaking the read-in logic, maybe even a one-liner. It's on the radar internally and should be fixed there.

from omniduct.

Thanks for reporting in @naoyak.

from omniduct.

By "empty" do you mean levels which do not have a "protocol" field? I think I'd even be willing to change the parser to just check if there is a "protocol" field at any level, and if so, interpret that as a Duct configuration. Would that work for you?

from omniduct.

@matthewwardrop I think that would work! If Omniduct simply ignores entries w/o a protocol entry instead of throwing the exception, it would likely accommodate our usage.

@OneRaynyDay can you confirm?

from omniduct.

Thanks @matthewwardrop! This looks great, we'll go with 1.1.9 going forward and hopefully this issue will be resolved for good on our end.

from omniduct.