konrads / spark-etl Goto Github PK

Reportable from driver?

and ensure -Dvar=value is used in the substitution

So that you can prune them out of the app.yaml

Ie. so that uri prefix is "parquet:" then we look for Parquet LoadWriter plugin, instantiate only that. Will cater for a combination of different strategies of reading/writing.

Might have a lifecycle:

• for local validation - just validate
• for running on the server (validate-remote, transform-load, etc)
  • instantiate context (eg. Oracle connection)
  • run the action
  • close the context
  • post run step, eg. to record passes/failures?

which will be responsible for savings all transform results. It will need:

• config section, eg:
  sinker:

  • impl: spark_etl.parquet.Sink
  • args: # optional
    • param1: strVal1
    • param2: strVal2

• an abstract class:

  • constructor(options: Map[String, String])
  • def validate(Seq[Transform]) # could fail if unsupported Transform output?
  • sink(transformsAndDfs: Seq[(Transform, DataFrame)])

• spark_etl.parquet.Sink, extending the above

• changes to MainUtil:

  • add Sink.validate() to extractCheck
  • add Sink.sink() to transform
  • consider renaming extractCheck, transformCheck, transform...?

app.yaml should contain:

• extract
• transform
• load

The pipelines are derived top-down, ie. see which "load" dependencies are defined in "transform", then which "transform" dependencies are defined in "extract"

Which will work inside Dataset.transform()
As demo implementation in "plugins" configuration section, which will eg. perform typing of Strings to Int/Double/Date/Timestamp/etc. Typing should be done Row[Strings] -> Row[Int/Double/Date/Timestamp], ie. not Row[String] -> case classes/classes.