Retaining original columns after inference about tensorflowonspark HOT 2 CLOSED

There are a couple ways to achieve this:

• add a tf.identity(x) to your model to echo whatever inputs you desire.
• use a Spark UDF to invoke the model using df.withColumn(udf(x)).

Unfortunately, I don't have any ready examples of these...

Alternatively, if you want to create a PR with your modifications to pipeline.py, I can take a look at incorporating it (assuming that we have a way to preserve backwards-compatibility for folks who expect the prior behavior).

from tensorflowonspark.

Thank you, @leewyang for your advice! I will try your advice next time in similar cases.

Regarding the PR, I don't want to contaminate your beautiful codebase.

What I did in pipeline.py was that:

1. _transform()
   Use a list to keep track of original columns other than the input_cols.
   dataset.select(all of the columns instead of just input features).
   append original columns' names in spark.createDataFrame() in addition to output_cols.
2. yield_batch()
   Add a list to contain additional original columns like tensors[].
   In the for item in iterable:, append the original columns into that list; yield tensors, original_columns_list.
   So, I also need to change _run_model_tf1 and _run_model_tf2.
3. _run_model_tf1 and _run_model_tf2
   Get the original columns list from yield_batch() along with tensors, put it in the result.extend(zip(*(here + python_outputs)))

Personally, I was looking for better ways of handing this. Thank you very much!

from tensorflowonspark.