Conditionally allow to keep partition_by columns when using PARTITIONED BY about arrow-datafusion HOT 3 OPEN

Seems like a good idea to me

from arrow-datafusion.

I spent some time looking into how to implement this. I have a very basic change that I think would bring the functionality:

main...hveiga:datafusion:issue-10971

I am looking for guidance on two aspects:

• Where is there best place in the code base to add a test case for this?
• If we want to allow to pass this flag as part of the COPY command (ie COPY (...) TO (...) STORED AS FILE_FORMAT (OPTIONS keep_partition_by_columns true)), what would be the right way to pass this flag to hive_style_partitions_demuxer?

I see that sql/src/parser.rs can parse the option and keep it the CopyTo struct under options, and then those get used to configure the different FormatOptions for the different output formats. This particular flag would apply to all formats I believe. Should I add it as another config to TableOptions ?

Thanks!

from arrow-datafusion.

Where is there best place in the code base to add a test case for this?

sqllogictests are a good place to start. You can do a COPY with partitions and the new option toggled, then create a table on just a single partition path, and do a select that verifies the partition column is in the file.

If we want to allow to pass this flag as part of the COPY command (ie COPY (...) TO (...) STORED AS FILE_FORMAT (OPTIONS keep_partition_by_columns true)), what would be the right way to pass this flag to hive_style_partitions_demuxer?

You could check for the new option in source_option_tuples and then update FileSinkConfig to hold that option and flow it through to the relevant functions. The relevant physical planner code is here

from arrow-datafusion.