overwrite SaveMode support for Spark write API about iceberg HOT 3 CLOSED

We can have small files committed as part of ETL/Streaming operations. An obvious next step is to create a job which can over-write files to create more compacted files. Spark module currently only supports "Append" while creating a writer. For over-writing files, there is a need to extend the writer to support other modes as well. We already have overWrite and rewriteFile APIs. Can they be used to implement the new functionality?

cc @rdblue @prakharjain09

from iceberg.

@prakharjain09, Spark is removing SaveMode because it has unreliable behavior. Instead, Spark is introducing overwrite APIs that match some of the APIs that Iceberg supports. See SupportsDynamicOverwrite and SupportsOverwrite in the latest Spark DSv2 design doc.

When Spark supports those, Iceberg will be able to support overwrite.

from iceberg.

I went ahead and added this recently in #246 to support overwrite in Spark 2.4 since it is taking a while to get 3.0 out. It should work the same way in 3.0, but we will no longer recommend using this API because it doesn't state the behavior sources should implement (overwrite, truncate, replace table are all allowed).

See the overwrite docs and #246.

from iceberg.