Truncate stats from Parquet files about iceberg HOT 6 CLOSED

DataFile and Metrics are the classes that contain metrics and are good candidates for where truncation could happen. I think we would want truncation to be configurable using settings in TableProperties. Metrics are scraped from Parquet metadata in ParquetMetrics, which is called by ParquetWriter.

You might want to explore passing a truncate length option to ParquetWriter. The writer would pass it to ParquetMetrics to truncate values right away. The setting would come from the table when creating a writer. For that, I think you'd update the write builder in Parquet.

from iceberg.

@aokolnychyi, #78 reminded me about this item as well.

from iceberg.

@rdblue is it an issue good for n00b? if yes, I am interested to take this one :)

from iceberg.

@feng-tao, I think this could be a good first issue. Let us know if you need any help or context.

from iceberg.

@rdblue , I am still reading the code base. It would be great if you could guide me a little bit on the context or the related code path. Thanks a lot :)

from iceberg.

thanks @rdblue , will take a look

from iceberg.