How to write in streaming fashion? about parquet4s HOT 6 CLOSED

Hi @mac01021!
If Scala's Stream doesn't help then you may try splitting your large batch into smaller batches. Check recently introduced https://github.com/mjakubowski84/parquet4s/blob/master/core/src/main/scala/com/github/mjakubowski84/parquet4s/IncrementalParquetWriter.scala
You should be able to write multiple batches to single file before closing.

from parquet4s.

Thanks @mjakubowski84!

This looks like just what I need!

Can you provide any guidance related to the cost of a batch? Will the file come out the same whether I write it in a few large batches vs many tiny batches?

from parquet4s.

Hard to tell! It is strongly correlated to the size of your entity. Performance relates also to the speed and latency of your IO. I recommend you to do a series of experiments with different setup.

from parquet4s.

But does it do anything specific I should know about, like creating a new row group for every batch?

from parquet4s.

No, nothing differs comparing to normal writer except fact that you have to call close on your own. Row groups, pages, etc, are still created according the options provided to the writer.

from parquet4s.

Awesome! Thanks!

from parquet4s.