BUG: reading boolean column with RLE encoding gives wrong values about fastparquet HOT 4 CLOSED

Yes, we very probably assume bools should be bit-packed. Is there really any case in which RLE achieves better packing or read performance?

https://parquet.apache.org/docs/file-format/data-pages/encodings/

Note that the RLE encoding method is only supported for the following types of data:

Repetition and definition levels
Dictionary indices
Boolean values in data pages, as an alternative to PLAIN encoding

The last point was certainly not there the last time I looked at the spec, but it might have been a while.

from fastparquet.

The last point was certainly not there the last time I looked at the spec, but it might have been a while.

The website is newer, but in the actual doc page it was added 6 years ago: apache/parquet-format#79

I am not aware of an extensive performance comparison, but from the discussion on the Arrow side, it was mentioned that also several other Parquet implementations (eg parquet-rs) already default to use RLE for boolean values.

from fastparquet.

I don't gather from the docs, it RLE bool encoding restricted to V2 data pages?

from fastparquet.

AFAIK encoding options are completely separate from V1 vs V2 data pages

from fastparquet.