RecordReaderImpl.getValueRange() may cause incorrect results about orc HOT 7 OPEN

Thank you for reporting, @PengleiShi .

2. Could you provide some sample ORC files?

Most of files wrote by trino have proper statistics. I will try to re-generate some problem orc files which can be public.
The meta of problem files is below:

from orc.

@dongjoon-hyun. Trino won't write string column statistics if string value is bigger than 64 bytes

Here is a orc file wrote by trino and contains only one row

Test with spark3.2

select * from xxx; 
select * from xxx where name is not null;

20220310_100444_03858_nbvwj_53625cc9-7183-4beb-be48-9d059d8fa560.zip

from orc.

Thank you for reporting, @PengleiShi .

1. Ya, I've heard that there exists ORC writers that doesn't generate statistics properly.
2. Could you provide some sample ORC files?

from orc.

AFAIK, this doesn't happen between Apache ORC writer and reader, right? @PengleiShi

from orc.

AFAIK, this doesn't happen between Apache ORC writer and reader, right? @PengleiShi

Yes, it doesn't. In the case i mentioned, the files were wrote by trino(which has own orc writer) and read by spark(which depends on Apache ORC reader).

from orc.

@PengleiShi do you mind sharing the stats of the problematic case above?
We currently trim StringStatistics to 1024 chars in the default writer https://issues.apache.org/jira/browse/ORC-203
I believe Presto should follow a similar logic.
In addition, on the Reader path we probably want to avoid skipping RowGroups when facing problematic/null ValueRange stats.

from orc.

@dongjoon-hyun. Trino won't write string column statistics if string value is bigger than 64 bytes Here is a orc file wrote by trino and contains only one row Test with spark3.2

select * from xxx; 
select * from xxx where name is not null;

20220310_100444_03858_nbvwj_53625cc9-7183-4beb-be48-9d059d8fa560.zip

@pgaref here i have uploaded a problematic file for test. Its meta shows below

from orc.