lemire / frameofreference Goto Github PK

If i want to compress 64bit integers, what should i do?

When compressing, the code adds some padding toward the end to preserve 32-bit alignment. This could be removed, at the cost of portability.

Hey,

How hard would it be to implement direct access here?

Best,
Krystian

Hi, I found that for a particular dataset of 32 64-bits numbers, the compressed-and-then-uncompressed array differ from the original one.

I debugged this and for the compression phase, the pack function chose is packforblock38_64 but when the compressed array is uncompress, the turbouncompress64 calls bits instead of bits64 yielding as result int b = 32 instead of int b = 38.

Additional notes

I created in a personal branch with the environment that I used for testing.

I adapted the test binary to read samples of 64 bits (test.src) and the sample file t2sample.txt can be found also in the same branch.

To reproduce the issue, compile and run ./test t2sample.txt.
Here is the output:

No OpenMP support
####### processing t2sample.txt
# Parsing vector file t2sample.txt

[turbo benchmark64]
vector size = 32
vector size = 0.25KB
compression rate:1.5
bits/int:43
volume: 0.17KB
idx: 1 original/decompressed: 1560036738145229/1560036738221396
idx: 2 original/decompressed: 1560036742630299/1560037934073620
idx: 3 original/decompressed: 1560036742631201/1560040499648280
<....>

The output shows how the original and the decompressed bytes differ from the begin.

I hope that this can be of any help, and sorry for any grammatical error, English is not my native language.