Streaming Data about turbopfor-integer-compression HOT 3 CLOSED

You must write a function to decompress the data and fill a 128/256 integer buffer first, then successively read and process a single value from this buffer.
The overhead is negligible as the whole processing is done within the L1 cache.
There is no way to test this directly with icapp, but by choosing a large number (several MB) in the input you can avoid the cache effect.

from turbopfor-integer-compression.

What you mention for decompression is understood, and that would be ok.
But what if my application only receives/obtains integers one at a time. In this situation I can only pass an input integer array of length 1 to the chosen compression function, before I then need to used the compressed data. When a second integer arrives I want to then append/update the previous compressed data by making a another call to the compression function.
At the end of the second call, the compressed data would effectively be the same as if I had buffered the input integers and passed in length 2 array.
Is it possible to compress integers as they arrive and append or update the compressed data without needing to buffer the integers into an array of length N before each compression call ?
I'm working with Timeseries data and I thought Gorilla style algorithms did streaming compression ?

from turbopfor-integer-compression.

You must first store each integer into a buffer and call the turbopfor compress function when the buffer is full.
TurboPFor functions are extremely optimized for speed and use simd whenever possible,
This is why there is no streaming compression of single values.

Here a simple example how to implement and simulate streaming in turbopfor:

#define N 128

struct {
int in[N]; // input buffer
int n; // number of values in input buffer
FILE *f; // output file
} stream_t;

void screat(stream *s, char *name) { // create a turbopfor stream (here a file)
s->n = 0;
s->f = fopen(name, "w");
}

int sflush(stream_t s) {
char out[N5]; // temporary output buffer
if(s->n) {
p4ndenc128v32(s->in, s->n, out);
fwrite(out, s->n,1, s->f);
s->n = 0;
}
}

void saddvalue ( stream *s, int value) {
if(s->n == N) sflush(s);
s->in[s->n++] = value;
}

from turbopfor-integer-compression.