The gipfeli from google

gipfeli's Introduction

Gipfeli, a high-speed compression library.

Introduction
============

Gipfeli is a high-speed compression/decompression library
aiming at slightly higher compression ratios (around 30 %
less bytes produced for text) than other high-speed compression
libraries. On a single core of a Core i7 processor in
64-bit mode, gipfeli compresses at about 180 MB/s or more
and decompresses at about 300 MB/s, typically 5x faster than
zlib, but does not quite achieve its compression density.

Gipfeli is written in C++, with no assembler code.

Contact
=======

Gipfeli is distributed through GitHub. For the latest version,
a bug tracker, and other information, see

  https://github.com/google/gipfeli

gipfeli's People

Contributors

Stargazers

Watchers

gipfeli's Issues

C API plans

Snappy provides a (pure) C interface – do you plan to provide one as well, perhaps with a similar API?

Decompressed data does not match original input

I'm improving the random-data test case in Squash to try buffers of random data up to 8 MiB, and I've noticed that Gipfeli starts to fail around a couple hundred KiB due to the decompressed data not matching the input. An example of problematic input is at http://code.coeusgroup.com/gipfeli-bug (357 KiB)

To verify that the issue was in Gipfeli and not Squash, I threw together this test:

#include <stdio.h>
#include <assert.h>

#include "gipfeli.h"
#include "stubs-internal.h"

int main(int argc, char *argv[]) {
  FILE* input = fopen (argv[1], "rb");
  assert (input != NULL);

  int r = fseek (input, 0, SEEK_END);
  assert (r == 0);
  size_t uncompressed_size;
  {
    long p = ftell (input);
    assert (p > 0);
    uncompressed_size = (size_t) p;
  }
  fseek (input, SEEK_SET, 0);

  uint8_t* uncompressed = (uint8_t*) malloc (uncompressed_size);
  assert (uncompressed != NULL);
  fread (uncompressed, 1, uncompressed_size, input);
  fclose (input);
  input = NULL;

  util::compression::Compressor* compressor =
    util::compression::NewGipfeliCompressor();

  size_t compressed_size = compressor->MaxCompressedLength (uncompressed_size);
  uint8_t* compressed = (uint8_t*) malloc (compressed_size);

  {
    util::compression::UncheckedByteArraySink sink((char*) compressed);
    util::compression::ByteArraySource source((const char*) uncompressed, uncompressed_size);

    compressed_size = compressor->CompressStream (&source, &sink);
    fprintf (stderr, "Compressed %p[%zu] -> %p[%zu]\n", uncompressed, uncompressed_size, compressed, compressed_size);
  }

  {
    std::string compressed_str((const char*) compressed, compressed_size);

    size_t decompressed_size;
    if (!compressor->GetUncompressedLength (compressed_str, &decompressed_size)) {
      abort ();
    }

    assert (decompressed_size == uncompressed_size);
    uint8_t* decompressed = (uint8_t*) malloc (decompressed_size);
    assert (decompressed != NULL);

    util::compression::UncheckedByteArraySink sink((char*) decompressed);
    util::compression::ByteArraySource source((const char*) compressed, compressed_size);

    if (compressor->UncompressStream (&source, &sink)) {
      fprintf (stderr, "Decompressed %p[%zu] -> %p[%zu]\n", compressed, compressed_size, decompressed, decompressed_size);
    } else {
      fprintf (stderr, "Failed to decompress.\n");
      abort ();
    }

    int res = memcmp (uncompressed, decompressed, uncompressed_size);
    assert (res == 0);

    free (decompressed);
  }

  delete compressor;
  free (compressed);
  free (uncompressed);

  return 0;
}

Just feed it the name of the uncompressed file as the first argument, it should trigger the issue. It works for most files, but seems not to like larger blocks of random data.