Git Product home page Git Product logo

zopfli's Introduction

Zopfli Compression Algorithm is a compression library programmed in C to perform
very good, but slow, deflate or zlib compression.

The basic function to compress data is ZopfliCompress in zopfli.h. Use the
ZopfliOptions object to set parameters that affect the speed and compression.
Use the ZopfliInitOptions function to place the default values in the
ZopfliOptions first.

ZopfliCompress supports deflate, gzip and zlib output format with a parameter.
To support only one individual format, you can instead use ZopfliDeflate in
deflate.h, ZopfliZlibCompress in zlib_container.h or ZopfliGzipCompress in
gzip_container.h.

ZopfliDeflate creates a valid deflate stream in memory, see:
http://www.ietf.org/rfc/rfc1951.txt
ZopfliZlibCompress creates a valid zlib stream in memory, see:
http://www.ietf.org/rfc/rfc1950.txt
ZopfliGzipCompress creates a valid gzip stream in memory, see:
http://www.ietf.org/rfc/rfc1952.txt

This library can only compress, not decompress. Existing zlib or deflate
libraries can decompress the data.

zopfli_bin.c is separate from the library and contains an example program to
create very well compressed gzip files. Currently the makefile builds this
program with the library statically linked in.

The source code of Zopfli is under src/zopfli. Build instructions:

To build zopfli, compile all .c source files under src/zopfli to a single binary
with C, and link to the standard C math library, e.g.:
gcc src/zopfli/*.c -O2 -W -Wall -Wextra -Wno-unused-function -ansi -pedantic -lm -o zopfli

A makefile is provided as well, but only for linux. Use "make" to build the
binary, "make libzopfli" to build it as a shared library. For other platforms,
please use the build instructions above instead.

Zopfli Compression Algorithm was created by Lode Vandevenne and Jyrki
Alakuijala, based on an algorithm by Jyrki Alakuijala.

zopfli's People

Contributors

davidsansome avatar dwbuiten avatar enh-google avatar fhanau avatar hello71 avatar jayxon avatar jibsen avatar jthlim avatar khernyo avatar kornelski avatar lvandeve avatar megabyte avatar mrkrzych00 avatar rossy avatar scop avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zopfli's Issues

z_crc_t should be introduced

What steps will reproduce the problem?
1. In OSX, long is 8bytes-long
2. crc32's type is here unsigned long, which is problematic

Please provide any additional information below.

#if !defined(Z_U4) && !defined(Z_SOLO) && defined(STDC)
#  include <limits.h>
#  if (UINT_MAX == 0xffffffffUL)
#    define Z_U4 unsigned
#  elif (ULONG_MAX == 0xffffffffUL)
#    define Z_U4 unsigned long
#  elif (USHRT_MAX == 0xffffffffUL)
#    define Z_U4 unsigned short
#  endif
#endif

#ifdef Z_U4
   typedef Z_U4 z_crc_t;
#else
   typedef unsigned long z_crc_t;
#endif

Original issue reported on code.google.com by [email protected] on 11 Jul 2013 at 6:11

Zopfli.exe crashes if file does not fit in memory

What steps will reproduce the problem?
1. Run zopfli.exe on a 2gb+ file
2. Crash inside the CRC function as it tries to array-index into a null 
pointer, coming from LoadFile, which failed to allocate a 2gb memory region.

What is the expected output? What do you see instead?
Expect: Error message or whatever.
Crash.

What version of the product are you using? On what operating system?
Latest build. Windows x86.

Related to http://code.google.com/p/zopfli/issues/detail?id=15.

Original issue reported on code.google.com by [email protected] on 30 Oct 2014 at 8:53

ZopFli ZLIB mode sets the wrong fLevel byte in ZLIB header

What steps will reproduce the problem?
1. Run zopfli -zlib
2. Examine second byte of output
3. It's 01, indicating minimal compression

What is the expected output? What do you see instead?

Expect that maximal compression value is shown.

zlibcontainer.c line 56 is:

  unsigned flevel = 0;

should be:

   unsigned flevel = 3;

https://www.ietf.org/rfc/rfc1950.txt

FLEVEL (Compression level)
These flags are available for use by specific compression
methods.  The "deflate" method (CM = 8) sets these flags as
follows:

   0 - compressor used fastest algorithm
   1 - compressor used fast algorithm
   2 - compressor used default algorithm
   3 - compressor used maximum compression, slowest algorithm

Original issue reported on code.google.com by [email protected] on 13 Oct 2014 at 2:54

Add option to zopflipng to not change the colorspace

What steps will reproduce the problem?
1. Use zopflipng to compress up_black.png (attached) as up_black2.png:
    zopflipng up_black.png up_black2.png
2. You can use GIMP (Image > Image Properties) or Mac's file info screen ⌘I 
to see that the color space was changed from RGB to Grayscale.

Attached is another PNG, up_gray.png.  zopflipng does not change this image's 
color space:
    zopflipng up_gray.png up_gray2.png

The reason why changing the color space is not desired in my case is because I 
want to create a sprite using ImageMagick.  However, when the sprite tiles have 
different color spaces, then the colors are changed.

I have attached up.png, generated with ImageMagick 6.8.7-7 via:
    convert up_gray.png up_black.png +append up.png
I have also attached up2.png, generated with ImageMagick 6.8.7-7 via:
    convert up_gray2.png up_black2.png +append up2.png
(Note: ImageMagick 6.8.8-2 and ImageMagick compiled from source at revision 
14367 have the same behavior.)

Notice how the colors are different in up.png and up2.png.

What is the expected output? What do you see instead?
I would like an option to *not* change the color space of the input PNG.


What version of the product are you using? On what operating system?
I am using zopflipng compiled from source @c54dc204 on Mac OS 10.9.1 
'Mavericks'.


Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 14 Jan 2014 at 4:33

Attachments:

Issues building with MSVC

After 
http://code.google.com/p/zopfli/source/detail?r=337d27f25ef15a6cf34fef2acd0613fd
dc411cb1 it's no longer possible to build with MSVC. I use MSVC 2012 but my 
guess is the same happens with older MSVC versions.

I'm getting:
error C3861: 'getline': identifier not 
found   C:\Users\xmr\Desktop\xhmikosr-zopfli\src\zopflipng\zopflipng_bin.cc 367 1   
zopfli


I tried this:

 src/zopflipng/zopflipng_bin.cc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/zopflipng/zopflipng_bin.cc b/src/zopflipng/zopflipng_bin.cc
index d264c07..fd1a2b0 100644
--- a/src/zopflipng/zopflipng_bin.cc
+++ b/src/zopflipng/zopflipng_bin.cc
@@ -19,6 +19,7 @@

 #include <stdlib.h>
 #include <stdio.h>
+#include <string>

 #include "lodepng/lodepng.h"
 #include "zopflipng_lib.h"
@@ -364,7 +365,7 @@ int main(int argc, char *argv[]) {
           size_t nbytes = 2;
           char* text = (char*) malloc(nbytes);
           printf("File %s exists, overwrite? (y/N) ", out_filename.c_str());
-          int bytes_read = getline(&text, &nbytes, stdin);
+          int bytes_read = std::getline(&text, &nbytes, stdin);
           confirmed = bytes_read > 0 && (text[0] == 'y' || text[0] == 'Y');
         }
         if (confirmed) {


but it fails:

Error   1   error C2780: 'std::basic_istream<_Elem,_Traits> 
&std::getline(std::basic_istream<_Elem,_Traits> 
&,std::basic_string<_Elem,_Traits,_Alloc> &)' : expects 2 arguments - 3 
provided    C:\Users\xmr\Desktop\xhmikosr-zopfli\src\zopflipng\zopflipng_bin.cc 368
    1   zopfli
Error   2   error C2784: 'std::basic_istream<_Elem,_Traits> 
&std::getline(std::basic_istream<_Elem,_Traits> 
&,std::basic_string<_Elem,_Traits,_Alloc> &,const _Elem)' : could not deduce 
template argument for 'std::basic_istream<_Elem,_Traits> &' from 'char 
**' C:\Users\xmr\Desktop\xhmikosr-zopfli\src\zopflipng\zopflipng_bin.cc 368 1   zo
pfli
Error   3   error C2780: 'std::basic_istream<_Elem,_Traits> 
&std::getline(std::basic_istream<_Elem,_Traits> 
&&,std::basic_string<_Elem,_Traits,_Alloc> &)' : expects 2 arguments - 3 
provided    C:\Users\xmr\Desktop\xhmikosr-zopfli\src\zopflipng\zopflipng_bin.cc 368
    1   zopfli
Error   4   error C2784: 'std::basic_istream<_Elem,_Traits> 
&std::getline(std::basic_istream<_Elem,_Traits> 
&&,std::basic_string<_Elem,_Traits,_Alloc> &,const _Elem)' : could not deduce 
template argument for 'std::basic_istream<_Elem,_Traits> &&' from 'char 
**' C:\Users\xmr\Desktop\xhmikosr-zopfli\src\zopflipng\zopflipng_bin.cc 368 1   zo
pfli


Original issue reported on code.google.com by XhmikosR on 7 May 2013 at 4:09

Add ZopfliPNG option to keep timestamp of image file

This feature would be useful especially for older images where the timestamp 
matters. PNGOUT has this feature.

What version of the product are you using? On what operating system?
Version unknown, downloaded from 
http://encode.ru/threads/1717-ZopfliPNG?p=33092&viewfull=1#post33092
Windows

Original issue reported on code.google.com by [email protected] on 28 Jul 2014 at 9:20

compress large image

What steps will reproduce the problem?
1. If I compress large image (larger than 5000x5000), I have:
...
IDAT 8192 bytes 8 Kbytes
IDAT 8192 bytes 8 Kbytes
IDAT 6418 bytes 6 Kbytes
uncompress returned Z_BUF_ERROR

What is the expected output? What do you see instead?
Work

What version of the product are you using? On what operating system?
last version

Please provide any additional information below.
It's file larger than 5MB.

Original issue reported on code.google.com by [email protected] on 18 Nov 2014 at 12:33

Encoding error 82: color conversion to palette requested while a color isn't in palette

What steps will reproduce the problem?
zopflipng --lossy_transparent 002_Md.png 

What is the expected output? What do you see instead?
Encoding error 82: color conversion to palette requested while a color isn't in 
palette
Decoding error 82: color conversion to palette requested while a color isn't in 
palette
There was an error

What version of the product are you using? On what operating system?
Custom build from current repository on Mac OS X 10.6.8

002_Md.png file attached

Original issue reported on code.google.com by [email protected] on 28 Nov 2013 at 1:29

Attachments:

Support piping input

For example, it would be nice to be able to do:

    tar -cv foo | zopfli -c > foo.tar.gz

For this example to work, Zopfli would have to support direct input as well 
(not just file names), e.g.

    zopfli -c -s "$(cat foo.txt)"

Original issue reported on code.google.com by [email protected] on 29 Apr 2013 at 11:56

Option to limit number of iterations by CPU time [patch]

I've added --timelimit option to zopflipng that stops iteration after the given 
number of seconds has elapsed, as measured by clock() function that's supposed 
to measure CPU time used.

This allows placing hard upper limit on how long optimization will take, e.g. 
when server needs to respond with optimized image on time or when impatient 
user is waiting for large image to optimize.

The time limit has the same inaccuracy as number of iterations: 
ZopfliLZ77Optimal() may be called several times (e.g. when strategy==3), 
multiplying actual number of iterations done, and so time. I did not try to 
compensate for that.

Original issue reported on code.google.com by [email protected] on 16 Jul 2013 at 2:59

Attachments:

does not build correctly when using Fedora default compiler options

I am trying to get zopfli into Fedora.  Fedora uses the following compiler 
options by default and this is a mandatory requirement.  

https://fedoraproject.org/wiki/Packaging:Guidelines#Compiler_flags

Other distros have similar packaging guidelines as well.  Please fix


gcc blocksplitter.c cache.c deflate.c gzip_container.c hash.c katajainen.c 
lz77.c squeeze.c tree.c util.c zlib_container.c zopfli.c -O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector 
--param=ssp-buffer-size=4 -m64 -mtune=generic
/tmp/cchyEmFs.o: In function `CalculateEntropy':
/home/rahul/work/rpmbuild/SOURCES/zopfli-acc035/tree.c:79: undefined reference 
to `log'
/home/rahul/work/rpmbuild/SOURCES/zopfli-acc035/tree.c:85: undefined reference 
to `log'
/home/rahul/work/rpmbuild/SOURCES/zopfli-acc035/tree.c:79: undefined reference 
to `log'
collect2: error: ld returned 1 exit status

Original issue reported on code.google.com by [email protected] on 3 Mar 2013 at 8:15

ZopfliPNG does not show progress

What steps will reproduce the problem?
1. Run ZopfliPNG with correct parameters to optimize an image.

What is the expected output? What do you see instead?
It would be nice if some progress info was shown (e.g. similar to PNGOUT 
progress info).
Currently, just "Optimizing [filename]" is shown.

What version of the product are you using? On what operating system?
Unknown version, downloaded from 
http://encode.ru/threads/1717-ZopfliPNG?p=33092&viewfull=1#post33092
Windows

Please provide any additional information below.
Thanks for ZopfliPNG, I like it very much.

Original issue reported on code.google.com by [email protected] on 28 Jul 2014 at 9:14

ZopfliPNG core dump errors in batch job when compiled with GCC 4.8.2

What steps will reproduce the problem?
1. Under CygWin when ZopfliPNG is compiled with gcc 4.8.2 or mingw-gcc 4.8.2 
and executed as batch job with bash script I get core dump errors
2. When compiled with gcc 4.7.3 or mingw-gcc 4.7.3 I don't have these errors
3. Also, in make file, you use CFLAGS for CXX instead CXXFLAGS

What is the expected output? What do you see instead?
assertion "expected_data_size == 0 || testlength == expected_data_size" failed: 
file "src/zopfli/deflate.c", line 272, function: AddLZ77Data

What version of the product are you using? On what operating system?
Latest source c54dc204ef42

Please provide any additional information below.

assertion "expected_data_size == 0 || testlength == expected_data_size" failed: 
file "src/zopfli/deflate.c", line 272, function: AddLZ77Data
assertion "expected_data_size == 0 || testlength == expected_data_size" failed: 
file "src/zopfli/deflate.c", line 272, function: AddLZ77Data
Stack trace:
Frame     Function  Args
00288D98  610304E2 (000000CC, 0000EA60, 000000A4, 00288DE8)
00288EA8  610DA8AD (80052C08, 80052C18, 00000000, 80052D40)

Original issue reported on code.google.com by [email protected] on 14 Apr 2014 at 1:35

DeflOpt can further compress Zopfli output

What steps will reproduce the problem?
1. compress with Zopfli
2. run DeflOpt on its output... it's sometimes further compressed


What is the expected output?
Zopfli is supposed to produce the smallest Deflate stream

What do you see instead?
It's not the case since DeflOpt can still reduce Zopfli output

What version of the product are you using?
Current source

On what operating system?
Linux

On enwik8:

***                 DeflOpt V2.07                 ***
***       Built on Wed Sep  5 18:56:30 2007       ***
***  Copyright (C) 2003-2007 by Ben Jos Walbeehm  ***

"Z:/home/caveman/zopfli/enwik8.gz"
Number of bytes saved: 1,745 (34,995,756 --> 34,994,011) (13,960 bits)
File rewritten.

Number of files processed  :        1
Number of files rewritten  :        1
Total number of bytes saved:    1,745


DeflOpt is avaible here:
http://www.walbeehm.com/download/
It can be run on Linux using Wine.

Defluff is a similar tool:
http://encode.ru/threads/1214-defluff-a-deflate-huffman-optimizer

If you need a tool to compare Deflate streams, defdb can be downloaded from 
here:
http://encode.ru/threads/1428-defdb-a-tool-to-dump-the-deflate-stream-from-gz-an
d-png-files

Original issue reported on code.google.com by [email protected] on 1 Mar 2013 at 5:04

Capitalize Makefile file name

What steps will reproduce the problem?
1. Clone repo
2. ls
3. What the?

What is the expected output? What do you see instead?
Makefile
makefile

What version of the product are you using? On what operating system?
master

Please provide any additional information below.
Petty, but GNU likes it that way and these people in this Stackoverflow Q&A 
seem to prefer it that way too. Sticking to convention here would be good.

http://stackoverflow.com/questions/12669367/should-i-name-makefile-or-makefile

Original issue reported on code.google.com by [email protected] on 11 Jan 2014 at 7:45

In verbose mode, block size does not include Huffman tables and should be in bits

What steps will reproduce the problem?
1. use verbose mode -v
2. reported block size does not include Huffman tables, is reported in bytes 
(less accurate than bits).

Block size reported in verbose mode doesn't make sense, blocks are measured in 
bits and not bytes and from what I've seen compared to defdb they do not 
encompass the Huffman tables found at the beginning of type 2 blocks.

Original issue reported on code.google.com by [email protected] on 2 Mar 2013 at 6:53

Zopfli is non-deterministic

While playing with Zopfli v1.0.0 on various systems, I was surprised to learn 
that using the same input files with the same Zopfli settings on different 
systems may lead to slightly different results.

Here’s the results for an Ubuntu box:

    File `tmp/test-1.js.gz` created.
    Original:   83059 bytes.
    Compressed: 26957 bytes.

    File `tmp/test-3.js.deflate` created.
    Original:   86916 bytes.
    Compressed: 19156 bytes.

    File `tmp/test-4.js.deflate` created.
    Original:   86916 bytes.
    Compressed: 19038 bytes.

    File `tmp/test-5-a.js.gz` created.
    Original:   86916 bytes.
    Compressed: 19077 bytes.

    File `tmp/test-5-b.js.gz` created.
    Original:   83059 bytes.
    Compressed: 26957 bytes.

On OS X:

    File `tmp/test-1.js.gz` created.
    Original:   83059 bytes.
    Compressed: 27082 bytes.

    File `tmp/test-3.js.deflate` created.
    Original:   86916 bytes.
    Compressed: 19132 bytes.

    File `tmp/test-4.js.deflate` created.
    Original:   86916 bytes.
    Compressed: 19037 bytes.

    File `tmp/test-5-a.js.gz` created.
    Original:   86916 bytes.
    Compressed: 19068 bytes.

    File `tmp/test-5-b.js.gz` created.
    Original:   83059 bytes.
    Compressed: 27082 bytes.

This is using the official v1.0.0 release, on both systems. Zopfli seems to 
create even smaller files on the Ubuntu box for some reason.

Any idea why this is happening? Can something be done about this, or is it a 
WONTFIX?

Original issue reported on code.google.com by [email protected] on 25 May 2013 at 6:29

Don't assume -lm is required

What steps will reproduce the problem?
1. The current makefile has -lm hardcoded in, this should be added optionally

What is the expected output? What do you see instead?
Haiku and perhaps other Operating Systems, have their math library built in and 
thus don't require the use of -lm when building.

What version of the product are you using? On what operating system?
git master on Haiku.

Please provide any additional information below.
For now to build on Haiku I just use "sed -i 's/\-lm//' makefile"

Original issue reported on code.google.com by [email protected] on 16 Mar 2013 at 5:53

cppcheck scan

Using the latest cppcheck git,

[zopfli\katajainen.c:192]: (error) Uninitialized variable: leaves
[zopfli\katajainen.c:228]: (error) Uninitialized struct member: pool.nodes
[zopflipng\lodepng\lodepng_util.cpp:403]: (error) Uninitialized variable: ct
[zopflipng\lodepng\lodepng_util.cpp:403]: (error) Uninitialized variable: 
decoded

Original issue reported on code.google.com by XhmikosR on 28 Jun 2013 at 7:44

[CONTRIBUTION / PATCH] C API for ZopfliPNG library

I couldn't find any guidelines for contributing to Zopfli, so I have attached 
my 3 patches as here on the issue tracker.

Patch #1 introduces build support for building ZopfliPNG as a shared library.
Patch #2 fixes a permissions bug; source files shouldn't be +x.
Patch #3 introduces a C API to libzopflipng.

To elaborate a big, I believe having a C API is beneficial in that it allows 
the library to be used from a lot more places and languages (e.g. Go, other C 
projects, Ruby, etc.), much easier (without SWIG madness). It also creates a 
stable and define ABI for the library, something which C++ lacks.

All patches generated via git format-patch, and apply cleanly to master.

Comments welcome and thanks for your time!

Original issue reported on code.google.com by [email protected] on 7 Nov 2014 at 3:44

Attachments:

No warning given for unexpected command line arguments

zopfli -i1000000 some_file executes almost instantaneously (as opposed to the expectation of it taking a very long time due to the high number of iterations). It is expecting --i rather than -i, but it doesn't make this clear when invoked.

How to use zopfli DEFLATE in zip archive?

Hi!
I was wondering if anyone could lend any guidance for how one could use zopfli 
to (re)compress a zip file? I'd like to use it to repack jar files, but I don't 
know of any way to combine the zopfli raw output with the proper headers.

libarchive's bsdtar can produce zip files and use an external compression 
program, but it expects the program to be able to stream from stdin to stdout I 
think.

I tried creating a wrapper (lets call it zopflinout):

#!/bin/sh
zopfli /dev/stdin --deflate -c

... and using

bsdtar -cv --format zip --use-compress-program zopflinout foo.zip *

... but this quickly segfaults and tells me 'Can't write to filter.'


Also (not really related, but...) here is a patch for the makefile to:

- Allow for environmental overrides for compiler variables.
- Build a static library.
- Build a shared library on Darwin.
- Use LDFLAGS.
- Actually use CXXFLAGS when building zopflipng (you have a variable for it but 
don't use it in the target).

... if you want?

Original issue reported on code.google.com by [email protected] on 1 Nov 2013 at 9:26

Attachments:

Tag a release every now and then

Please consider tagging a release every now and then as per http://semver.org/.

This would be useful for various obvious reasons, but I’m asking this 
specifically because it would allow me to create a 
[Homebrew](http://mxcl.github.io/homebrew/) formula so people using OS X can 
install Zopfli even more easily. (All they’d have to do is enter `brew 
install zopfli` in their shell.)

How about tagging the current release as v0.1.0 and then go from there?

Note: you may also want to display the version number in the help screen 
(`zopfli -h`).

Original issue reported on code.google.com by [email protected] on 19 Apr 2013 at 7:53

libzopfli shouldn't allows options.numiterations = 0

I had a bug in a function calculating iterations based on file size, which 
caused this.

src/zopfli/deflate.c:272: AddLZ77Data: Assertion `expected_data_size == 0 || 
testlength == expected_data_size' failed

It's not necessarily a bug, but maybe libzopfli should protect against it still.

Original issue reported on code.google.com by pdknsk on 4 Jun 2014 at 3:20

Is the deflate based output accetable for browsers that don't support gzip?

Since I don't see it on the site, and I know that gzip uses a different crc32 
from the standard "deflate" algorithm, and since I want to use it for 
precompressed data, I was wondering if the deflate based output is similar 
enough to the zip based output that I could use it on the site.

What I'm doing is quite simple, I have static resources that are precompressed, 
and I store the filename.gz, filename.zip, and filename. With an nginix 
extension, I'm able to tell it to use the precompressed ones. Since zopfli is 
_better_ than even advzip(which uses 7zip's deflate) by ~1-2%, I am planning on 
using it. Advzip is able to recompress gzip/zip streams to see if it's smaller 
and still make it work the same to the decompressor. So does zopfli's deflate 
output work for browsers that don't suppot gzip compression?

I know this is very runon, and I couldn't think of a better way to word this, 
I'm sorry for it being very run-ony but it looks to be a very intersting 
project, and I'd love to start using it but I need to know that one thing. 
Finally as an extra question, do you guys have plans to recompress png files? 
Since it's just deflate, ala advpng with your tool? Or is that out of the 
question currently, because that would be an interesting extra application of 
the technology. Anyway thanks for making the tool open source, and I await the 
answers.

Original issue reported on code.google.com by [email protected] on 1 Mar 2013 at 1:44

Parallel zopfli

Gzip has a parallel implementation called 'pigz'.
Since zopfli needs a lot more processing power, it would be great if it could 
be parallellized (if the algorithm allows it, of course).

Original issue reported on code.google.com by [email protected] on 1 Mar 2013 at 9:05

zopflipng 32 bit crash

When zopflipng is compiled 32 bit no optimization with gcc 3.4.6 20060404 (Red 
Hat 3.4.6-11) or 4.4.6 20120305 (Red Hat 4.4.6-4), it cores on the attached 
test file, problem.png.  I am running off of the latest head, 
c54dc204ef4278f949a965dc90e693799b6aae41.

zopflipng generates the following error message:
zopflipng: src/zopfli/deflate.c:272: AddLZ77Data: Assertion `expected_data_size 
== 0 || testlength == expected_data_size' failed.
Aborted (core dumped)

The error does not appear in 32 bit mode when zopflipng is compiled with -O2 
(as is its default in its makefile) or at all when zopflipng is compiled 64 
bit.  That is, the error doesn't show up in 64 bit regardless of the 
optimization level.

I have done some debugging and the error appears to originate with lodepng 
stomping on the stack of the CustomPNGDeflate function during a call to 
lodepng_realloc (lodepng.cpp:68).

The memory I noticed being stomped is options.numiterations in 
CustomPNGDeflate.  To see lodepng stomping the stack, compile a 32 bit 
unoptimized zopflipng, fire up gdb and set a breakpoint at zopflipng_lib.cc:50. 
 Grab the address of options.numiterations with 'p &options.numiterations' and 
set a hardware watchpoint on the memory 'watch *(int *)0xffffcbb8'.  Then let 
the program run until something writes to that memory (press 'c').  You should 
see some output like the following:
Hardware watchpoint 3: *(int *)0xffffcbb8

Old value = 15
New value = 80
_int_realloc (av=0xf7e943a0, oldp=0x8081a20, oldsize=80, nb=1512)
    at malloc.c:5253
5253      if (__builtin_expect (oldp->size <= 2 * SIZE_SZ, 0)

Issuing the backtrace command ('bt') will show you the call stack:
(gdb) bt
#0  _int_realloc (av=0xf7e943a0, oldp=0x8081a20, oldsize=80, nb=1512)
    at malloc.c:5253
#1  0xf7d761ac in __libc_realloc (oldmem=0x8081a28, bytes=1506)
    at malloc.c:3826
#2  0x080510a3 in lodepng_realloc (ptr=0x8081a28, new_size=1506)
    at src/zopflipng/lodepng/lodepng.cpp:68
#3  0x08055b8d in lodepng_chunk_create (out=0xffffcdc4, outlength=0xffffcdc8, 
    length=1418, type=0x8071fc0 "IDAT", 
    data=0x8081a78 "x\001\354Wan\263:\020\264\201\003\060\220\003\340$\a0Q\016") at src/zopflipng/lodepng/lodepng.cpp:2457
#4  0x0805c32f in addChunk (out=0xffffcdc4, chunkName=0x8071fc0 "IDAT", 
    data=0x8081a78 "x\001\354Wan\263:\020\264\201\003\060\220\003\340$\a0Q\016", length=1418) at src/zopflipng/lodepng/lodepng.cpp:4879
#5  0x0805c7ab in addChunk_IDAT (out=0xffffcdc4, data=0x808c648 "", 
    datasize=40385, zlibsettings=0xffffcff8)
    at src/zopflipng/lodepng/lodepng.cpp:4989
#6  0x0805ed56 in lodepng_encode (out=0xffffce24, outsize=0xffffce20, 
    image=0xf7c5f008 "\377\377\377", w=392, h=205, state=0xffffcfd0)
    at src/zopflipng/lodepng/lodepng.cpp:5771
#7  0x0805ffdf in lodepng::encode (out=std::vector of length 0, capacity 0, 
    in=0xf7c5f008 "\377\377\377", w=392, h=205, state=...)
    at src/zopflipng/lodepng/lodepng.cpp:6214
#8  0x080600c9 in lodepng::encode (out=std::vector of length 0, capacity 0,
---Type <return> to continue, or q <return> to quit---

If you continue to run the program the watchpoint will trigger more as the 
stack of CustomPNGDeflate is further abused.

If I come up with a patch I'll post it.

Original issue reported on code.google.com by [email protected] on 31 Jul 2013 at 10:57

Attachments:

zopflipng resets the resolution / image DPI

What steps will reproduce the problem?
1. Download the attached PNG.
2. On a terminal, execute:  zopflipng preview.png preview.min.png
3. Use an image viewer application to determine the image DPI of `preview.png' 
versus `preview.min.png'.  For example, open the images in Mac's Preview.app 
and press ⌘I to show the inspector.

What is the expected output? What do you see instead?
The image DPI changed from 144 DPI to 72 DPI.  I expected the image DPI to 
remain unchanged.

What version of the product are you using? On what operating system?
I am using zopflipng compiled from source @c54dc204 on Mac OS 10.10 'Yosemite'.

Please provide any additional information below.
Not changing the resolution of the PNG is important for creating assets for 
high-DPI displays such as a Retina display.

Original issue reported on code.google.com by [email protected] on 29 Oct 2014 at 11:12

Attachments:

segmentation fault with big Files


haw@meikes:~/ > ls home-arbeit.tar -lah
-rw-r--r-- 1 haw haw 3.5G Apr 16 14:52 home-arbeit.tar

haw@meikes:~/ > zopfli -v home-arbeit.tar
Saving to: home-arbeit.tar.gz
zsh: segmentation fault  zopfli -v home-arbeit.tar

Debian 6.0 with own Kernel
haw@meikes:~/ > uname -a
Linux meikes.wurpts-online.de 3.7.1amd64 #2 SMP Fri Jan 4 13:35:13 CET 2013 
x86_64 GNU/Linux

Need me Information?

Original issue reported on code.google.com by [email protected] on 16 Apr 2013 at 1:01

Resulting in larger files

I am making use of a project called AdvanceCOMP, in it's compression options 
allows zopfli (-4, or --shrink-insane). While going through PNG's for 
Torchlight there are dozens of png's that end up larger. I've attached the 
gloves so you can review and test it yourself as appropriate.

advpng.exe -z -4 */*.png

43653       43653 100% wardrobe/dragon_gloves.png (Bigger 45713)
89125       89125 100% wardrobe/heavyleather_boots.png (Bigger 96720)
11412       10817  94% wardrobe/heavyleather_boots_alt01.png
90564       90564 100% wardrobe/heavyleather_chest.png (Bigger 98420)
89328       89328 100% wardrobe/heavyleather_gloves.png (Bigger 97619)

At worst the compression should max at the same size; This means the problem's 
root comes from some sections of the data are compressing better while others 
end up being left as uncompressed. I noted this in some of my own compression 
experiments years ago.


Viable solution:
 With sections that have no compression on them (causing expansion) they should instead compress and find an identical matching length. This is most likely part of another match. If this at the beginning or end of another compressed section, that section will truncate to allow the uncompressed section to compress at a 1:1 rate (so long as the other match remains long enough to retain compression).

I am not aware of the full details of zlib compression, so there needs to be an 
additional rule.

If in the instance that a match is found in the middle of another match, then 
it should split the match into two matches avoiding the middle (on purpose) to 
give the inner match to the uncompressed section. This should only happen in 
the case that these 3 matches takes less space than 1 match & 1 non-match.

How much more complexity this will give I'm not sure, nor how much extra time 
it will take.

Original issue reported on code.google.com by [email protected] on 2 Nov 2013 at 9:00

Attachments:

ZopfliPNG does not show version number

What steps will reproduce the problem?
1. Run ZopfliPNG.

What is the expected output? What do you see instead?
Product version should be shown.
Product version is not shown.

What version of the product are you using? On what operating system?
Downloaded from 
http://encode.ru/threads/1717-ZopfliPNG?p=33092&viewfull=1#post33092
Windows

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 28 Jul 2014 at 8:56

Authoritative version source

There are currently two places to get version information in the zopfli 
baseline:
1. The latest annotated tag via 'git describe --abbrev=0' (currently 1.0.0)
2. The zopfli shared library build target's version number (currently 1.0.1)

It seems like the shared library version number is the most up to date but it's 
not particularly clear where to expect to find the current version.  Could we 
get a VERSION file (and cat it into the shared library filename)?

Original issue reported on code.google.com by [email protected] on 25 Jul 2013 at 5:24

Mark Adler's reaction

Mark Adler who wrote Zlib with Mark Nelson just expressed its feelings about 
Zopfli on the [Zlib-devel] mailing list:
"That's cool, but it seems like an awful lot of effort for a small improvement. 
 Perhaps it's time to add a better compression method to HTTP's 
accept-encoding."

I can only agree, Deflate was written around 1994 (when operating systems like 
Windows 95 required 24MB of main memory and Unicode was a toddler), its 
definitely time to move on!
Even ZIP archives can handle better compression algorithms (Deflate64, bzip2, 
PPMD, LZMA) since years.
WebP has a search window that is way larger than Deflate (32k), HTML5 only 
recommends UTF-8 encoding and Deflate is still single byte oriented... imagine 
how poorly it performs on pages holding text written in Devanāgarī (3 bytes 
per Unicode code point in UTF-8, main script used by nearly half of the 
population in India).

Original issue reported on code.google.com by [email protected] on 3 Mar 2013 at 7:13

alloc memory by caller

I'm trying to make use of zopfli library in advanceCOMP, and I found that I 
can't alloc memory by caller. If I do so, zopfli segfaults in 
ZOPFLI_APPEND_DATA(0, out, outsize)

I wonder if you can change it to make use the memory allocated by caller 
directly?

Original issue reported on code.google.com by [email protected] on 18 Mar 2013 at 8:37

Same input and options should be same result.

Same input and options should be same result for algorithm's reliability.
But squeeze.c uses random function ``Ran()'', so sometimes gets another result.
I suggest reset variables m_w and m_z at beginning of Deflate() in deflate.c.

Original issue reported on code.google.com by [email protected] on 4 Mar 2013 at 12:30

zopflipng crash

What steps will reproduce the problem?
1. Download the file
2. Run the command
3. Watch it crash

What is the expected output? What do you see instead?
I expect it not to crash.

What version of the product are you using? On what operating system?
Latest git on windows(cygwin).

Please provide any additional information below.
I have attached two files which cause the issue. The actual errors are:

for wisp.png:

$ zopflipng --splitting=3 --filters=01234mepb wisp.png wisp.png
Optimizing wisp.png
assertion "sublen[*length] == s->lmc->dist[lmcpos]" failed: file 
"src/zopfli/lz77.c", line 187, function: TryGetFromLongestMatchCache
Aborted (core dumped)

for wisp2.png:

$ zopflipng --splitting=3 --filters=01234mepb wisp.png wisp.png 
Optimizing wisp.png
Filter strategy zero: 2193428 bytes
Filter strategy one: 2265039 bytes
Filter strategy two: 2367388 bytes
assertion "s->lmc->length[lmcpos] == 1 && s->lmc->dist[lmcpos] == 0" failed: 
file "src/zopfli/lz77.c", line 220, function: StoreInLongestMatchCache
Aborted (core dumped)

Original issue reported on code.google.com by [email protected] on 22 Jun 2013 at 6:27

Attachments:

SIGBUS error on mips64el architecture

I'm having a SIGBUS error with zopfli-1.0.0 on OpenBSD/loongson (which is a 
little-endian MIPS64 architecture, with strict alignment).

The following hello.c file:
#include <stdio.h>

int main(void)
{
    printf("hello, world\n");

    return 0;
}

triggers a SIGBUS error in zopfli:
$ zopfli hello.c
Starting program: /usr/local/bin/zopfli hello.c

Program received signal SIGBUS, Bus error.
0x0000000006987cb8 in ZopfliFindLongestMatch (s=0x7ffdca40, h=0x7ffdc988, 
    array=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", pos=40, size=78, limit=38, 
    sublen=0x7ffdc780, distance=0x7ffdc9d4, length=0x7ffdc9d6)
    at src/zopfli/lz77.c:126
126     while (scan < safe_end && *((size_t*)scan) == *((size_t*)match)) {
#0  0x0000000006987cb8 in ZopfliFindLongestMatch (s=0x7ffdca40, h=0x7ffdc988, 
    array=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", pos=40, size=78, limit=38, 
    sublen=0x7ffdc780, distance=0x7ffdc9d4, length=0x7ffdc9d6)
    at src/zopfli/lz77.c:126
    same0 = Variable "same0" is not available.
#0  0x0000000006987cb8 in ZopfliFindLongestMatch (s=0x7ffdca40, h=0x7ffdc988, 
    array=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", pos=40, size=78, limit=38, 
    sublen=0x7ffdc780, distance=0x7ffdc9d4, length=0x7ffdc9d6)
    at src/zopfli/lz77.c:126
#1  0x0000000006988490 in ZopfliLZ77Greedy (s=0x7ffdca40, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", instart=1, inend=78, store=0x7ffdca60)
    at src/zopfli/lz77.c:399
#2  0x0000000006982d78 in ZopfliBlockSplit (options=0x7ffdcd90, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", instart=0, inend=78, maxblocks=15, 
    splitpoints=0x7ffdcb00, npoints=0x7ffdcaf8)
    at src/zopfli/blocksplitter.c:310
#3  0x00000000069855f4 in DeflateSplittingFirst (options=0x7ffdcd90, btype=2, 
    final=1, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", instart=0, inend=78, bp=0x7ffdccd7 "", 
    out=0x7ffdcdb0, outsize=0x7ffdcda8) at src/zopfli/deflate.c:569
#4  0x0000000006985904 in ZopfliDeflatePart (options=0x7ffdcd90, btype=2, 
    final=1, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", instart=0, inend=78, bp=0x7ffdccd7 "", 
    out=0x7ffdcdb0, outsize=0x7ffdcda8) at src/zopfli/deflate.c:668
#5  0x0000000006985adc in ZopfliDeflate (options=0x7ffdcd90, btype=2, final=1, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", insize=78, bp=0x7ffdccd7 "", out=0x7ffdcdb0, 
    outsize=0x7ffdcda8) at src/zopfli/deflate.c:687
#6  0x0000000006985ef8 in ZopfliGzipCompress (options=0x7ffdcd90, 
    in=0x5336c200 "#include <stdio.h>\n\nint main(void)\n{\n\tprintf(\"hello, world\\n\");\n\n\treturn 0;\n}\n", insize=78, out=0x7ffdcdb0, outsize=0x7ffdcda8)
    at src/zopfli/gzip_container.c:96
#7  0x000000000698b52c in ZopfliCompress (options=Variable "options" is not 
available.
) at src/zopfli/zopfli_lib.c:33
#8  0x000000000698b174 in main (argc=2, argv=0x7ffdce60)
    at src/zopfli/zopfli_bin.c:96

Original issue reported on code.google.com by [email protected] on 22 Jun 2013 at 11:37

ZopfliPNG cannot read filenames with special characters

What steps will reproduce the problem?
1. Run ZopfliPNG with input filename that contains non-English chars.

What is the expected output? What do you see instead?
The image should be read correctly, no matter of what chars are used in its 
filename. (The same applies to writing output.)
Decoding error 48 is shown instead.

What version of the product are you using? On what operating system?
Unknown, downloaded from 
http://encode.ru/threads/1717-ZopfliPNG?p=33092&viewfull=1#post33092
Windows


Please provide any additional information below.

As an example, the attached screenshot shows:
1) an error with a filename containing all capital letters from the French 
alphabet that are not in the English alphabet,
2) comparison with PNGOUT (although incorrect chars are shown when processing, 
the input is read and the output is saved with a correct name),
3) correct filenames in the title of the window.

There are 3 levels of failure, depending on chars used:
1) The file is read by both ZopfliPNG and PNGOUT. Example: Filename contains 
"u" with umlaut.
2) The file is not read by ZopfliPNG but it is read by PNGOUT. Example: The 
case described with the attached screenshot.
3) The file is read neither by ZopfliPNG nor by PNGOUT. Example: Filename 
contains all Czech letters (https://en.wikipedia.org/wiki/Czech_alphabet).

Original issue reported on code.google.com by [email protected] on 28 Jul 2014 at 10:37

Attachments:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.