google / pik Goto Github PK

A new lossy/lossless image format for photos and the internet

License: MIT License

Makefile 0.31% C++ 98.50% C 0.53% CMake 0.65%

pik's Introduction

Note There is currently no development going on here. Parts of this project have been used in JPEG XL, which is under active development.

PIK

PIK is a well-rounded image format for photos and the internet.

Project Goals

PIK is a modernized variant of JPEG with similar goals: efficient storage and delivery of photos and web images. It is designed from the ground up for high quality and fast decoding.

Features enabling high quality (perceptually lossless):

built-in support for psychovisual modeling via adaptive quantization and XYB color space
powerful coding tools: 4x4..32x32 DCT, AC/DC predictors, chroma from luma, nonlinear loop filter, enhanced DC precision
full-precision (32-bit float) processing, plus support for wide gamut and high dynamic range

In addition to fully- and perceptually lossless encodings, PIK achieves a good balance of quality/size/speed across a wide range of bitrates (0.5 - 3 bpp). PIK enables automated/unsupervised compression because it guarantees that the target quality is maintained over the entire image. It prioritizes authenticity, a faithful representation of the original, over aesthetics achievable by by hallucinating details or 'enhancing' (e.g. sharpening/saturating) the input.

Features enabling fast decoding (> 1 GB/s multithreaded):

Parallel processing of large images
SIMD/GPU-friendly, benefits from SSE4 or AVX2
Cache-friendly layout
Fast and effective entropy coding: context modeling with clustering, rANS

Other features:

Alpha, animations, color management, 8-32 bits per channel
Flexible progressive scans (quality/resolution/viewport)
Graceful upgrade path for existing JPEGs: can reduce their size by 22% without loss; or low generation loss because both use 8x8 DCT
New, compact and extensible container format
Royalty-free commitment

PIK's responsive mode encoder supports passes equivalent to lowering the resolution by 4x or 8x. The format supports more flexible passes, with any level of detail from equivalent to 8x downsampling to full resolution. The amount of detail in a pass does not need to be uniform: areas of the image can be sent with higher detail. The impact of responsive mode on encoded image size is low, averaging to about 2% for a 3 pass responsive image (8x, 4x, full resolution). In such a configuration, first two passes take on average 20% of image size each.

Size comparisons

PIK achieves perceptually lossless encodings at about 40% of the JPEG bitrate.
PIK can also store fully lossless at about 75% of 8-bit PNG size (or even 60% for 16-bit).
PIK's perceptually lossless encodings are about 10% of the size of a fully lossless representation.

Build instructions

The software currently requires an AVX2 and FMA capable CPU, e.g. Haswell. Building currently requires clang 6 or newer.

In order to build, the following instructions can be used:

git submodule update --init
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j8

This creates cpik and dpik binaries in build/.

Usage

Basic usage is as follows:

cpik [--distance <d>] input.png output.pik

The optional --distance command line argument to cpik is a Butteraugli distance (see http://github.com/google/butteraugli), which indicates the largest acceptable error. Larger values lead to smaller files and lower quality. The default value of 1.0 should yield a perceptually lossless result.

Note that the bitstream is still under development and not yet frozen.

Related projects

JPEG XL (reference implementation of image format)
Butteraugli (HVS-aware image differences)
Guetzli (JPEG encoder with denser packing)

pik's People

Contributors

Stargazers

Watchers

pik's Issues

Having trouble building on macOS High Sierra (10.13.3)

When I attempt to compile using make -j8 in macOS High Sierra, I receive the following error:

image_io.cc:1031:31: error: assigning to 'boolean' from incompatible type 'bool'
  cinfo.do_fancy_upsampling = true;
                              ^~~~

Full compiler output: compiler_output.log

Is there anything I'm doing wrong, or is this a legitimate error? Here's my system info (using a Kaby Lake Intel, currently):

Hardware Overview:
Model Identifier: MacBookPro14,2
Processor Name: Intel Core i5
Processor Speed: 3.1 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Memory: 16 GB
Boot ROM Version: MBP142.0169.B00
SMC Version (system): 2.44f1

Cpik refuses to encode png file

The attached image fails to encode:

bin/cpik google-chrome.png 3.0 chrome.pik
Not a JPEG file: starts with 0x89 0x50

This is technically correct, it is a png file, and the first two bytes are the correct bytes for the png header. The file does have an alpha channel. If this is not supported, the tool should report that, rather than complaining about jpeg files, as the usage indicates that the input should be a png file.

Python binding

any plans to release Python binding for pik ?

Status of bitstream

I wonder if the bitstream for Pik is frozen, or still subject to change?
Are you close to doing a minor release?

Thank you.

Don't let Google hurt mankind

@jan-wassenberg
@deymo
@sboukortt
The ecological and human impact of dropping support for JPEG XL is too high to tolerate.
Use your relationships/scholar authority to make chromium change its mind about the removal.
The manager behind this decision must have undiagnosed early dementia yet I'm sure there's not just one brain behind this decision and that you can make them decide otherwise.
https://bugs.chromium.org/p/chromium/issues/detail?id=1178058#c84

It is likely their mob politics didn't realize that webp2 is abandoned https://chromium.googlesource.com/codecs/libwebp2/+/1251ca748c17278961c0d0059b744595b35a4943%5E%21/
that AVIF is massively inferior as it is slow to encode/decode (cf Facebook engineers comments) and that is lacks the disruptive feature of losslessly recompressing existing JPEGs without generation loss.
And they probably don't realize that JPEG XL is originating from Google, cf PIK.
Once they'll learn this info they should be expected to switch that little boolean in their head.

very visible chroma macroblock artifacts on some images (probably bug)

I tried using cpik with target Butteraugli score 2.0 on some images. On some of them, the results had very visible chroma macroblock artifacts. For example this original image:

became this after cpik/dpik:

I assume this is a bug in either the encoder or decoder.

compilation issue with mingw-gcc900-20190108 under Windows 10 {PIK(ver. 10012019)}

Hi
I've used the commands in g++:
-fPIC -ftree-vectorize -g0 -O3 -D__x86_64__ -DBITS_IN_JSAMPLE=8 -DENABLE_JPEG=0 -DPIK_ENABLE_ASSERT=1 -DPIK_CRASH_ON_ERROR=1 -DPROFILER_ENABLED=1 -DNSTDMUTEX_RECURSION_CHECKS -DNPIK_CMS_VERBOSE -DNPIK_PNG_VERBOSE -DNPIK_EXT_VERBOSE -DCMS_IS_WINDOWS_ -DCMS_RELY_ON_WINDOWS_STATIC_MUTEX_INIT -D__LITTLE_ENDIAN__ -DSIMD_ENABLE=0
and has messages:
ac_predictions.cc: In instantiation of 'void pik::{anonymous}::AddBlockExcept0HVDTo(const float*, float*) [with bool add = false]':

ac_predictions.cc:387:38:   required from 'void pik::{anonymous}::UpSample4x4BlurDCT(const pik::Rect&, const ImageF&, const Ub4Kernel&, const pik::AcStrategyImage&, const pik::Rect&, pik::ImageF*) [with bool add = false; pik::ImageF = pik::Image<float>; pik::{anonymous}::Ub4Kernel = std::array<std::array<float, 4>, 3>]'
ac_predictions.cc:645:62:   required from here
ac_predictions.cc:43:12: error: assignment of read-only location '* block'
   43 |   block[0] = 0.0f;
ac_predictions.cc:44:12: error: assignment of read-only location '*(block + 4)'
   44 |   block[1] = 0.0f;
ac_predictions.cc:45:12: error: assignment of read-only location '*(block + ((sizetype)(((long long unsigned int)N) * 4)))'
   45 |   block[N] = 0.0f;
ac_predictions.cc:46:16: error: assignment of read-only location '*(block + ((sizetype)(((long long unsigned int)(((int)N) + 1)) * 4)))'
   46 |   block[N + 1] = 0.0f;

compressed_image.cc: In function 'pik::Image3F pik::ReconOpsinImage(const pik::PassHeader&, const pik::GroupHeader&, const pik::Quantizer&, const pik::Rect&, pik::DecCache*, pik::PassDecCache*, pik::PikInfo*)':
compressed_image.cc:959:47: error: unable to deduce 'std::initializer_list<auto>&&' from '{0, (((long long unsigned int)xsize_blocks) + 1)}'
  959 |         for (size_t x : {0UL, xsize_blocks + 1}) {
      |                                               ^
compressed_image.cc:959:47: note:   deduced conflicting types for parameter 'auto' ('long unsigned int' and 'long long unsigned int')
compressed_image.cc:972:47: error: unable to deduce 'std::initializer_list<auto>&&' from '{0, (((long long unsigned int)ysize_blocks) + 1)}'
  972 |         for (size_t y : {0UL, ysize_blocks + 1}) {
      |                                               ^
compressed_image.cc:972:47: note:   deduced conflicting types for parameter 'auto' ('long unsigned int' and 'long long unsigned int')
compressed_image.cc:989:47: error: unable to deduce 'std::initializer_list<auto>&&' from '{0, (((long long unsigned int)xsize_blocks) + 1)}'
  989 |         for (size_t x : {0UL, xsize_blocks + 1}) {
      |                                               ^
compressed_image.cc:989:47: note:   deduced conflicting types for parameter 'auto' ('long unsigned int' and 'long long unsigned int')
compressed_image.cc:997:47: error: unable to deduce 'std::initializer_list<auto>&&' from '{0, (((long long unsigned int)ysize_blocks) + 1)}'
  997 |         for (size_t y : {0UL, ysize_blocks + 1}) {

os_specific.cc: In function 'pik::Status pik::RunCommand(const std::vector<std::__cxx11::basic_string<char> >&)':
os_specific.cc:255:65: error: invalid use of member 'void std::__cxx11::basic_ostringstream<_CharT, _Traits, _Alloc>::str(const __string_type&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_ostringstream<_CharT, _Traits, _Alloc>::__string_type = std::__cxx11::basic_string<char>]' (did you forget the '&' ?)
  255 |   printf(stderr, "Warning: Using system() on string: %s\n", cmd.str.c_str());
      |                                                             ~~~~^~~
os_specific.cc:256:24: error: invalid use of member 'void std::__cxx11::basic_ostringstream<_CharT, _Traits, _Alloc>::str(const __string_type&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_ostringstream<_CharT, _Traits, _Alloc>::__string_type = std::__cxx11::basic_string<char>]' (did you forget the '&' ?)
  256 |   int ret = system(cmd.str.c_str());

PS. I understand that JPEG and PPM files aren't imported even with ENABLE_JPEG. The implementation is in the process of creation.

Unfortunate project name

Though quite amusing for Danes and Dutch, the name of this project is quite unfortunate when translated:

Danish: https://translate.google.com/#da/en/pik
Dutch: https://translate.google.com/#nl/en/pik

PIK not competitive at low bits-per-pixel bitrates

Spun off from #33 to separate different points out as requested.

we are still working on PIK and extending its usability in the low bit rate end. We don't yet know if we can make PIK competitive in the low bit rate.

This is actually a major concern; depending on use cases, if low bitrate is desired, one would logically choose WebP that performs very well at those bitrates; at high bitrates, one would use either high-q jpeg (if file size is still a concern) or a lossless format. This leaves very little for PIK to fill in a void -- a new image format has to not just improve, it has to give a convincing reason to choose it in a particular situation.

FMA/AVX2 fallback testfile

Hi,
I'm working on a FMA/AVX2 fallback for older CPUs. #8 (not final)
I want showing you my first result of encoding and decoding, processed by an i7 2600k. The cpu have only AVX but no AVX2 or FMA.

Encoded file: (please remove the ".txt" for encoding)
1.pik.txt

What do you thinking?

State of the art entropy coding

https://github.com/Cyan4973/FiniteStateEntropy
This is used by zstd the succcesor of gzip and deflate, maybe you could benefit from it too for JPEG XL?

Questions regarding Browser support

Question: when will PIK be supported in Chrome and Firefox? Just wondering
P.S. would be sweet if there is Pillow support for Python.

Add an .editorconfig file to the pik repo

Adding an .editorconfig file to this repository has the following advantages:

keeps the code format as defined - regardless who is contributing
reduces PR review complexity (code formatting changes are gone)
works for common IDE's

heap-buffer-overflow in pik::ReplicateValue

Hi.

I found a heap-buffer-overflow bug in pik.

Please confirm.

Thanks.

Summary: heap-buffer-overflow
Browser/OS: Ubuntu 17.04 64bit
Steps to reproduce:
1.Download the .POC files.
2.Execute the following command
: ./dpik $PoC /dev/null
PoC download : PoC or PoC2

=================================================================
==18642==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625000002100 at pc 0x0000005cff46 bp 0x7fff7fe17d30 sp 0x7fff7fe17d28
WRITE of size 4 at 0x625000002100 thread T0
#0 0x5cff45 in pik::ReplicateValue(pik::HuffmanCode*, int, int, pik::HuffmanCode) /root/gwanyeong/pik/huffman_decode.cc:45:16
#1 0x5cff45 in pik::BuildHuffmanTable(pik::HuffmanCode*, int, unsigned char const*, int, unsigned short*) /root/gwanyeong/pik/huffman_decod e.cc:166
#2 0x5d3409 in pik::HuffmanDecodingData::ReadFromBitStream(pik::BitReader*) /root/gwanyeong/pik/huffman_decode.cc:345:18
#3 0x660cce in pik::DecodePlane(unsigned char const*, unsigned long, int, int, pik::Image<int>*) /root/gwanyeong/pik/opsin_codec.cc:992:8
#4 0x5811c6 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const &, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:392:10
#5 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image 3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
#6 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
#7 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
#8 0x7f20fde843f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
#9 0x41b759 in _start (/root/gwanyeong/pik/bin/dpik+0x41b759)

0x625000002100 is located 0 bytes to the right of 8192-byte region [0x625000000100,0x625000002100)
allocated by thread T0 here:
#0 0x50a1b0 in operator new(unsigned long) (/root/gwanyeong/pik/bin/dpik+0x50a1b0)
#1 0x660c45 in __gnu_cxx::new_allocator<pik::HuffmanCode>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/ ../../../../include/c++/6.3.0/ext/new_allocator.h:104:27
#2 0x660c45 in std::allocator_traits<std::allocator<pik::HuffmanCode> >::allocate(std::allocator<pik::HuffmanCode>&, unsigned long) /usr/bi n/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/alloc_traits.h:436
#3 0x660c45 in std::_Vector_base<pik::HuffmanCode, std::allocator<pik::HuffmanCode> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_6 4-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:170
#4 0x660c45 in std::_Vector_base<pik::HuffmanCode, std::allocator<pik::HuffmanCode> >::_M_create_storage(unsigned long) /usr/bin/../lib/gcc /x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:185
#5 0x660c45 in std::_Vector_base<pik::HuffmanCode, std::allocator<pik::HuffmanCode> >::_Vector_base(unsigned long, std::allocator<pik::Huff manCode> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:136
#6 0x660c45 in std::vector<pik::HuffmanCode, std::allocator<pik::HuffmanCode> >::vector(unsigned long, std::allocator<pik::HuffmanCode> con st&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:280
#7 0x660c45 in pik::HuffmanDecodingData::HuffmanDecodingData() /root/gwanyeong/pik/./huffman_decode.h:38
#8 0x660c45 in pik::DecodePlane(unsigned char const*, unsigned long, int, int, pik::Image<int>*) /root/gwanyeong/pik/opsin_codec.cc:991
#9 0x5811c6 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const &, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:392:10
#10 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Imag e3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
#11 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
#12 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
#13 0x7f20fde843f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/gwanyeong/pik/huffman_decode.cc:45:16 in pik::ReplicateValue(pik::HuffmanCode*, int, int, pik::HuffmanCode)
Shadow bytes around the buggy address:
0x0c4a7fff83d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff83e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff83f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff8400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff8410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a7fff8420:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8430: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8460: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff8470: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==18642==ABORTING

Build with VS is completely broken

It used to build a few months ago (if anybody was willing to fiddle with cmake), but current pik isn't even close to building on VS.

Can you please provide proper instructions on how to build pik on VS, or better, simply add proper VS projects instead of dealing with cmake mess.

During cmake step it fails with these messages:

-- Configuring done
CMake Error at CMakeLists.txt:26 (add_library):
  Cannot find source file:

    bit_buffer.h

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx


CMake Error at CMakeLists.txt:169 (add_executable):
  Cannot find source file:

    png2y4m.cc

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx


CMake Error at CMakeLists.txt:169 (add_executable):
  Cannot find source file:

    y4m2png.cc

  Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
  .hxx .in .txx


-- Generating done
-- Build files have been written to: C:/pik0/msvc

C:\pik0\msvc>

When I try to build I get these errors:

1>c:\pik0\af_stats.h(52): error C2665: 'pik::N_NONE::sqrt': none of the 2 overloads could convert all the argument types
1>  c:\pik0\simd/scalar.h(500): note: could be 'pik::N_NONE::scalar<double> pik::N_NONE::sqrt(const pik::N_NONE::scalar<double>)'
1>  c:\pik0\simd/scalar.h(497): note: or       'pik::N_NONE::scalar<float> pik::N_NONE::sqrt(const pik::N_NONE::scalar<float>)'
1>  c:\pik0\af_stats.h(52): note: while trying to match the argument list '(double)'

and most importantly pikcommon doesn't contain any files. It's empty:

when I fix these (by fixing broken cmake files), I get these:

1>c:\pik0\simd/x86_sse4.h(1356): error C2912: explicit specialization 'pik::N_SSE4::vec_sse4<float,4> pik::N_SSE4::load<float>(pik::N_SSE4::Desc<T,4,Target>,const float *__restrict )' is not a specialization of a function template
1>          with
1>          [
1>              T=float,
1>              Target=pik::SSE4
1>          ]

Can you provide proper build tools?

Segmentation fault

I have compile latest pik and i get a segmentation fault with the following command

cpik bench.png bench_1.0_new.pik

Details :
Thread 1 "cpik" received signal SIGSEGV, Segmentation fault. 0x00000000004fd44c in pik::ANSSymbolWriter::FlushToBitStream() ()

Backtrace :

#0  0x00000000004fd44c in pik::ANSSymbolWriter::FlushToBitStream() ()
#1  0x00000000004fa9be in pik::EncodeImage[abi:cxx11](pik::Image3<short> const&, int, pik::PikImageSizeInfo*) ()
#2  0x000000000048df17 in pik::EncodeToBitstream[abi:cxx11](pik::QuantizedCoeffs const&, pik::Quantizer const&, pik::NoiseParams const&, pik::ColorTransform const&, bool, pik::PikInfo*) ()
#3  0x00000000004c39e1 in pik::OpsinToPik(pik::CompressParams const&, pik::Header const&, pik::MetaImage<float> const&, pik::ThreadPool*, pik::PaddedBytes*, pik::PikInfo*) ()
#4  0x00000000004ca2ba in bool pik::PixelsToPikT<pik::MetaImage<float> >(pik::CompressParams const&, pik::MetaImage<float> const&, pik::ThreadPool*, pik::PaddedBytes*, pik::PikInfo*) ()
#5  0x0000000000409ed9 in pik::(anonymous namespace)::CompressAndWrite(int, char**) ()
#6  0x00007ffff682b18b in __libc_start_main () from /lib64/libc.so.6
#7  0x0000000000408d4a in _start ()

GCC version : 8.1.1

pik with maxerror 0.5 hangs for 47hrs +

Stats

resource doesn't seem to be an issue.other pik files with other max error have been generated successfully.

Allow SIMD usage outside of pik namespace

The SIMD library in here is very nice and I have been trying to use it in a project. There is a single minor change to allow these macros to be used outside of the pik namespace.

The only difference is here: https://github.com/google/pik/blob/master/pik/simd/shared.h#L86-L87

#define SIMD_FULL(T) Full<T, SIMD_TARGET>
#define SIMD_PART(T, N) Part<T, N, SIMD_TARGET>

becomes

#define SIMD_FULL(T) Full<T, ::pik::SIMD_TARGET>
#define SIMD_PART(T, N) Part<T, N, ::pik::SIMD_TARGET>

and everything works well.

Aside: Have you guys considered separating the simd project out? it is one of the best out there that I've found.

Rdv shit

compilation with gcc 6.3 on cpu without avx2 /sse4.2

Hello,

I had a lot of trouble finding a version that works on a penryn processor. In the end this commit:
52f2d45
works with this very small patch:

diff --git a/Makefile b/Makefile
index 5e210ad..7cebd30 100644
--- a/Makefile
+++ b/Makefile
-override CXXFLAGS += -std=c++11 -Wall -O3 -fPIC -DSIMD_ENABLE=4 -msse4.2 -maes -I. -I../ -Ithird_party/brotli/c/include/ -Wno-sign-compare
+override CXXFLAGS += -std=c++11 -Wall -O3 -fPIC -DSIMD_ENABLE=4 -msse4.1 -I. -I../ -Ithird_party/brotli/c/include/ -Wno-sign-compare

Between the work of dominikhlbg in #8 an this commit, it always compiles(sometimes with small patches) and executes correctly with Intel® Software Development Emulator, like this:
~/Downloads/sde-external-8.16.0-2018-01-30-lin/sde64 -- ./bin/cpik test.png test.pik

After this commit i always get invalid instruction.
Could you look what is wrong with next commit? Is a newer gcc needed?

Regards,

Problem with std::max

Hi
I can't compile the latest PIK. How to deal with errors in GCC 7.3.0.

In file included from tile_flow.h:38,
                 from tile_flow.cc:17:
image.h: In member function 'int64_t pik::WrapClamp::operator()(int64_t, int64_t) const':
image.h:606:39: error: no matching function for call to 'max(long int, const int64_t&)'
     return std::min(std::max(0L, coord), size - 1L);
                                       ^
In file included from c:\msys801\include\c++\8.0.1\algorithm:61,
                 from tile_flow.h:29,
                 from tile_flow.cc:17:
c:\msys801\include\c++\7.3.0\bits\stl_algobase.h:219:5: note: candidate: 'template<class _Tp> constexpr const _Tp& std::max(const _Tp&, const _Tp&)'
     max(const _Tp& __a, const _Tp& __b)
     ^~~
c:\msys801\include\c++\7.3.0\bits\stl_algobase.h:219:5: note:   template argument deduction/substitution failed:
In file included from tile_flow.h:38,
                 from tile_flow.cc:17:

tile_flow.cc:1406:54: error: no matching function for call to 'max(size_t, long unsigned int)'
       num_instances_(std::max(pool->NumThreads(), 1ul)) {
                                                      ^
In file included from c:\msys801\include\c++\8.0.1\algorithm:61,
                 from tile_flow.h:29,
                 from tile_flow.cc:17:
c:\msys801\include\c++\7.3.0\bits\stl_algobase.h:219:5: note: candidate: 'template<class _Tp> constexpr const _Tp& std::max(const _Tp&, const _Tp&)'
     max(const _Tp& __a, const _Tp& __b)
     ^~~
c:\msys801\include\c++\7.3.0\bits\stl_algobase.h:219:5: note:   template argument deduction/substitution failed:
tile_flow.cc:1406:54: note:   deduced conflicting types for parameter 'const _Tp' ('long long unsigned int' and 'long unsigned int')
       num_instances_(std::max(pool->NumThreads(), 1ul)) {
                                                      ^
In file included from c:\msys801\include\c++\8.0.1\algorithm:61,
                 from tile_flow.h:29,
                 from tile_flow.cc:17:
c:\msys801\include\c++\8.0.1\bits\stl_algobase.h:265:5: note: candidate: 'template<class _Tp, class _Compare> constexpr const _Tp& std::max(const _Tp&, const _Tp&, _Compare)'
     max(const _Tp& __a, const _Tp& __b, _Compare __comp)
     ^~~
c:\msys801\include\c++\7.3.0\bits\stl_algobase.h:265:5: note:   template argument deduction/substitution failed:
tile_flow.cc:1406:54: note:   deduced conflicting types for parameter 'const _Tp' ('long long unsigned int' and 'long unsigned int')
       num_instances_(std::max(pool->NumThreads(), 1ul)) {
                                                      ^
In file included from c:\msys801\include\c++\8.0.1\algorithm:62,
                 from tile_flow.h:29,
                 from tile_flow.cc:17:
c:\msys801\include\c++\8.0.1\bits\stl_algo.h:3462:5: note: candidate: 'template<class _Tp> constexpr _Tp std::max(std::initializer_list<_Tp>)'
     max(initializer_list<_Tp> __l)
     ^~~
c:\msys801\include\c++\7.3.0\bits\stl_algo.h:3462:5: note:   template argument deduction/substitution failed:
tile_flow.cc:1406:54: note:   mismatched types 'std::initializer_list<_Tp>' and 'long long unsigned int'
       num_instances_(std::max(pool->NumThreads(), 1ul)) {
                                                      ^
In file included from c:\msys801\include\c++\8.0.1\algorithm:62,
                 from tile_flow.h:29,
                 from tile_flow.cc:17:
c:\msys801\include\c++\7.3.0\bits\stl_algo.h:3468:5: note: candidate: 'template<class _Tp, class _Compare> constexpr _Tp std::max(std::initializer_list<_Tp>, _Compare)'
     max(initializer_list<_Tp> __l, _Compare __comp)
     ^~~
c:\msys801\include\c++\7.3.0\bits\stl_algo.h:3468:5: note:   template argument deduction/substitution failed:
tile_flow.cc:1406:54: note:   mismatched types 'std::initializer_list<_Tp>' and 'long long unsigned int'
       num_instances_(std::max(pool->NumThreads(), 1ul)) {

Build fails on macOS due to undeclared type '__m128i' in cache_aligned.h

Fixed by adding #include <emmintrin.h> to cache_aligned.h.

c++ -c  -std=c++11 -Wall -O3 -fPIC -msse2 -mavx2 -mfma -mlzcnt -mbmi2 -I. -Ithird_party/brotli/c/include/ -Wno-sign-compare -I/usr/local/include/libpng16 cpik.cc -o obj/cpik.o
In file included from cpik.cc:19:
In file included from ./image.h:35:
./cache_aligned.h:59:26: error: use of undeclared identifier '__m128i'
    static_assert(sizeof(__m128i) % sizeof(T) == 0, "Cannot divide");
                         ^
./cache_aligned.h:60:11: error: unknown type name '__m128i'
    const __m128i* const from = reinterpret_cast<const __m128i*>(from_items);
          ^
./cache_aligned.h:60:56: error: unknown type name '__m128i'
    const __m128i* const from = reinterpret_cast<const __m128i*>(from_items);
                                                       ^
./cache_aligned.h:61:5: error: unknown type name '__m128i'
    __m128i* const to = reinterpret_cast<__m128i*>(to_items);
    ^
./cache_aligned.h:61:42: error: unknown type name '__m128i'
    __m128i* const to = reinterpret_cast<__m128i*>(to_items);
                                         ^
./cache_aligned.h:63:11: error: unknown type name '__m128i'
    const __m128i v0 = _mm_load_si128(from + 0);
          ^
./cache_aligned.h:64:11: error: unknown type name '__m128i'
    const __m128i v1 = _mm_load_si128(from + 1);
          ^
./cache_aligned.h:65:11: error: unknown type name '__m128i'
    const __m128i v2 = _mm_load_si128(from + 2);
          ^
./cache_aligned.h:66:11: error: unknown type name '__m128i'
    const __m128i v3 = _mm_load_si128(from + 3);
          ^
9 errors generated.

ease comparison with other formats

Hello, butteraugli is incredibly sensitive. I had a lot of issues comparing to other formats.
A simple ffmpeg -i image.jpg image.yuv needed to compare with other formats will completely ruin the butteraugli score because rgb24 -> yuv420 conversion. By default yuv will have limited range (luma goes from 16 to 235) that adds even more losses.

I propose in the following script a way to compare fairly pik to h264/h265/aom/jpg. It seems to win without much trouble if the wanted result is minimal change compared to original.
benchmark.txt

Is there any way to use yuv420p as input to pik?
Do you have any better settings for h264/hevc/av1?
(jpeg quality 93 has similar butteraugli score to pik at almost double size)

Provide a Binary for Windows

Hi there,
I hope that you provide a Binary .exe for Windows in your release.

heap-buffer-overflow in pik::DecodeImageData

Hi.

I found a heap-buffer-overflow bug in pik.

Please confirm.

Thanks.

Summary: heap-buffer-overflow
Browser/OS: Ubuntu 17.04 64bit
Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./dpik $PoC /dev/null
PoC download : PoC

------------
Valgrind Log
------------
==29337== Use of uninitialised value of size 8
==29337==    at 0x13B7EC: Advance (bit_reader.h:54)
==29337==    by 0x13B7EC: pik::ReadHistogram(int, std::vector<int, std::allocator<int> >*, pik::BitReader*) (histogram_decode.cc:88)
==29337==    by 0x14C6D8: DecodeHistograms (opsin_codec.cc:672)
==29337==    by 0x14C6D8: pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) (opsin_codec.cc:733)
==29337==    by 0x14CC34: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:882)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Conditional jump or move depends on uninitialised value(s)
==29337==    at 0x13B8FD: pik::ReadHistogram(int, std::vector<int, std::allocator<int> >*, pik::BitReader*) (histogram_decode.cc:112)
==29337==    by 0x14C6D8: DecodeHistograms (opsin_codec.cc:672)
==29337==    by 0x14C6D8: pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) (opsin_codec.cc:733)
==29337==    by 0x14CC34: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:882)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Conditional jump or move depends on uninitialised value(s)
==29337==    at 0x14C765: DecodeHistograms (opsin_codec.cc:683)
==29337==    by 0x14C765: pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) (opsin_codec.cc:733)
==29337==    by 0x14CC34: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:882)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Conditional jump or move depends on uninitialised value(s)
==29337==    at 0x14C78A: DecodeHistograms (opsin_codec.cc:683)
==29337==    by 0x14C78A: pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) (opsin_codec.cc:733)
==29337==    by 0x14CC34: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:882)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Conditional jump or move depends on uninitialised value(s)
==29337==    at 0x1496D2: ReadSymbol (opsin_codec.cc:702)
==29337==    by 0x1496D2: pik::DecodeImageData(unsigned char const*, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> > const&, int, pik::ANSSymbolReader*, pik::Image3<short>*) (opsin_codec.cc:751)
==29337==    by 0x14CC5D: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:884)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Use of uninitialised value of size 8
==29337==    at 0x1496A7: ReadSymbol (opsin_codec.cc:698)
==29337==    by 0x1496A7: pik::DecodeImageData(unsigned char const*, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> > const&, int, pik::ANSSymbolReader*, pik::Image3<short>*) (opsin_codec.cc:751)
==29337==    by 0x14CC5D: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:884)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
==29337== Conditional jump or move depends on uninitialised value(s)
==29337==    at 0x14CC68: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:886)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
==29337==
Pik check failed at opsin_codec.cc:886
==29337==
==29337== Process terminating with default action of signal 6 (SIGABRT)
==29337==    at 0x5DD177F: raise (raise.c:58)
==29337==    by 0x5DD3379: abort (abort.c:89)
==29337==    by 0x14CCDB: pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) (opsin_codec.cc:886)
==29337==    by 0x11CCE9: pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) (compressed_image.cc:394)
==29337==    by 0x1266D0: pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) (pik.cc:544)
==29337==    by 0x10A89D: pik::(anonymous namespace)::Decompress(char const*, char const*) (dpik.cc:58)
==29337==    by 0x5DBC3F0: (below main) (libc-start.c:291)
=================================================================
==10207==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc4fdbcf40 at pc 0x000000657ee5 bp 0x7ffc4fdbc4b0 sp 0x7ffc4fdbc4a8
READ of size 4 at 0x7ffc4fdbcf40 thread T0
    #0 0x657ee4 in pik::BitReader::BitReader(unsigned char const*, unsigned long) /root/gwanyeong/pik/./bit_reader.h:36:36
    #1 0x657ee4 in pik::DecodeImageData(unsigned char const*, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> > const&, int, pik::ANSSymbolReader*, pik::Image3<short>*) /root/gwanyeong/pik/opsin_codec.cc:744
    #2 0x65c319 in pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) /root/gwanyeong/pik/opsin_codec.cc:884:10
    #3 0x581268 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:394:10
    #4 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
    #5 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
    #6 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
    #7 0x7f538d2d23f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #8 0x41b759 in _start (/root/gwanyeong/pik/bin/dpik+0x41b759)

Address 0x7ffc4fdbcf40 is located in stack of thread T0 at offset 160 in frame
    #0 0x5af4df in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:525

  This frame has 4 object(s):
    [32, 48) 'header'
    [64, 96) 'source'
    [128, 160) 'encoded_img' <== Memory access at offset 160 overflows this variable
    [192, 744) 'img'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /root/gwanyeong/pik/./bit_reader.h:36:36 in pik::BitReader::BitReader(unsigned char const*, unsigned long)
Shadow bytes around the buggy address:
  0x100009faf990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009faf9a0: 00 00 00 00 00 00 00 00 00 00 00 00 f3 f3 f3 f3
  0x100009faf9b0: f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00
  0x100009faf9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009faf9d0: 00 00 00 00 f1 f1 f1 f1 00 00 f2 f2 00 00 00 00
=>0x100009faf9e0: f2 f2 f2 f2 00 00 00 00[f2]f2 f2 f2 00 00 00 00
  0x100009faf9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009fafa00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009fafa10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009fafa20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100009fafa30: 00 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==10207==ABORTING

Optimizing pik against ssimulacra

maxerror,filesize,butteraugli,ssimulacra
1.0,117241,0.999972,0.02952750

Its clear that pik is optimizing against butteraugli but it would be nice to see it optimized for ssimulacra as well

Pik means Dick in Dutch

Not sure if that's what you want to communicate here, but that's the first thing I saw before realizing it's wordplay on Pic...

memory order argument to atomic operation is invalid

···sh
../pik/cache_aligned.cc:99:74: error: memory order argument to atomic operation is invalid [-Werror,-Wuser-defined-warnings]
uint64_t expected_max = max_bytes_in_use.load(std::memory_order_acq_rel);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/atomic:928:7: note: from 'diagnose_if' attribute on 'load':
_LIBCPP_CHECK_LOAD_MEMORY_ORDER(__m)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/atomic:573:3: note: expanded from macro '_LIBCPP_CHECK_LOAD_MEMORY_ORDER'
_LIBCPP_DIAGNOSE_WARNING(__m == memory_order_release ||
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__config:1251:21: note: expanded from macro '_LIBCPP_DIAGNOSE_WARNING'
attribute((diagnose_if(VA_ARGS, "warning")))
^ ~~~~~~~~~~~
1 error generated.

Daily build?

subj

The creation of custom providers could be exemplified in more detail. This should also be possible in independent packages that make use of cocoda-sdk:

import { BaseProvider } from "cocoda-sdk"

export default class MyProvider extends BaseProvider {
  // what is expected to be implemented here?
}

It may make sense to do #60 first.

Originally posted by @nichtich in gbv/cocoda-sdk#61

project README contains no clear purpose, intent, or target goals

e.g.

"aim to improve speed of [encode|decode]"
"to see if we can improve the quality [per byte of]"
"thought it would be fun to..."
etc.

Plugin for Irfanview/Xnview?

subj

Keep the top-level directory clean of source files

Newcomer suggestion: move all of the source files into an src/ directory. This would make it easier to navigate the source tree and to locate other files such as the readme (both locally and on GitHub); to check whether there is something like a Makefile or CMakeLists.txt or autogen.sh (because the first thing you do after cloning is of course running make, not reading the readme).

encoder is very slow - can we skip Butteraugli optimization and use fixed quality settings?

I tried cpik on some images and noticed that it is very slow. This is to be expected from an early prototype/PoC implementation, but I wonder how much of it is inherent complexity of the algorithm.

I assume that much of it is due to some kind of iterative optimization involving Butteraugli (like in Guetzli).

Would it be possible to tell the encoder to use some kind of fixed quality settings (e.g. like in a typical JPEG or WebP encoder), so it's not adaptive to the image contents but probably much faster?

It would be interesting to compare the pik format to the jpeg (e.g. mozjpeg) and webp formats in a fair way. The iterative perceptual-metric-based optimization is a nice thing to have, but that is somewhat orthogonal to the image format itself (you could also apply it to JPEG – cf. Guetzli – and I assume to WebP as well).

Instructions on Building the Project

There are no clear build instructions on the page for PIK encoder and decoder. The same would be very helpful.

stack-buffer-overflow in pik::HuffmanDecodingData::ReadFromBitStream

Hi.

I found a stack-buffer-overflow bug in pik.

Please confirm.

Thanks.

Summary: stack-buffer-overflow
Browser/OS: Ubuntu 17.04 64bit
Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./dpik $PoC /dev/null
PoC download : PoC

=================================================================
==30122==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc74c2fb20 at pc 0x0000005d3f0e bp 0x7ffc74c2f1b0 sp 0x7ffc74c2f1a8
READ of size 4 at 0x7ffc74c2fb20 thread T0
    #0 0x5d3f0d in pik::BitReader::FillBitBuffer() /root/gwanyeong/pik/./bit_reader.h:46:39
    #1 0x5d3f0d in pik::BitReader::ReadBits(int) /root/gwanyeong/pik/./bit_reader.h:68
    #2 0x5d3f0d in pik::HuffmanDecodingData::ReadFromBitStream(pik::BitReader*) /root/gwanyeong/pik/huffman_decode.cc:269
    #3 0x660cce in pik::DecodePlane(unsigned char const*, unsigned long, int, int, pik::Image<int>*) /root/gwanyeong/pik/opsin_codec.cc:992:8
    #4 0x6a0578 in pik::Quantizer::Decode(unsigned char const*, unsigned long) /root/gwanyeong/pik/quantizer.cc:141:10
    #5 0x581212 in pik::CompressedImage::DecodeQuantization(unsigned char const*, unsigned long) /root/gwanyeong/pik/compressed_image.cc:380:21
    #6 0x581212 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:393
    #7 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
    #8 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
    #9 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
    #10 0x7fabb4e083f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #11 0x41b759 in _start (/root/gwanyeong/pik/bin/dpik+0x41b759)

Address 0x7ffc74c2fb20 is located in stack of thread T0 at offset 160 in frame
    #0 0x5af4df in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:525

  This frame has 4 object(s):
    [32, 48) 'header'
    [64, 96) 'source'
    [128, 160) 'encoded_img' <== Memory access at offset 160 overflows this variable
    [192, 744) 'img'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow /root/gwanyeong/pik/./bit_reader.h:46:39 in pik::BitReader::FillBitBuffer()
Shadow bytes around the buggy address:
  0x10000e97df10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000e97df20: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
  0x10000e97df30: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00
  0x10000e97df40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000e97df50: f1 f1 f1 f1 00 00 f2 f2 00 00 00 00 f2 f2 f2 f2
=>0x10000e97df60: 00 00 00 00[f2]f2 f2 f2 00 00 00 00 00 00 00 00
  0x10000e97df70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000e97df80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000e97df90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10000e97dfa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 f3 f3 f3
  0x10000e97dfb0: f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==30122==ABORTING

PIK means COCK in Danish and Dutch

PIK means COCK in both Danish and Dutch. Yes - Cock as in Penis.

I suggest you change the name of PIK to something less obscene.

PIK not exceptional for high resolution photographic sources -> use existing solutions?

Spun off from #33 to separate different points out as requested.

While WebP compresses well at low bit rates and doesn't exhibit as bad block artefacts as JPEG, WebP becomes increasingly more inefficient at higher bit rates (when one tries to compress without visual loss), and at high quality (somewhere above quality 80-90 in libjpeg) it becomes less efficient in compression density than JPEG as implemented in libjpeg. Also, some photographers have discussed the need to manually review WebP re-compression while discussing that an approach based on guetzli/butteraugli could possibly be fully automated. PIK is the guetzli/butteraugli approach refined to the best possible state.

Once again the choice of format based on required bitrate is important. If focused on high resolution photographic sources, then the advances in compression from Guetzli can already be used in standard JPEG images (or one could consider JXR which is already (ISO) standardized if things like wide-gamut and transparency are desired on top of photographic-focused compression techniques). Since the comparison showcase focuses solely on high bitrate/high quality compression, what more does Butteraugli bring to the table there that makes the new image format needed, without a doubt? Since Butteraugli only aids in finding the optimal psychovisual distance, why can't all of this just be made in a slow-but-accurate JPEG/JXR compressor instead of making an entirely new format?
Especially with JXR offering broader dynamic color ranges, and being designed from the ground up for HD photographic material, a highly-optimized compressor for that format would very likely result in at least competitive, if not superior images than what PIK could produce.

Format's intent is unclear

The format's intent seems to be very unclear from the general description and information available.

Without diving into the code, it seems to aim to be a direct replacement for jpeg, with all its known flaws as a web image format (most notably the lack of transparency). Comparing compression ratios, I wonder why we'd need a new image format to begin with -- what's wrong with WebP, that overcomes several major issues for a web image format and achieves great compression ratios as well?

Why not just focus on keeping standardized image formats and better compression like through Guetzli, so you're not asking the software community to consider, implement and adopt a new image format when the previous new one has barely been adopted yet?

What intent is there, really, to introduce jpeg-in-a-new-jacket?

I don't know how to compiled version 16102018?

adaptive_reconstruction.cc:140:27: error: inlining failed in call to always_inline 'void pik::{anonymous}::ClampBlockToOriginalDCT_Y(const float*, size_t, const float*, const float*, float, float, float, float, float*, float*, float*, float*)': target specific option mismatch

adaptive_reconstruction.cc:174:27: error: inlining failed in call to always_inline 'void pik::{anonymous}::ClampBlockToOriginalDCT_XB(const float*, size_t, const float*, const float*, const float*, int32_t, float, const float*, const float*, float, float, float, float, float*) [with TagXB = pik::TagX]': target specific option mismatch

af_edge_preserving_filter.cc:53:24: fatal error: af_epf.cctest: No such file or directory #define SIMD_ATTR_IMPL "af_epf.cctest"

compressed_image.cc:15: simd/x86_avx2.h: In lambda function: simd/x86_avx2.h:1439:33: error: inlining failed in call to always_inline 'void pik::store(pik::vec_avx2<float, 8>, pik::Full<float, pik::AVX2>, float*)': target specific option mismatch SIMD_ATTR_AVX2 SIMD_INLINE void store(const vec_avx2<float> v, dct.h:153:34: error: inlining failed in call to always_inline 'void pik::ComputeTransposedScaledDCT(const From&, const To&) [with long long unsigned int N = 8; From = pik::FromLines; To = pik::ScaleToBlock<8>]': target specific option mismatch static SIMD_ATTR PIK_INLINE void ComputeTransposedScaledDCT(const From& from,

dct_util.cc:6: dct.h:153:34: error: inlining failed in call to always_inline 'void pik::ComputeTransposedScaledDCT(const From&, const To&) [with long long unsigned int N = 8; From = pik::FromBlock<8>; To = pik::ToBlock<8>]': target specific option mismatch static SIMD_ATTR PIK_INLINE void ComputeTransposedScaledDCT(const From& from,

dct_util.cc:1: simd/x86_avx2.h:1371:44: error: inlining failed in call to always_inline 'pik::vec_avx2<float, 8> pik::load(pik::Full<float, pik::AVX2>, const float*)': target specific option mismatch SIMD_ATTR_AVX2 SIMD_INLINE vec_avx2<float> load(

dc_predictor.cc:151:36: error: inlining failed in call to always_inline 'static void pik::{anonymous}::PixelNeighborsY::Store(pik::{anonymous}::PixelNeighborsY::PixelV, pik::DC*, size_t)': target specific option mismatch static SIMD_ATTR PIK_INLINE void Store(const PixelV dc, DC* PIK_RESTRICT row,

opsin_inverse.cc:15: simd/x86_avx2.h: In function 'void __static_initialization_and_destruction_0(int, int)': simd/x86_avx2.h:61:30: error: inlining failed in call to always_inline 'pik::vec_avx2<T, N>::vec_avx2() [with T = float; long long unsigned int N = 8]': target specific option mismatch SIMD_ATTR_AVX2 SIMD_INLINE vec_avx2() {}

PIK as a Web Image format (small sizes/non-photographic) not adding anything

Web graphics also tend to not be photographic and/or small size, which would be once again in the realm of WebP for photographic or PNG for drawn images, or even entering the vector graphics area.

cpik segfaults when encoding from PNG file.

Running latest cpik code on macOS High Sierra (10.13.3). Attempts to encode a pik image from a PNG source with --distance 1.0 gives the following error:

Segmentation fault: 11

Any suggestions? Or do you need any other info? I realize I'm a bit thin on specifics, but this is all I can provide. Output from uname -a:

Darwin Phobos 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Other than that, encoding from JPEG with the same Butteraugli distance appears to work fine!

heap-buffer-overflow in ANSSymbolReader::DecodeHistograms

Hi.

I found a heap-buffer-overflow bug in pik.

Please confirm.

Thanks.

Summary: heap-buffer-overflow
Browser/OS: Ubuntu 17.04 64bit
Steps to reproduce:
1.Download the .POC files.
2.Execute the following command
: ./dpik $PoC /dev/null
PoC download : PoC

--------------------------------------------
gdb log
--------------------------------------------
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
#1 0x00007ffff6ae537a in __GI_abort () at abort.c:89
#2 0x00007ffff6b27090 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6c3d000 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff6b30c3a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7ffff6c39bb0 "free(): invalid size", action=3)
at malloc.c:5048
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3904
#5 0x00007ffff6b34d2c in __GI___libc_free (mem=<optimized out>) at malloc.c:2984
#6 0x00005555555987b4 in __gnu_cxx::new_allocator<int>::deallocate (this=0x7fffffffdb60, __p=<optimized out>) at /usr/include/c++/6/ext/new_allocator.h:110
#7 std::allocator_traits<std::allocator<int> >::deallocate (__a=..., __n=<optimized out>, __p=<optimized out>)
at /usr/include/c++/6/bits/alloc_traits.h:462
#8 std::_Vector_base<int, std::allocator<int> >::_M_deallocate (__n=<optimized out>, __p=<optimized out>, this=0x7fffffffdb60)
at /usr/include/c++/6/bits/stl_vector.h:178
#9 std::_Vector_base<int, std::allocator<int> >::~_Vector_base (this=0x7fffffffdb60, __in_chrg=<optimized out>) at /usr/include/c++/6/bits/stl_vector.h:160
#10 std::vector<int, std::allocator<int> >::~vector (this=0x7fffffffdb60, __in_chrg=<optimized out>) at /usr/include/c++/6/bits/stl_vector.h:427
#11 pik::ANSSymbolReader::DecodeHistograms (in=0x7fffffffdb80, symbol_lut_size=0, symbol_lut=0x0, num_histograms=8, this=0x7fffffffdc30)
at opsin_codec.cc:671
#12 pik::DecodeHistograms (data=data@entry=0x5555557ce454 "\265b", len=len@entry=1000, num_contexts=num_contexts@entry=3, symbol_lut=symbol_lut@entry=0x0,
symbol_lut_size=symbol_lut_size@entry=0, decoder=decoder@entry=0x7fffffffdc30, context_map=0x7fffffffdc10) at opsin_codec.cc:733
#13 0x0000555555598c35 in pik::DecodeImage (data=data@entry=0x5555557ce454 "\265b", len=len@entry=1000, stride=stride@entry=64,
coeffs=coeffs@entry=0x7fffffffe160) at opsin_codec.cc:882
#14 0x0000555555568cea in pik::CompressedImage::Decode (xsize=<optimized out>, ysize=<optimized out>,
data="\201\061\000\000\000@\000\b\340&;\032\030\004S\253>0\327еb\000\000\224KҺ\361\212\352\000\001\000\000\000\f\000\346\317\354\253pn\213\254\t9^\035\224\331!\213\366w\255 \227#%傆\230o\347\067k\362T\260uu\207\004\223>\223\325}\016\000\000\347\344\306\030r\203\231\375?Q4<(\247k\235\241\333۴\256\335\367s\241IK/v,\024)\246'Y\006\241\221\064c±S2\265\346h\243\251⫣\363\240\016k\274\270F4V\v\337\277\360Ob\301\202\226â\206\277 $ۘ\371|\256\346\253\326Yj\342#\275\332\357\345\374\336{\365\275\347\367\356\356\275w\274\367\356\275w\357\356\275\367\364\336", <incomplete sequence \357>...,
info=info@entry=0x7fffffffe3c0) at compressed_image.cc:394
#15 0x00005555555726d1 in pik::PikToPixels (params=..., compressed=..., planes=0x7fffffffe360, aux_out=0x7fffffffe3c0) at pik.cc:544
#16 0x000055555555689e in pik::(anonymous namespace)::Decompress (pathname_in=<optimized out>, pathname_out=0x7fffffffe81d "/dev/null") at dpik.cc:58
#17 0x00007ffff6ace3f1 in __libc_start_main (main=0x555555556470 <main(int, char**)>, argc=3, argv=0x7fffffffe588, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe578) at ../csu/libc-start.c:291
#18 0x00005555555564da in _start ()
=================================================================
==11691==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62500000c100 at pc 0x00000067171d bp 0x7ffcdf67b9f0 sp 0x7ffcd f67b9e8
WRITE of size 1 at 0x62500000c100 thread T0
#0 0x67171c in pik::ANSSymbolReader::DecodeHistograms(unsigned long, unsigned char const*, unsigned long, pik::BitReader*) /ro ot/karas/pik/opsin_codec.cc:684:47
#1 0x65c26f in pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) /root/karas/pik/opsin_codec.cc:733:12
#2 0x65c26f in pik::DecodeImage(unsigned char const*, unsigned long, int, pik::Image3<short>*) /root/karas/pik/opsin_codec .cc:882
#3 0x581268 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator< char> > const&, pik::PikInfo*) /root/karas/pik/compressed_image.cc:394:10
#4 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const &, pik::Image3<unsigned char>*, pik::PikInfo*) /root/karas/pik/pik.cc:543:29
#5 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/karas/pik/dpik.cc:58:7
#6 0x50f124 in main /root/karas/pik/dpik.cc:80
#7 0x7f8bdd1f13f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
#8 0x41b759 in _start (/root/karas/pik/bin/dpik+0x41b759)

0x62500000c100 is located 0 bytes to the right of 8192-byte region [0x62500000a100,0x62500000c100)
allocated by thread T0 here:
#0 0x50a1b0 in operator new(unsigned long) (/root/karas/pik/bin/dpik+0x50a1b0)
#1 0x5551e7 in __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux- gnu/6.3.0/../../../../include/c++/6.3.0/ext/new_allocator.h:104:27
#2 0x5551e7 in std::allocator_traits<std::allocator<unsigned char> >::allocate(std::allocator<unsigned char>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/alloc_traits.h:436
#3 0x5551e7 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) /usr/bin/../lib/gc c/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:170
#4 0x5551e7 in std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) /usr/bin/../lib/gc c/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/vector.tcc:557

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/karas/pik/opsin_codec.cc:684:47 in pik::ANSSymbolReader::DecodeHistogram s(unsigned long, unsigned char const*, unsigned long, pik::BitReader*)
Shadow bytes around the buggy address:
0x0c4a7fff97d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff97e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff97f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff9800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c4a7fff9810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a7fff9820:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff9830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff9840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff9850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff9860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c4a7fff9870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==11691==ABORTING
----------------------------------------------------
EXPLOITABLE plugin log
----------------------------------------------------
Classification: EXPLOITABLE
Faulting Frame:
__gnu_cxx::new_allocator<int>::deallocate @ 0x00005555555987b4: in /root/karas/pik/bin/dpik
Disassembly:
Stack Head (17 entries):
__GI_raise @ 0x00007ffff6ae377f: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
__GI_abort @ 0x00007ffff6ae537a: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
__libc_message @ 0x00007ffff6b27090: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
malloc_printerr @ 0x00007ffff6b30c3a: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
_int_free @ 0x00007ffff6b30c3a: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
__GI___libc_free @ 0x00007ffff6b34d2c: in /lib/x86_64-linux-gnu/libc-2.24.so (BL)
__gnu_cxx::new_allocator< @ 0x00005555555987b4: in /root/karas/pik/bin/dpik
std::allocator_traits<std @ 0x00005555555987b4: in in /root/karas/pik/bin/dpik
std::_Vector_base<int, @ 0x00005555555987b4: in 0x5555555987b4 in /root/karas/pik/bin/dpik
std::_Vector_base<int, @ 0x00005555555987b4: in 0x5555555987b4 in /root/karas/pik/bin/dpik
std::vector<int, @ 0x00005555555987b4: in 0x5555555987b4 in /root/karas/pik/bin/dpik
pik::ANSSymbolReader::Dec @ 0x00005555555987b4: in /root/karas/pik/bin/dpik
pik::DecodeHistograms @ 0x00005555555987b4: in /root/karas/pik/bin/dpik
pik::DecodeImage @ 0x0000555555598c35: in /root/karas/pik/bin/dpik
pik::CompressedImage::Dec @ 0x0000555555568cea: in /root/karas/pik/bin/dpik
pik::PikToPixels @ 0x00005555555726d1: in /root/karas/pik/bin/dpik
Registers:
rax=0x0000000000000000 rbx=0x0000000000000058 rcx=0x00007ffff6ae377f rdx=0x0000000000000000
rsi=0x00007fffffffd510 rdi=0x0000000000000002 rbp=0x00007fffffffd9b0 rsp=0x00007fffffffd588
r8=0x0000000000000000 r9=0x00007fffffffd510 r10=0x0000000000000008 r11=0x0000000000000246
r12=0x0000000000000058 r13=0x00007fffffffd7c0 r14=0x00007fffffffd7c0 r15=0x00007ffff7ff4000
rip=0x00007ffff6ae377f efl=0x0000000000000246 cs=0x0000000000000033 ss=0x000000000000002b
ds=0x0000000000000000 es=0x0000000000000000 fs=0x0000000000000000 gs=0x0000000000000000
Extra Data:
Description: Heap error
Short description: HeapError (10/22)
Explanation: The target's backtrace indicates that libc has detected a heap error or that the target was executing a heap function when it stopped. This could be due to heap corruption, passing a bad pointer to a heap function such as free(), etc. Since heap errors might include buffer overflows, use-after-free situations, etc. they are generally considered exploitable.
---END SUMMARY---

heap-buffer-overflow in pik::DecodeACData

Hi.

I found a heap-buffer-overflow bug in pik.

Please confirm.

Thanks.

=================================================================
==30053==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6140000001d8 at pc 0x00000065a9fa bp 0x7fff62ba5550 sp 0x7fff62ba5548
READ of size 1 at 0x6140000001d8 thread T0
    #0 0x65a9f9 in pik::DecodeACData(unsigned char const*, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::ANSSymbolReader*, pik::Image3<short>*) /root/gwanyeong/pik/opsin_codec.cc:835:25
    #1 0x65cf0f in pik::DecodeAC(unsigned char const*, unsigned long, pik::Image3<short>*) /root/gwanyeong/pik/opsin_codec.cc:897:10
    #2 0x5812b0 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:395:10
    #3 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
    #4 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
    #5 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
    #6 0x7f37d45a63f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #7 0x41b759 in _start (/root/gwanyeong/pik/bin/dpik+0x41b759)

0x6140000001d8 is located 0 bytes to the right of 408-byte region [0x614000000040,0x6140000001d8)
allocated by thread T0 here:
    #0 0x50a1b0 in operator new(unsigned long) (/root/gwanyeong/pik/bin/dpik+0x50a1b0)
    #1 0x65cbaf in __gnu_cxx::new_allocator<unsigned char>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/ext/new_allocator.h:104:27
    #2 0x65cbaf in std::allocator_traits<std::allocator<unsigned char> >::allocate(std::allocator<unsigned char>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/alloc_traits.h:436
    #3 0x65cbaf in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:170
    #4 0x65cbaf in std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_append(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/vector.tcc:557
    #5 0x65cbaf in std::vector<unsigned char, std::allocator<unsigned char> >::resize(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/bits/stl_vector.h:677
    #6 0x65cbaf in pik::DecodeHistograms(unsigned char const*, unsigned long, unsigned long, unsigned char const*, unsigned long, pik::ANSSymbolReader*, std::vector<unsigned char, std::allocator<unsigned char> >*) /root/gwanyeong/pik/opsin_codec.cc:729
    #7 0x65cbaf in pik::DecodeAC(unsigned char const*, unsigned long, pik::Image3<short>*) /root/gwanyeong/pik/opsin_codec.cc:894
    #8 0x5812b0 in pik::CompressedImage::Decode(int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pik::PikInfo*) /root/gwanyeong/pik/compressed_image.cc:395:10
    #9 0x5afa67 in pik::PikToPixels(pik::DecompressParams const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::Image3<unsigned char>*, pik::PikInfo*) /root/gwanyeong/pik/pik.cc:543:29
    #10 0x50f124 in pik::(anonymous namespace)::Decompress(char const*, char const*) /root/gwanyeong/pik/dpik.cc:58:7
    #11 0x50f124 in main /root/gwanyeong/pik/dpik.cc:80
    #12 0x7f37d45a63f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/gwanyeong/pik/opsin_codec.cc:835:25 in pik::DecodeACData(unsigned char const*, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> > const&, pik::ANSSymbolReader*, pik::Image3<short>*)
Shadow bytes around the buggy address:
  0x0c287fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8000: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c287fff8010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c287fff8030: 00 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa
  0x0c287fff8040: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c287fff8050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8070: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c287fff8080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==30053==ABORTING

duplicate symbol when build static lib

Code does not compile with Clang 3.4

The first batch of errors (though not the only ones):

clang++ -c  -std=c++11 -Wall -O3 -fPIC -mavx2 -mfma -mlzcnt -mbmi2 -I. -Wno-sign-compare -I/usr/include/libpng12   cpik.cc -o obj/cpik.o
In file included from cpik.cc:15:
In file included from ./image_io.h:27:
./image.h:155:28: error: use of undeclared identifier '__builtin_assume_aligned'
    return static_cast<T*>(PIK_ASSUME_ALIGNED(row, 64));
                           ^
./compiler_specific.h:102:40: note: expanded from macro 'PIK_ASSUME_ALIGNED'
#define PIK_ASSUME_ALIGNED(ptr, align) __builtin_assume_aligned((ptr), (align))
                                       ^
In file included from cpik.cc:15:
In file included from ./image_io.h:27:
./image.h:162:28: error: use of undeclared identifier '__builtin_assume_aligned'
    return static_cast<T*>(PIK_ASSUME_ALIGNED(row, 64));
                           ^
./compiler_specific.h:102:40: note: expanded from macro 'PIK_ASSUME_ALIGNED'
#define PIK_ASSUME_ALIGNED(ptr, align) __builtin_assume_aligned((ptr), (align))
                                       ^
In file included from cpik.cc:15:
In file included from ./image_io.h:27:
./image.h:174:48: error: use of undeclared identifier '__builtin_assume_aligned'
    return static_cast<uint8_t * PIK_RESTRICT>(PIK_ASSUME_ALIGNED(p, 64));
                                               ^
./compiler_specific.h:102:40: note: expanded from macro 'PIK_ASSUME_ALIGNED'
#define PIK_ASSUME_ALIGNED(ptr, align) __builtin_assume_aligned((ptr), (align))
                                       ^
In file included from cpik.cc:15:
In file included from ./image_io.h:27:
./image.h:178:54: error: use of undeclared identifier '__builtin_assume_aligned'
    return static_cast<const uint8_t * PIK_RESTRICT>(PIK_ASSUME_ALIGNED(p, 64));
                                                     ^
./compiler_specific.h:102:40: note: expanded from macro 'PIK_ASSUME_ALIGNED'
#define PIK_ASSUME_ALIGNED(ptr, align) __builtin_assume_aligned((ptr), (align))
                                       ^
4 errors generated.
make: *** [obj/cpik.o] Error 1

compilation issue with MSYS2 under Windows 10

Hello,

I can not compile PIK successfully with MSYS2 under Windows 10.( I have quite many errors).

Could you post the latest cpik and dpik binaries for Windows 10?

Many thanks!
Cheers,
Raphael

Lossless transcoding from JPEG

Quote from the README:

We are planning to keep the format 8x8 DCT based

Is one of the goals is to be able to losslessly re-pack JPEG images to PIK (keeping the actual data of 8x8 blocks intact), saving some space due to improved entropy coding?
If that's true that's going to be great IMO, as there are a lot of images for which the original uncompressed versions cannot be found anymore, but they could still benefit from some size reduction without further losses (and ending up in a format that's going to be widely supported, unlike Lepton or arithmetic coded JPEGs).

google / pik Goto Github PK

pik's Introduction

PIK

Project Goals

Size comparisons

Build instructions

Usage

Related projects

pik's People

Contributors

Stargazers

Watchers

Forkers

pik's Issues

Recommend Projects

Recommend Topics

Recommend Org