google / brunsli Goto Github PK

It was difficult for me to find the official website URL after finding this repository, so I'd suggest adding https://brunsli.dev/ to the repository description and README.

Hi, I‘m using Ubuntu 22.04. and I put my emsdk under folder /opt/emsdk.
However, I got the following ERROR message. Compiler is not EMSRIPOM

 -- Compiler is not EMSCRIPTEN
 CMake Deprecation Warning at build/_deps/brotli-src/CMakeLists.txt:5 (cmake_minimum_required):
   Compatibility with CMake < 2.8.12 will be removed from a future version of
   CMake.

   Update the VERSION argument <min> value or use a ...<max> suffix to tell
   CMake that the project does not need compatibility with older versions.

 Build type is 'Release'
 -- Compiler is not EMSCRIPTEN
 Configuring done (0.4s)

Can anybody give me a hand? Thank you ..

Hi,
Thanks for sharing this great repo for compress lossless jpeg

I build on my local machine and want to try the compress/decompress
but I meet Illegal instruction (core dumped) error, debug information is like below
Could you help to check if I miss something to lead this？
test pic I'm using is here

ubuntu@ubuntu:~/paper/brunsli/build$ lldb-7 ./cbrunsli world.topo.bathy.200407.3x21600x10800.jpg
(lldb) target create "./cbrunsli"
Current executable set to './cbrunsli' (x86_64).
(lldb) settings set -- target.run-args  "world.topo.bathy.200407.3x21600x10800.jpg"
(lldb) r
Process 16101 launched: '/home/code/brunsli/build/cbrunsli' (x86_64)
Process 16101 stopped
* thread #1, name = 'cbrunsli', stop reason = signal SIGILL: illegal instruction operand
    frame #0: 0x0000000000455c90 cbrunsli`std::_Bit_iterator::operator*() const + 32
cbrunsli`std::_Bit_iterator::operator*:
->  0x455c90 <+32>: shlxq  %rdi, %rdx, %rdx
    0x455c95 <+37>: leaq   -0x10(%rbp), %rdi
    0x455c99 <+41>: callq  0x455d30                  ; std::_Bit_reference::_Bit_reference(unsigned long*, unsigned long)
    0x455c9e <+46>: movq   -0x10(%rbp), %rax
(lldb) bt
* thread #1, name = 'cbrunsli', stop reason = signal SIGILL: illegal instruction operand
  * frame #0: 0x0000000000455c90 cbrunsli`std::_Bit_iterator::operator*() const + 32
    frame #1: 0x00000000004445a6 cbrunsli`std::vector<bool, std::allocator<bool> >::operator[](unsigned long) + 70
    frame #2: 0x0000000000459bea cbrunsli`brunsli::(anonymous namespace)::ProcessSOF(unsigned char const*, unsigned long, brunsli::JpegReadMode, unsigned long*, brunsli::JPEGData*) + 1130
    frame #3: 0x00000000004592df cbrunsli`brunsli::ReadJpeg(unsigned char const*, unsigned long, brunsli::JpegReadMode, brunsli::JPEGData*) + 1151
    frame #4: 0x0000000000433b07 cbrunsli`ProcessFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 151
    frame #5: 0x0000000000433d53 cbrunsli`main + 195
    frame #6: 0x00007ffff718c830 libc.so.6`__libc_start_main(main=(cbrunsli`main), argc=2, argv=0x00007fffffffd388, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffd378) at libc-start.c:291
    frame #7: 0x0000000000433599 cbrunsli`_start + 41
(lldb) di
cbrunsli`std::_Bit_iterator::operator*:
    0x455c70 <+0>:  pushq  %rbp
    0x455c71 <+1>:  movq   %rsp, %rbp
    0x455c74 <+4>:  subq   $0x20, %rsp
    0x455c78 <+8>:  movq   %rdi, -0x18(%rbp)
    0x455c7c <+12>: movq   -0x18(%rbp), %rdi
    0x455c80 <+16>: movq   (%rdi), %rsi
    0x455c83 <+19>: movb   0x8(%rdi), %al
    0x455c86 <+22>: movb   %al, %dil
    0x455c89 <+25>: movl   $0x1, %ecx
    0x455c8e <+30>: movl   %ecx, %edx
->  0x455c90 <+32>: shlxq  %rdi, %rdx, %rdx
    0x455c95 <+37>: leaq   -0x10(%rbp), %rdi
    0x455c99 <+41>: callq  0x455d30                  ; std::_Bit_reference::_Bit_reference(unsigned long*, unsigned long)
    0x455c9e <+46>: movq   -0x10(%rbp), %rax
    0x455ca2 <+50>: movq   -0x8(%rbp), %rdx
    0x455ca6 <+54>: addq   $0x20, %rsp
    0x455caa <+58>: popq   %rbp
    0x455cab <+59>: retq
(lldb) up
frame #1: 0x00000000004445a6 cbrunsli`std::vector<bool, std::allocator<bool> >::operator[](unsigned long) + 70
cbrunsli`std::vector<bool, std::allocator<bool> >::operator[]:
    0x4445a6 <+70>: movq   %rax, -0x10(%rbp)
    0x4445aa <+74>: movq   %rdx, -0x8(%rbp)
    0x4445ae <+78>: movq   -0x10(%rbp), %rax
    0x4445b2 <+82>: movq   -0x8(%rbp), %rdx
(lldb) up
frame #2: 0x0000000000459bea cbrunsli`brunsli::(anonymous namespace)::ProcessSOF(unsigned char const*, unsigned long, brunsli::JpegReadMode, unsigned long*, brunsli::JPEGData*) + 1130
cbrunsli`brunsli::(anonymous namespace)::ProcessSOF:
    0x459bea <+1130>: movq   %rax, -0x98(%rbp)
    0x459bf1 <+1137>: movq   %rdx, -0x90(%rbp)
    0x459bf8 <+1144>: leaq   -0x98(%rbp), %rdi
    0x459bff <+1151>: callq  0x455d60                  ; std::_Bit_reference::operator bool() const

clang version I'm using

clang-7 -v
clang version 7.0.1-svn348686-1~exp1~20181221231927.53 (branches/release_70)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/5
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/5.4.0
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6.0.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/6.0.0
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/5
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/5.4.0
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/6.0.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/5.4.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/6.0.0
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64

Usual JPEG XL files start with bytes ff 0a. As far as I understand, it follows ISOBMFF as other modern ISO formats do. Brunsli files start with 0a 04 42 d2 d5 4e. Why is it so different? If both of these codecs are under the same umbrella name, MIME type and file extension, why they don't share a similar signature? Why Brunsli signature is so long?

Could you please add support for larger JPEG such as those:

https://drive.google.com/open?id=0ByLIAFlgldSoN21wX3E3aGdRUVk
https://drive.google.com/open?id=0ByLIAFlgldSoNTFlZFRKVU1GVkE

Hi,

Safari supports WASM, but the encoding speed is 10x slower with it (v13.1.1 or Technology Preview 109) than it is with Chrome or Firefox (https://google.github.io/brunsli/).
Is there something you can do to achieve similar performances across browsers?

Hello all!

I see your last release (0.1) is from 2019: https://github.com/google/brunsli/releases/tag/v0.1

Are you considering a new release, or this project follows bleeding edge approach?

Dear all,

I read the brunsli's ANS related code, such as AddCode or PutSymbol..., As you know, the ANS has a lot of variant, but I can not figure out which type of ANS the brunsli is using.

Could you please help me to understand the brunsli's ANS algorithm?

Hello,

Currently brunsli produces static libraries for C++. Can we create a static library (.a) for C also?

At conda-forge we build with dynamically loaded libraries.

It would be nice to have a switch to enable both.

I had to do a few things:

1. Disable your vendored version of brotli
2. Link to all necessary libraries of brotli using their dyanmic version.

Still a bit of a work in progress, but the patch I had to use can be seen here:
https://github.com/conda-forge/staged-recipes/pull/11913/files

Can we have some instructions on how to setup wasm to do this please?

Hello there!

I've run compilation and got error:
rm -f libbrotli.a
ar -crs libbrotli.a bin/obj/c/common/dictionary.o bin/obj/c/common/transform.o bin/obj/c/dec/bit_reader.o bin/obj/c/dec/decode.o bin/obj/c/dec/huffman.o bin/obj/c/dec/state.o bin/obj/c/enc/backward_references.o bin/obj/c/enc/backward_references_hq.o bin/obj/c/enc/bit_cost.o bin/obj/c/enc/block_splitter.o bin/obj/c/enc/brotli_bit_stream.o bin/obj/c/enc/cluster.o bin/obj/c/enc/compress_fragment.o bin/obj/c/enc/compress_fragment_two_pass.o bin/obj/c/enc/dictionary_hash.o bin/obj/c/enc/encode.o bin/obj/c/enc/encoder_dict.o bin/obj/c/enc/entropy_encode.o bin/obj/c/enc/histogram.o bin/obj/c/enc/literal_cost.o bin/obj/c/enc/memory.o bin/obj/c/enc/metablock.o bin/obj/c/enc/static_dict.o bin/obj/c/enc/utf8_util.o
c++ -Wl,-gc-sections bin/obj/c/common/ans_params.o bin/obj/c/common/constants.o bin/obj/c/common/context.o bin/obj/c/common/huffman_tree.o bin/obj/c/common/lehmer_code.o bin/obj/c/common/platform.o bin/obj/c/common/predict.o bin/obj/c/common/quant_matrix.o bin/obj/c/dec/ans_decode.o bin/obj/c/dec/brunsli_decode.o bin/obj/c/dec/context_map_decode.o bin/obj/c/dec/decode.o bin/obj/c/dec/histogram_decode.o bin/obj/c/dec/huffman_decode.o bin/obj/c/dec/huffman_table.o bin/obj/c/dec/jpeg_data_writer.o bin/obj/c/dec/state.o bin/obj/c/enc/ans_encode.o bin/obj/c/enc/brunsli_encode.o bin/obj/c/enc/context_map_encode.o bin/obj/c/enc/encode.o bin/obj/c/enc/histogram_encode.o bin/obj/c/enc/jpeg_data_reader.o bin/obj/c/enc/jpeg_huffman_decode.o bin/obj/c/enc/write_bits.o bin/obj/c/tools/cbrunsli.o
-lm -Lthird_party/brotli -lbrotli -o bin/cbrunsli
ld: unknown option: -gc-sections
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [cbrunsli] Error 1

Hello,

I am trying to package brunsli into openSUSE Tumbleweed Linux.
While building the package, RPMLINT warning show up as shown below.

libbrunsli0.1.x86_64: W: shared-lib-calls-exit /usr/lib64/libbrunslienc-c.so exit@GLIBC_2.2.5
[   37s] This library package calls exit() or _exit(), probably in a non-fork()
[   37s] context. Doing so from a library is strongly discouraged - when a library
[   37s] function calls exit(), it prevents the calling program from handling the
[   37s] error, reporting it to the user, closing files properly, and cleaning up any
[   37s] state that the program has. It is preferred for the library to return an
[   37s] actual error code and let the calling program decide how to handle the
[   37s] situation.

I personally don't know what this meant, is it something to worry?
Also, is it intended to not have share lib version?
Full log here: https://build.opensuse.org/package/live_build_log/home:andythe_great/brunsli/openSUSE_Tumbleweed/x86_64

Thanks

Win10, Chrome 76.0.3809.100 x64

image: https://drive.google.com/file/d/1N1tRXP45Fj_BgeB-qQnmuV3hl2X8gTxl/view?usp=sharing

brunslicodec-wasm.js:69 Uncaught abort(OOM). Build with -s ASSERTIONS=1 for more info.

Arithmetic coding is a part of the JPEG standard. All the related patents are expired years ago. Most of the software don't support such JPEGs, but still it would be nice if it will be supported as input for lossless conversion from JPEG to JPEG XL, just to be sure that any valid JPEG can be converted losslessly.

As far as I understand, it won't be possible to restore original JPEG file in this case, because all the stored information about original JPEG file is Huffman related. But I think, it would be enough to preserve just image data without possibility to restore exactly original JPEG file (like in #80).

I posted some examples of arithmetic coded JPEGs here.

We require tagged releases when building scientific software. The existing tag is five years old.

Thank you for making your software available

The .so file is not versioned which does not match Linux distributions expectations and prevent to add brunsli package to most of them.
Could you add versions to .so file, please?

I add

#define BRUNSLI_EXPERIMENTAL_GROUPS 1

in c/tools/cbrunsli.cc

then try to compress a jpg, but a core dumped. The jpg picture is in the attachment.

c/enc/write_bits.cc:26 (~Storage)
Aborted (core dumped)

Here is a list that I would like to see implemented.
It can be either Image => BMP/PNG intermediate => JXL or Image => direct to JXL

• BMP
• PNG
• WebP
• BPG
• HEIF (HEIC/AVCI)
• AVIF
• FLIF
• Other JPEG formats

subj

On windows, dll require the API to be explicitly exported.

Could you please specify which headers should be public and exported?

Thanks!

Currently all the CI jobs can't find don't run the ctests, because no tests are found:

Run cd out && ctest -j 2
Test project /home/runner/work/brunsli/brunsli/out
No tests were found!!!

Dear All:
In the brunsli's code, it seems run like below:
1) context modeling, let's say we get 27 x 9 bins(histogram) for example.
2) cluster the histogram(27x9) to reduce the context bin's count, for example after cluster, 27x9 reduce to 5.
3) build 5 ANS table.
4) during encode, get a context value and map it to one of the 5 ANS table then perform ANS encode(PutSymbol).

it seems that we need know all of the DCT coefficients, and then we perform encode with ANS, in other words, when ANS table is built, for each symbol(18 in brunsli) the probability never changed, it use static probability distributions. am i right?

As you know, in dropbox's lepton there is a arithmetic encoder to encode the residual of the DCT coefficents, the arithmetic encoder can adjust the probability internally, see below red line:

so, in this way, during lepton's arithmetic encoding, the probability is dynamic adjust locally.

my concern is since ANS is using static probability distributions instead of dynamic adjust it locally, the finally compression ratio maybe will get worse, am I right? if it is true, could we adjust probability dynamically in brunsli's ANS to improve the compression ratio?

I just thought this might interest researchers https://github.com/hilloc-submission/hilloc

Many packages and integrators prefer or even require at least one release tagged to integrate a library, so that regressions don't impact them and to have a reliable source to download from. It would be nice if Brunsli could tag a (pre-)release as soon as possible, either 0.1 or 1.0, to accelerate adoption and thereby community involvement.

We ported Brunsli algorithm to Android system using Cmake and NDK and have built an app to verify the compress and decompress, it works well.

But when we enable the multithread mode add_compile_definitions(BRUNSLI_EXPERIMENTAL_GROUPS) the algorithm can't finish the compress/decompress task, it runs forever.

It might be possible to expand to animation formats

• GIF
• APNG
• HEIFS (HEICS/AVCS)
• FLIF
• WebP
• BPG
• AVIF

Is anyone able to build this as a Windows Binary? Cmake fails and the current Windows binary is very old/artifact expired...

Would like to know if Python and Golang is supported.
Python apps that are common:

• Pillow
• OpenCV
• ffmpeg
• QtPy with PyQt5 or PySide2
• (others)

As far as I can see, JXL created by Brunsli can be extracted exactly to the same (byte-by-byte) JPEG file. Could compression ratio be a bit better if we recompress just JPEG image data losslessly, without storing additional information how original JPEG was encoded?

Looks good, can the image resolution still be displayed?

https://alapblog.hu/wp-content/uploads/2019/08/regio-07-002.jpg

Message only :
Processing alapblog_regio-07-002.jpg as JPEG
Input size: 4354115

No .jxl output.

Looks like an interoperability problem. Steps to reproduce:

Step 1. We'll use the following “why is the Internet like this” meme (quoted from “Oshi no Ko” manga) as an example JPEG:

Download it.

Step 2. Let's confirm that the problem is (relatively) recent.

Visit the “Actions” section and open the page of CMake build 77. Its “windows” artifact looks like that:

windows.zip

Unpack cbrunsli.exe from this ZIP to the subdirectory where the meme from step 1 resides.

Run the command cbrunsli.exe "why is the Internet like this.jpg" in that subdirectory.

The following why is the Internet like this.jpg.brn file (55141 bytes long) is generated (see in the attached ZIP):

why is the Internet like this 1.zip

Dragging and dropping that file to https://brunsli.dev/ generates a decoded picture:

Step 3. Let's confirm where the problem started.

Visit the “Actions” section and open the page of the next CMake build: CMake build 78. Its “windows” artifact looks like that:

windows78.zip

Unpack cbrunsli.exe from this ZIP to the subdirectory where the meme from step 1 resides (replacing cbrunsli.exe that was unpacked by the previous step).

Run the command cbrunsli.exe "why is the Internet like this.jpg" in that subdirectory.

The following why is the Internet like this.jpg.brn file (55483 bytes long) is generated (see in the attached ZIP):

why is the Internet like this 2.zip

Its difference from the result of step 2 is made obvious by its different filesize.

Dragging and dropping that file to https://brunsli.dev/ generates an error message (“Reconstuction failed”):

Step 4. Let's confirm that the problem persists.

Visit the “Actions” section and open the page of the latest CMake build: CMake build 85. Its “windows” artifact looks like that:

windows85.zip

Unpack cbrunsli.exe from this ZIP to the subdirectory where the meme from step 1 resides (replacing cbrunsli.exe that was unpacked by the previous step).

Run the command cbrunsli.exe "why is the Internet like this.jpg" in that subdirectory.

Confirm that the generated file why is the Internet like this.jpg.brn has the same filesize (55483 bytes) as the file generated by step 3 and that files' contents do not differ.

Dragging and dropping that file to https://brunsli.dev/ generates the same error message (“Reconstuction failed”).

(However, newer dbrunsli.exe correctly decodes files generated by the corresponding newer cbrunsli.exe.

Does that mean that https://brunsli.dev/ needs an update?)

Looks like an interoperability problem. Steps to reproduce:

Step 1. We'll use the following picture of the “Dude! Let Me In! I'm A Fairy!” meme:

Step 2. When I drag-and-drop the meme at https://google.github.io/brunsli/ to the transcoder, the following JPEG XL file is created:

(ZIP archive of Brunsli's JXL file)

When I drag the JPEG XL file back to the transcoder, the returned JPEG file is byte-by-byte equivalent of the original (step 1), I can confirm it using fc /b original.jpg decoded.jpg on Windows.

Step 3. This time I'll try to repeat step 2 using JPEG XL's reference software. I do not build it myself; I'll use an unofficial build (for Windows) distributed by user Jamaika on Doom9 forum; that build is originally hosted on sendspace.com, but here's a backup copy:

(ZIP archive of Jamaika's build of JPEG XL's reference software)

When I run cjpegxl_jxl --version it reports version 0.0.1-6334944f (seems to correspond to the latest commit in the public repo of JPEG XL's reference software).

When I run cjpegXL_jxl -s 9 on the original JPEG file on Windows, the following JPEG XL is created:

(ZIP archive of JXL file generated by Jamaika's build)

When I run djpegXL_jxl --jpeg on this result, the following JPEG file is decoded:

Its size (84 836 bytes) is larger than original's (80 496 bytes) and it lacks all of the Exif metadata of the original JPEG. However, ImageMagick's comparison magick compare -metric ae original.jpg decoded.jpg nul reports that these files are equal pixel-by-pixel, though not byte-by-byte.

Step 4. When I drag-and-drop the JPEG XL file from step 3 (generated by Jamaika's build) at https://google.github.io/brunsli/ to the transcoder, the following error appears:

Brunsli cannot decode the JPEG file that was compressed in JPEG XL's reference software's unofficial build.

Step 5. Vice versa, when I use Jamaika's build's djpegXL_jxl --jpeg on Brunsli's JPEG XL file (from step 2), the following JPEG file is decoded:

It has the same number of bytes (84 836 bytes) that Jamaika's build's decoded from its own JPEG XL (step 3), but much worse quality.

The above steps were performed in Windows 7 Professional 64-bit (Service Pack 1) running on Intel Core i7-3770 CPU.

The results may indicate that Jamaika's build is built incorrectly.

The results may also indicate that JPEG XL's reference software's authors tried to fix their own version of #80 but failed (the decoded JPEG file started to retain only pixel-by-pixel compatibility instead of its former byte-by-byte, but what's the point if the decoded file becomes larger, and lacks metadata, and does not have full interoperability with https://google.github.io/brunsli/ then).

Would someone kindly make their own build of JPEG XL's reference software and check what's really going on?

The certificate was valid from 05/Mar/2023 through 03/Jun/2023.

Is it possible to make brunsli handle 12bit per sample JPEGs?
The standard libjpeg has a 12bit build mode, which is used sometimes. It is still using Huffman encoding.

Thanks

OSS-Fuzz has found a bug in this project. Please see https://oss-fuzz.com/testcase?key=4850079462653952 for details and reproducers.

This issue is mirrored from https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66583 and will auto-close if the status changes there.

If you have trouble accessing this report, please file an issue at https://github.com/google/oss-fuzz/issues/new.

When compiling Brunsli with Visual Studio 2019 it currently throws 74 warnings.

A display of the ratio would be quite good