Abysmal compression speed for x-ray from Silesa about brotli HOT 6 CLOSED

As Brotli is deliberately designed to be asymmetric (favouring decompression speed over compression speed) it would be interesting to know where it ranks on decompression time on that image.

from brotli.

Decompression speed is about right. Look at the benchmark I linked to. The results table can be sorted by decompression speed; just click on the "Decompression Speed" header of the results table.

The problem is that this is a huge outlier for brotli's compression speed. It's so slow that it basically amounts to a DoS vulnerability. It takes an hour to compress an 8.1 MiB file (on a Xeon E3-1225 v3), which is 5.62 KiB/s. The next slowest file (ooffice, which is a 5.87 MiB DLL from OpenOffice.org 1.01) is 59.9 KiB/s.

I think a lot of the compression speeds are pretty bad when you look at Brotli's competition (LZHAM is probably the closest competitor—it offers pretty similar ratios and decompression speed but much more stable, and generally faster compression). But 5.62 KiB/s is just unacceptable.

from brotli.

It's probably not unrelated that brotli also allocates a lot of memory for this file. One of my ARM boards (with 1 GiB memory) mercifully terminates with a std::bad_alloc exception (hopefully the others will, as well). An old laptop with 1 GiB memory and plenty of swap space, OTOH, is still happily thrashing after 50 hours. Peak memory usage for that process right now is 1790520 KiB, current is 1266228. For an 8.1 MiB file.

from brotli.

I believe this has been fixed.
I've measured compression of x-ray on odroid-c1 with broltli (level 11) - it took 5 min 41 sec, so the speed is about 24 KiB/s. On rpi-2 it would take 10-15 minutes to compress.

With lower level it works much faster. E.g. with level 1 it takes less than 2 sec; on level 9 it works 1 min 24 sec.

Probably it is time to reevaluate brotli compression...

from brotli.

FWIW, time -f "%M" shows that brotli uses ~310MB of memory when compress x-ray (on level 11)

from brotli.

I just updated the benchmark, and this issue does seem to be fixed.

Brotli has made a lot of progress in the last few months.

from brotli.