technopagan / adept-jpg-compressor Goto Github PK

From what I have seen apparently most of the work done is based on the characteristics of the Luma (Y) component and only the "global" quality setting is eventually decreased at no point the script tries to reduce Chroma quality independently of Luma (or even better Cr and Cb separately) since it has no way to measure if the result is qualitatively worse or not.

(submitted by Frédéric Kayser, [email protected])

Right now, processing e.g. a high resolution image with 64x64 pixel tiles can lead to a slightly larger output image than the input. This is of course counter-productive and wasteful.

Adept neeeds to implement a check to see if the output is actually smaller than the input and only of that's true, go ahead and save the output image (and overwrite the input image if the config is set this way). Adept should do nothing in normal mode if the output is bigger (aside from deleting the output) and in verbose mode actually apologize to the user for wasting their time.

Right now, jpegoptim < v. 1.3, its newest version which is very fresh and still will take some time until 1.3 hits all major Linux repositories, cannot save JPG as progressive.

JPEGtran can save progressive JPGs. As we are already using JPEGrescan after jpegoptim: Make a round of tests of JPEGrescan vs. JPGtran and if JPEGtran is a valid alternative, then use it and make sure output JPGs are set to be progressive per default.

Command:
jpegtran -optimize -progressive -outfile outputfilename.jpg

Besides having a variable with in integer in the configuration section to define default (NOT high!) JPG compression, aka quality, it would be great if we had an alternative setting like "inherit", which invokes a simple identify to read the quality setting of the incoming JPG that's about to be processed by Adept.

Since the merge of #31, I'm seeing errors like below:

stat: cannot read file system information for '\'%z\'': No such file or directory
./bin/adept.sh: line 181: File:: command not found

I do have GNU stat installed.

Right now, we're using "bc -l" to make all calculations within Bash. The "-l" switch however might not be necessary and in some situations, we might get around using "bc" altogether. This might slightly improve performance.

Currenty, the functions calling upon ImageMagick commands assume that everything goes well within these commands. There is no return value we can unit-test easily.

What we should implement is checking for the exit state of each command and see if they are all Null (OK) and alternatively, abort execution as soon as one of them returns anything else but Null.

Of course, even when aborting, we will still be running cleanup in order not to leave any temporary files behind and we should use the opportunity to provide the user with helpful debugging information (Anything is better than some NonNull-error code!).

Make the process of Adept more accessible to users by creating a series of step-by-step demo images and putting them into the documentation for easy consumption.

Currently, /dev/shm is hardcoded as storage path within Adept. This is not very portable.

If we create a simple test function which checks for the availability of /dev/shm, /run/shm et al. and then falls back onto /tmp/ for storage if nothing else was found, Adept instantly becomes more portable, esp. for OSX.

Why are tiles square?

Apparently their minimum size does not take chroma subsampling in consideration, for instance if I have a small 256 by 128 pixels JPEG with 4:2:0 or 4:2:2 chroma subsampling AJC will pick an optimaltilesize of 8, and work on tiles that hold less than an entire native Cb or Cr matrix.

I would suggest to set the smallest tile size to the size of an entire macroblock (or MCU in JPEG speak) based on the current JPEG being optimized, in these cases 8 by 16 and 16 by 16.

(submitted by Frédéric Kayser, [email protected])

After merging the saliency mapper and binary search functionality for threshold values, the unit tests suite is currently out of date and thus broken.

The old Sobel unit tests have to be removed and replaced by tests that check the new functions for the saliency mapper.

Tile size is the no.1 factor in Adept runtime performance. Running Adept on high resolution images with the default tile size of 64x64 pixel will often take > 20s.

Yet, high resolution images most often do not contain a greater number of different important aspects within a single image, but are most likely depicting the same contents as lower resolution shots. Therefore, it is a good idea to dynamically adapt the tile size in stepf of multiples of 8 (64,128,256...) according to the incoming images resolution.

This will greatly enhance runtime while not reducing compression success and make Adept far more interesting for automated image compression.

While using IM's identify to check on the validity of an incoming JPEG image has the benefit of not intrudicing yet another tool & thus dependency, it is prone to return incorrect results: if a JPEG image is only partially transferred but its headers are intact, IM identify will vaildate it as a valid file.

Let's do extensive tests to make sure that no invalid files can be processed through Adept. In case IM's identify should not match our expectations, we might switch to a tool such as jpeginfo.

See:

• https://github.com/tjko/jpeginfo
• http://manpages.ubuntu.com/manpages/hardy/man1/jpeginfo.1.html

After making the code more modular & thus testable, it is now time to set up a test-suite of unit tests via the BATS unit testing framework: https://github.com/sstephenson/bats

Since basically what AJC is doing is simulating ROI (Region Of Interest) in a file format that does not have this feature (the Quantization Tables are applied to the entire image), the result is that it actually shaves off some DCT coefficients (rounds them down or turns them to zero) from time to time, why not doing it with a dedicated tool directly in memory at the DCT level instead of playing around with hundreds of files and half a dozen tools? This way it could be integrated right into a JPEG encoder that takes optimization at heart like Mozjpeg.

(submitted by Frédéric Kayser, [email protected])

To further reduce bytesize, Adept needs a more granular approach to compression.

As of now, the only decisive factor if we can apply high compression to a tile is the black-white median decimal "0.825". If a tile ranks below that threshold, it gets compressed more heavily.

We can introduce a multi-threshold step system here that e.g. applies no compression, a little compression (~q=75) or high compression of the median is zero. This way, Adept may be able to save more bytes.

In several places, we are relying on "sort -V" to sort out file / tile processing order.

While this is convenient, it breaks OSX compatibility (although that is a minor concern) and causes additional processing time.

If we made sure that ALL naming schemes for all tiles created during Adept runtime would be sorted perfectly without additional sorting afterwards, we could increase compatibility and cut down processing time.

Hi there!

Currently I compile my ImageMagick executable myself and I use the following configure options:

./configure \
  --disable-static \
  --enable-shared \
  --with-jpeg \
  --with-png \
  --with-quantum-depth=8 \
  --with-rsvg \
  --with-webp \
  --without-bzlib \
  --without-djvu \
  --without-dps \
  --without-fftw \
  --without-fontconfig \
  --without-freetype \
  --without-gvc \
  --without-jbig \
  --without-jp2 \
  --without-lcms \
  --without-lcms2 \
  --without-lqr \
  --without-lzma \
  --without-magick-plus-plus \
  --without-openexr \
  --without-pango \
  --without-perl \
  --without-tiff \
  --without-wmf \
  --without-x \
  --without-xml \
  --without-zlib

Sadly, running the adept unit tests (great thing) fails:

 ✓ Find Tools
 ✓ Find Tools with path resolving
 ✓ Validate Input JPEG
 ✓ Read Image Dimension
 ✓ Optimize Tile Size
 ✓ Optimize Black/White Threshold
 ✓ Slice Image into Tiles
 ✗ Retrieve Black/White Median
   (in test file /usr/local/src/adept-jpg-compressor/unittests/tests_adept.bats, line 56)
 ✓ Calculate Tile Count for Reassembly

9 tests, 1 failure

Which options are really necessary to compile ImageMagick and use it with adept?

This project is already included in homebrew-headonly - although it hasn't been updated to the new version AFIK - however it would be awesome if adept could make it into mainline Homebrew.

If this is in a "stable" state, would you consider the following so that this project may be considered for inclusion in Homebrew?

1. Tag a stable release version as per Homebrew's requirements

After implementing automated black-white threshold detection and automatic tile-size adaption, the perceived runtime performance has dropped.

In the course of implementing unit tests, let's also implement function runtime measurements via "time" to see which processes take up the lion's share of total runtime. This will guide us to the parts of Adept's code that neeed priority in optimization.

After installing the dependencies, I am getting the following on any jpg:

identify: no decode delegate for this image format `JPEG' @ error/constitute.c/ReadImage/501.

ubuntu@ubuntu-desktop:~/work/adept-jpg-compressor$ /usr/bin/convert -version
Version: ImageMagick 6.7.7-10 2014-03-06 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ which jpegrescan
/bin/jpegrescan
ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ which SaliencyDetector
/bin/SaliencyDetector
ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ which jpegoptim
/usr/bin/jpegoptim
ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ uname -a
Linux ubuntu-desktop 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ ls -l | grep "test.jpg"
-rw-rw-r-- 1 ubuntu ubuntu 59564 Dec 13 19:51 test.jpg
ubuntu@ubuntu-desktop:/work/adept-jpg-compressor$ bash adept.sh test.jpg
identify: no decode delegate for this image format `JPEG' @ error/constitute.c/ReadImage/501.

Thanks in advance for any help. My apologies if this is something simple that I've overlooked.

Public Domain Mark 1.0:

https://github.com/ImageProcessing-ElectronicPublications/jpegquant/blob/master/LICENSE

https://creativecommons.org/licenses/mark/1.0/

Not?

Write a script which finds each function and within it detects global and local variables. Then it outputs a graphed list of:

functionname

• global variables created & altered by this function
  --- globalvar_name (number of alterations within function)
  --- globalvar_name (number of alterations within function)
• local variables created & altered by this function
  --- localvar_name (number of alterations within function)
  --- localvar_name (number of alterations within function)

This output will enable us to find unnecessary global variables, illogical redefinitions of variables etc.

Make a one line installation script that easily installs it with dependencies on Ubuntu 12.04 and 14 and this is going to become the default way to compress jpegs for web.
(Bonus) options for resizing. the images.
(Bonus 2) options for batch compression.

Make default compression a variable parameter right alongside the high compression variable.

This will enable people to use techniques like the double-density JPG processing for HiDPI devices.

Also, it's an important step towards making Adept respect the quality setting of the incoming image.

To further optimize compression rate, we should implement automated optimization for Huffman tables.

Suggested reading: http://www.impulseadventure.com/photo/optimized-jpeg.html

Maybe JPEGtran can help:
http://www.phpied.com/installing-jpegtran-mac-unix-linux/
http://www.mikebrittain.com/blog/2010/01/27/batch-processing-your-jpegs-with-jpegtran/

We need to compare it to JPEGrescan though!

To make it easier for people to assess if they can successfully run Adept, make a list of platforms the script has been tested on already and put it into the script's header section next to the requirement software list.

Blurring, also known as smoothing, is a great way to prepare an image or parts of an image for JPEG compression because the algorithm can compress these smoothed areas more efficiently.

While blurring per default introduces recognisable tile-blocks in a reassembled image due to noise / texture etc, we might be able to create a set of rules to decide if a tile is capable of receiving smoothing to enhance compression.

While thresholds and other detection mechanisms still need to be defined, this might be a good way to further reduce JPEG bytesize.