While reading the step tests, I noticed that the format is stored globally instead of for each convert step. So if multiple processing steps are given, the format from the last step is used for all intermediate steps. This is not such a good idea if the format is lossy, like jpeg.

Maybe it would be better to store the format with every intermediate step or force a lossless format for intermediate steps.

In most cases this should be avoidable, because pretty much all transformations can be done in a single convert call.

One test that shows the problem:

  test "works with local files" do
    steps = [["ff", "/app/foo.jpg"],
             ["p", "convert", "-thumbnail 273x273^^ -gravity center -crop 273x273+0+0 +repage -draw 'polygon 0,0 273,273 273,0 fill none matte 135,135 floodfill'", "png"],
             ["p", "thumb", "892x320#"],
             ["e", "jpg"]]
    commands = %Steps{file: "/app/foo.jpg",
                convert: "#{Config.convert_command} -'[0]' -thumbnail 273x273^^ -gravity center -crop 273x273+0+0 +repage -draw 'polygon 0,0 273,273 273,0 fill none matte 135,135 floodfill' jpg:- | #{Config.convert_command} - -resize 892x320^^ -gravity center -crop 892x320+0+0 +repage -strip jpg:-",
                format: "jpg"}
    assert(commands == Steps.deserialize(steps))
  end

In the first processing step, png is specified as format, but the convert command used jpg instead.

The current version of the dragonfly gem serialized the encode processor as:

 ["p", "encode", "jpg", "-quality 50"]

But dragonfly-server uses:

 ["e", "jpg", "-quality 50"]

Is there a reason for the different syntax or is it a bug in the elixir implementation?

I already have a fix for this locally, which currently supports both variants, but I'd like to know wether I should keep or drop support for the "e" syntax.

There are many scenarios, in which the same images are requested from the backend multiple times in a short period:

• Same file fetched as download and for thumbnail (eg. PDF thumb)
• Master/detail/zoom: In ecommerce applications it's common to have the same image as a thumb on the list page, than another thumb and small image on the detail page and maybe even a large/original size image for a zoom view
• Generating smaller images for different screen resolutions (eg. generating smaller images for phones than desktops)
• Generating different formats for different user agents (eg. jpeg, webp, jpeg2000)

If a new original is added, it is likely to stay "hot" for some time, until all of the common variants are generated.

My idea is to transform the fetch operations into the same job syntax and cache them the same way as generated versions. There are some caveats when generating cache keys, eg. signed urls would need their cache keys to be generated without signing or they'd be useless. Ideally there would also be the same cache key used for a simple fetch/download job from a client and the same operation cached as part of eg. a thumbnail job.

It would probably also be a good idea to be able to configure a different cache server for backend fetches to avoid cache trashing and be able to allocate cache size separately for originals.

What do you think about the idea?

I'm a bit undecided, because in many cases this can just as well be built using a simple reverse proxy in nginx and pointing the http_engine_host at that proxy. Then again the same can be said for the caching of job results. In my current deployments of the rack dragonfly I've just put an nginx proxy in front of it, so the rack app is idle unless new images are added.

See attached image as an example.