kemonomachi / experhash Goto Github PK

When using this package as a dependency in another project, the priv dir is not created.
This causes the symlink into the build directory to not be created by mix.
The solution is:

• Add a .keep file in the priv dir.
• Remove priv from .gitignore and instead explicitly ignore the generated bin file priv/experhash.

The non-wrapping version seems to be slightly more accurate. From Neal Krawetz's blog:

"I have tried a few variations of the dHash algorithm. For example, David's initial proposal used an 8x8 image and wrapped the last comparison (computing the pixel difference between P[0] and P[7] for the 8th comparison). This actually performs a little worse than my 9x8 variation (more false-positives), but only by a little."

A bit neater than having to use the Github version.

Make sure documentation can be generated and looks good.

Email Convo about the issue

For DD-Hash.

for(int i = 0; i < SIZE; ++i) {
for(int j = 0; j < SIZE; ++j) {
row_hash[i] |= (pixels[i*SIZE + j] < pixels[i*SIZE + j+1]) << (7-j);
        col_hash[i] |= (pixels[i + j*SIZE] < pixels[(i + (j+1)*SIZE)%PIXEL_COUNT + j/8]) << (7-j);
      }
    }

here you are shifting by (7-j)

If we change the size from 8 to 32 or 64, how will that effect on the algorithm.

Hi Zubair,

I'm using vectors of 8-bit integers to assemble the hash, which means
that the shifting distance (7-j) should be independent of the size.
Currently it is not, and it is not possible to change the size because
of that. This needs fixing.

I'll need to look it over before I can give you a definite answer. It's
been a while since I wrote this. If possible, could you open an issue
about this on Github? [1] Feel free to include this mail.

I'm going out of town for the Easter holiday soon, so I won't have time
until next week. I'll get back to you next weekend. If you have any
additional questions, please feel free to ask.

• Ookami

- Need to work on that

Hey I'm not really sure where to ask this, or if this is the appropriate place but figured you may have the most experience to address this or at least point me in the right direction.

I installed Imagemagick(7) through brew on MacOS Sierra (10.12) with seemingly no issue. I'm able to run magick in the command line etc. However when I try to run Experhash I receive the following error:

dyld: Library not loaded: /usr/local/lib/libMagick++-6.Q16.5.dylib
Referenced from: /Users/plato/Desktop/test_app/api/_build/test/lib/experhash/priv/experhash_port
Reason: image not found

When I go and check the /usr/local/lib/ directory I see the following files exist:

libMagick++-7.Q16HDRI.4.dylib
libMagick++-7.Q16HDRI.dylib

but not libMagick++-6.Q16.5.dylib which I assume makes sense since I installed 7. Now I guess the confusion I have is that it seems the Experhash results are different than running Imagemagick directly from the command line which leads me to believe it may have to do with the makefile (https://github.com/kemonomachi/experhash/blob/master/Makefile#L10).

If it helps at all when I run,
Magick++-config --cppflags --cxxflags --ldflags --libs

in the command line I get the following back:

-DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/usr/local/Cellar/imagemagick/7.0.7-30/include/ImageMagick-7
-DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/usr/local/Cellar/imagemagick/7.0.7-30/include/ImageMagick-7
-L/usr/local/Cellar/imagemagick/7.0.7-30/lib -lMagick++-7.Q16HDRI -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI
-L/usr/local/Cellar/imagemagick/7.0.7-30/lib -lMagick++-7.Q16HDRI -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI

And if it helps here are my environment variables within ~/.zshrc.

export PATH="/usr/local/bin:$PATH"
export MAGICK_HOME=“/usr/local/Cellar/imagemagick/7.0.7-30”
export PATH="$MAGICK_HOME/bin:$PATH"
export DYLD_LIBRARY_PATH="$MAGICK_HOME/lib/"
export PKG_CONFIG_PATH="$MAGICK_HOME/lib/pkgconfig"

Tl;DR
It seems like the Makefile is referencing the wrong dylib and I'm not sure how to make it point to the "right" Imagemagick libraries which work when called straight from the command line.

• Remove GenServer parameter from the hamming_distance function in experhash.ex and do defdelegate like for the other functions (deprecate old one to be nice)
• Remove C++ version

When make fails with a compile error or similar, mix test just keeps on trucking, using the old executable. This annoys me.

I wrote a Hamming distance function that doesn't need to go all the way to GenServer, C++ and back. Maybe you want to use it? It allows to run the image comparison in parallel and is much faster.

  def hamming_distance(hash1, hash2) do
    {:ok, hamming_value_distance(:binary.bin_to_list(hash1), :binary.bin_to_list(hash2), 0)}
  end

  defp hamming_value_distance([], [], sum), do: sum
  defp hamming_value_distance([val1 | tail1], [val2 | tail2], sum) do
    value = val1 ^^^ val2
    %{^value => distance} = %{0 => 0, 1 => 1, 2 => 1, 3 => 2, 4 => 1, 5 => 2, 6 => 2, 7 => 3, 8 => 1, 9 => 2, 10 => 2, 11 => 3, 12 => 2, 13 => 3, 14 => 3, 15 => 4, 16 => 1, 17 => 2, 18 => 2, 19 => 3, 20 => 2, 21 => 3, 22 => 3, 23 => 4, 24 => 2, 25 => 3, 26 => 3, 27 => 4, 28 => 3, 29 => 4, 30 => 4, 31 => 5, 32 => 1, 33 => 2, 34 => 2, 35 => 3, 36 => 2, 37 => 3, 38 => 3, 39 => 4, 40 => 2, 41 => 3, 42 => 3, 43 => 4, 44 => 3, 45 => 4, 46 => 4, 47 => 5, 48 => 2, 49 => 3, 50 => 3, 51 => 4, 52 => 3, 53 => 4, 54 => 4, 55 => 5, 56 => 3, 57 => 4, 58 => 4, 59 => 5, 60 => 4, 61 => 5, 62 => 5, 63 => 6, 64 => 1, 65 => 2, 66 => 2, 67 => 3, 68 => 2, 69 => 3, 70 => 3, 71 => 4, 72 => 2, 73 => 3, 74 => 3, 75 => 4, 76 => 3, 77 => 4, 78 => 4, 79 => 5, 80 => 2, 81 => 3, 82 => 3, 83 => 4, 84 => 3, 85 => 4, 86 => 4, 87 => 5, 88 => 3, 89 => 4, 90 => 4, 91 => 5, 92 => 4, 93 => 5, 94 => 5, 95 => 6, 96 => 2, 97 => 3, 98 => 3, 99 => 4, 100 => 3, 101 => 4, 102 => 4, 103 => 5, 104 => 3, 105 => 4, 106 => 4, 107 => 5, 108 => 4, 109 => 5, 110 => 5, 111 => 6, 112 => 3, 113 => 4, 114 => 4, 115 => 5, 116 => 4, 117 => 5, 118 => 5, 119 => 6, 120 => 4, 121 => 5, 122 => 5, 123 => 6, 124 => 5, 125 => 6, 126 => 6, 127 => 7, 128 => 1, 129 => 2, 130 => 2, 131 => 3, 132 => 2, 133 => 3, 134 => 3, 135 => 4, 136 => 2, 137 => 3, 138 => 3, 139 => 4, 140 => 3, 141 => 4, 142 => 4, 143 => 5, 144 => 2, 145 => 3, 146 => 3, 147 => 4, 148 => 3, 149 => 4, 150 => 4, 151 => 5, 152 => 3, 153 => 4, 154 => 4, 155 => 5, 156 => 4, 157 => 5, 158 => 5, 159 => 6, 160 => 2, 161 => 3, 162 => 3, 163 => 4, 164 => 3, 165 => 4, 166 => 4, 167 => 5, 168 => 3, 169 => 4, 170 => 4, 171 => 5, 172 => 4, 173 => 5, 174 => 5, 175 => 6, 176 => 3, 177 => 4, 178 => 4, 179 => 5, 180 => 4, 181 => 5, 182 => 5, 183 => 6, 184 => 4, 185 => 5, 186 => 5, 187 => 6, 188 => 5, 189 => 6, 190 => 6, 191 => 7, 192 => 2, 193 => 3, 194 => 3, 195 => 4, 196 => 3, 197 => 4, 198 => 4, 199 => 5, 200 => 3, 201 => 4, 202 => 4, 203 => 5, 204 => 4, 205 => 5, 206 => 5, 207 => 6, 208 => 3, 209 => 4, 210 => 4, 211 => 5, 212 => 4, 213 => 5, 214 => 5, 215 => 6, 216 => 4, 217 => 5, 218 => 5, 219 => 6, 220 => 5, 221 => 6, 222 => 6, 223 => 7, 224 => 3, 225 => 4, 226 => 4, 227 => 5, 228 => 4, 229 => 5, 230 => 5, 231 => 6, 232 => 4, 233 => 5, 234 => 5, 235 => 6, 236 => 5, 237 => 6, 238 => 6, 239 => 7, 240 => 4, 241 => 5, 242 => 5, 243 => 6, 244 => 5, 245 => 6, 246 => 6, 247 => 7, 248 => 5, 249 => 6, 250 => 6, 251 => 7, 252 => 6, 253 => 7, 254 => 7, 255 => 8}
    distance + hamming_value_distance(tail1, tail2, sum)
  end

Experhash is working fine locally, but in production I am getting an error. I'm using Distillery 2 and Docker to make releases on Ubuntu 16.04. Imagemagick is installed on both the docker image and on the production server, and mix compile runs during the build.

Genserver times out, but I believe that is just a symptom.

experhash-0.1.1/priv/experhash_port: Syntax error: word unexpected (expecting ")")

Phoenix 1.4.3
Elixir 1.8.1
Distillery 2.0 using the new Elixir provider

11:07:51.097 request_id=FZXvw-ylKvX9vlEAAADx [debug] Processing with DistanceWeb.DocumentController.new/2
  Parameters: %{}
  Pipelines: [:browser]
11:07:51.098 request_id=FZXvw-ylKvX9vlEAAADx [info] Sent 200 in 3ms
11:08:02.951 request_id=FZXvxq9fFd2k4d0AAAEB [info] POST /documents
11:08:02.955 request_id=FZXvxq9fFd2k4d0AAAEB [debug] Processing with DistanceWeb.DocumentController.create/2
  Parameters: %{"_csrf_token" => "RysAfnR6Yk4BenoDIB0KAgxlDDU8JgAA3n//DH7xH77LcEr0EP8EVQ==", "_utf8" => "✓", "document" => %{"file" => %Plug.Upload{content_type: "image/png", filename: "a028.png", path: "/tmp/plug-1555/multipart-1555412882-15036320304116-1"}}}
  Pipelines: [:browser]
/home/deploy/lib/experhash-0.1.1/priv/experhash_port: 1: /home/deploy/lib/experhash-0.1.1/priv/experhash_port: Syntax error: word unexpected (expecting ")")
11:08:12.959 request_id=FZXvxq9fFd2k4d0AAAEB [info] Converted exit {:timeout, {GenServer, :call, [#PID<0.1680.0>, {:command, {:hash, :dd, "media/1555412882-15036320304116-1-a028-orig.png", 8}}, 10000]}} to 500 response
11:08:12.961 request_id=FZXvxq9fFd2k4d0AAAEB [error] Process #PID<0.1678.0> terminating
** (exit) an exception was raised:
    ** (ErlangError) Erlang error: {{:timeout, {GenServer, :call, [#PID<0.1680.0>, {:command, {:hash, :dd, "media/1555412882-15036320304116-1-a028-orig.png", 8}}, 10000]}}, {DistanceWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "redacted", method: "POST", owner: #PID<0.1678.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["documents"], path_params: %{}, port: 80, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "en-US,en;q=0.5"}, {"connection", "upgrade"}, {"content-length", "30026"}, {"content-type", "multipart/form-data; boundary=---------------------------7719490031198181894888934539"}, {"cookie", "_distance_key=SFMyNTY.g3QAAAABbQAAAAtfY3NyZl90b2tlbm0AAAAYdEUvUTAyVTZJTU1PQ1h4Mkk1NHBqdz09.Ojpdcrhb9AN8xoU4Sjxl5rR-uCsb6wFd-FJlt6vuz4Y"}, {"host", "redacted"}, {"referer", "https://redacted/documents/new"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0"}, {"x-cluster-client-ip", "97.89.62.78"}, {"x-forwarded-for", "97.89.62.78"}], request_path: "/documents", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}}
(phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:49: Phoenix.Endpoint.Cowboy2Handler.init/2
(cowboy) /redacted/distance/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2

Hamming distance tests could use some love. At least make sure the tests hit all edge cases. Perhaps do exhaustive testing on ultra-short hashes (8 bits? 16 bits?)

Remove this manual step.
Don't commit any binary files (like #2 suggested)