Problems when using the plugin with `amqp.node` about rabbitmq-message-deduplication HOT 14 CLOSED

I'll make available both releases tonight then.

from rabbitmq-message-deduplication.

I couldn't find anything in the AMQP specifications that mandates the use of strings, so I went ahead and reported this issue to amqp.node as well:
amqp-node/amqplib#448

from rabbitmq-message-deduplication.

Greetings,

The node amqp library has been already problematic in the past, see issue #5.

These types of issue do not seem to arise with Erlang and Python libraries.

Nevertheless, I am not sure about whether it's a problem of the client library or of RabbitMQ itself. The broker seems to enforce strict type checking but the documentation doesn't seem to specify the type formatting of the headers.

I will contact the RabbitMQ community and double-check with them if the documentation is lacking.

As a temporay workaround, I'd suggest to use an integer (set to 1) instead of the boolean true. IIRC, the Queue is not sanitizing the parameters as the exchange does so, theoretically, it should get you going.

from rabbitmq-message-deduplication.

Thanks for the quick reply and for looking into this!

Yes, none of the node libraries seem to be well tested. I'm already playing with the thought of having my own Erlang service between RabbitMQ and the client.

I tried with integer 1 but unfortunately that didn't work either.

from rabbitmq-message-deduplication.

Can you please share the error you get on the client with that?

You should also see some message on the RabbitMQ logs, could you please share that as well?

from rabbitmq-message-deduplication.

There are no errors on both the client and RabbitMQ. All messages get enqueued. Here's the logs from docker-compose:

rabbitmq_1           | 2018-08-08 13:21:28.024 [info] <0.530.0> Management plugin started. Port: 15672
rabbitmq_1           | 2018-08-08 13:21:28.024 [info] <0.636.0> Statistics database started.
rabbitmq_1           | 2018-08-08 13:21:28.026 [info] <0.33.0> Application rabbitmq_management started on node rabbit@ef7f93221d8d
rabbitmq_1           |  completed with 4 plugins.
rabbitmq_1           | 2018-08-08 13:21:28.366 [info] <0.5.0> Server startup complete; 4 plugins started.
rabbitmq_1           |  * rabbitmq_management
rabbitmq_1           |  * rabbitmq_web_dispatch
rabbitmq_1           |  * rabbitmq_management_agent
rabbitmq_1           |  * rabbitmq_message_deduplication
rabbitmq_1           | 2018-08-08 13:22:16.778 [info] <0.669.0> accepting AMQP connection <0.669.0> (172.21.0.1:46046 -> 172.21.0.2:5672)
rabbitmq_1           | 2018-08-08 13:22:16.787 [info] <0.669.0> connection <0.669.0> (172.21.0.1:46046 -> 172.21.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
rabbitmq_1           | 2018-08-08 13:22:16.818 [info] <0.669.0> closing AMQP connection <0.669.0> (172.21.0.1:46046 -> 172.21.0.2:5672, vhost: '/', user: 'guest')

from rabbitmq-message-deduplication.

Looking through the entire log, there are some issues before that, though. Here's the log file.

Sorry for not having noticed those earlier. I'm surprised RabbitMQ starts up at all. The errors:

[error] <0.38.0> Loading of /var/lib/rabbitmq/mnesia/rabbit@ef7f93221d8d-plugins-expand/rabbitmq_message_deduplication-0.3.2/ebin/Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue.beam failed: badfile

and

[error] emulator beam/beam_load.c(1863): Error loading module 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':
   This BEAM file was compiled for a later version of the run-time system than 20.
   To fix this, please recompile this module with an 20 compiler.
   (Use of opcode 163; this emulator supports only up to 159.)

The Docker image used is https://github.com/docker-library/rabbitmq and reports:

root@ef7f93221d8d:/# erl
Erlang/OTP 20 [erts-9.3.3.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3.3.2

from rabbitmq-message-deduplication.

I was looking into the code and I was, in fact, a bit puzzled. Even if the value is passed as string it should still work because a string still evaluates to true in Elixir.

iex(1)> if "true", do: 1, else: 0 
1

The problem you are facing is that the .ez file you downloaded from the releases was compiled against a non backward compatible version of Erlang.

I'd suggest you to try building from the source using the same Erlang in the Docker container.

I'll figure out if I can build with different Erlang versions in the meantime.

from rabbitmq-message-deduplication.

I instead opted for upgrading Erlang in the Dockerfile and it indeed runs fine now!

My Dockerfile, for reference:

FROM rabbitmq:3-management

# Download latest Erlang from Erlang Solutions
RUN sed 's/stretch/testing/g' /etc/apt/sources.list
RUN sed 's/buster/testing/g' /etc/apt/sources.list
ADD https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/esl-erlang_21.0-1~debian~stretch_amd64.deb /
RUN apt update && apt install -y --no-install-recommends /esl-erlang_21.0-1~debian~stretch_amd64.deb

# Install deduplication plugin:
# https://github.com/noxdafox/rabbitmq-message-deduplication
ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.3.2/rabbitmq_message_deduplication-0.3.2.ez /plugins
ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.3.2/elixir-1.6.6.ez /plugins
RUN chmod 644 /plugins/*
RUN rabbitmq-plugins enable rabbitmq_message_deduplication

I want to take this opportunity to thank you not only for the great support, but also for the work that you put into this plugin. This is going to save me a ton of time and work. 👍

from rabbitmq-message-deduplication.

Ouch! I just finished building a version with Erlang 20.3 and wanted you to try it 😅.

Please bear in mind that the minimum version of RabbitMQ supporting Erlang 21.0 is 3.7.7. Below that you will experience issues.

Erlang 21.0 brought noticeable performance improvements but it also came with some backward incompatibilities.

from rabbitmq-message-deduplication.

I will of course try the version, this is just a temporary solution (running Debian testing isn't the best of ideas in production, and the Docker builds are very slow). Just point me to the build and I'll report 😃

from rabbitmq-message-deduplication.

A sidenote: I just tried applying the deduplication to a priority queue and RabbitMQ crashed. I am using 3.7.7 with Erlang 21.0 now. That should probably be an issue of its own, but I'll have to do some more tests first.

from rabbitmq-message-deduplication.

Here's the plugin built with Erlang 20.3.

plugins.zip

from rabbitmq-message-deduplication.

Just built a rabbitmq:3-management container with those files. RabbitMQ starts up without errors and the queue is indeed duplication free. Well done! 👍

from rabbitmq-message-deduplication.