Comments (14)
New release 0.3.7
is ready, please let me know if it fixes your issue.
from rabbitmq-message-deduplication.
Hello,
RabbitMQ provides a bit cumbersome way to extend queues. The plugin needs to override the original implementation and delegate any call to it which it is not supposed to handle.
This unfortunately means that any bug within the plugin's queue logic will affect the queues even when deduplication is not enabled.
Could you please provide a complete trace of the error you get? The provided information is too little to make a guess right now.
from rabbitmq-message-deduplication.
Hi.
That's what I was suspecting (hoping wasn't the case). Are there any other suggested methods of de-duping messages, perhaps without the plugin?
Is it not possible to signal something in these cases? All that happens in my consumer is the channel goes down (my consumer logic reconnects). Is this crash something I can determine from a consumer?
Here is a full error/trace:
> 2019-01-09 11:08:43.334 [error] <0.4202.0> Restarting crashed queue 'results' in vhost '/'.
> 2019-01-09 11:08:43.334 [error] <0.3984.0> ** Generic server <0.3984.0> terminating
> ** Last message in was {notify_down,<0.4052.0>}
> ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.3984.0>,[],[],[],undefined,undefined,[],[],crashed,2,[],<<"/">>,#{user => <<"guest">>}},none,true,rabbit_priority_queue,{passthrough,'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.3984.0>,[],[],[],undefined,undefined,[],[],crashed,2,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.366.0>,<<232,156,174,124,178,250,40,184,238,42,120,208,20,210,106,144>>,#{},{state,#Ref<0.4282032052.1860567042.187513>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.371.0>,#Ref<0.4282032052.1860567042.187518>,#Ref<0.4282032052.1860567041.186610>,#Ref<0.4282032052.1860567042.187519>,#Ref<0.4282032052.1860567042.187520>,{4000,800}},{client_msstate,<0.363.0>,<<160,206,87,153,252,30,7,241,6,230,21,177,18,52,127,33>>,#{},{state,#Ref<0.4282032052.1860567041.186575>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",<0.364.0>,#Ref<0.4282032052.1860567041.186576>,#Ref<0.4282032052.1860567041.186573>,#Ref<0.4282032052.1860567041.186577>,#Ref<0.4282032052.1860567041.186578>,...}},...}}},...}
> ** Reason for termination ==
> ** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',requeue,[[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.3984.0>,[],[],[],undefined,undefined,[],[],crashed,2,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.366.0>,<<232,156,174,124,178,250,40,184,238,42,120,208,20,210,106,144>>,#{},{state,#Ref<0.4282032052.1860567042.187513>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.371.0>,#Ref<0.4282032052.1860567042.187518>,#Ref<0.4282032052.1860567041.186610>,#Ref<0.4282032052.1860567042.187519>,#Ref<0.4282032052.1860567042.187520>,{4000,800}},{client_msstate,<0.363.0>,<<160,206,87,153,252,30,7,241,6,230,21,177,18,52,127,33>>,#{},{state,#Ref<0.4282032052.1860567041.186575>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",<0.364.0>,#Ref<0.4282032052.1860567041.186576>,#Ref<0.4282032052.1860567041.186573>,#Ref<0.4282032052.1860567041.186577>,#Ref<0.4282032052.1860567041.186578>,{4000,800}}},true,0,4096,0,0,0,0,0,0,infinity,0,0,0,0,0,0,{rates,0.08608385627104598,0.08608385627104598,0.08608385627104598,0.08608385627104598,...},...}}],...},...]}
> 2019-01-09 11:08:43.335 [error] <0.3984.0> CRASH REPORT Process <0.3984.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':requeue([], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.3984.0>,[],[],[],...},...}) line 279 in gen_server2:terminate/3 line 1166
> 2019-01-09 11:08:43.335 [error] <0.851.0> Supervisor {<0.851.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],none,[],[],[],undefined,undefined,...}, declare, <0.850.0>) at <0.3984.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':requeue([], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.3984.0>,[],[],[],...},...}) line 279 in context child_terminated
> 2019-01-09 11:08:43.337 [error] <0.3985.0> ** Generic server <0.3985.0> terminating
> ** Last message in was {notify_down,<0.4059.0>}
> ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,[],[],[],undefined,undefined,[],[],crashed,0,[],<<"/">>,#{user => <<"guest">>}},none,true,rabbit_priority_queue,{passthrough,'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,[],[],[],undefined,undefined,[],[],crashed,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},0,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/74KYYG532VV24470DYFGC7T30",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"bulk_uptime_checks">>}},{{client_msstate,<0.366.0>,<<73,146,104,10,126,110,113,0,194,157,242,103,191,238,88,218>>,#{},{state,#Ref<0.4282032052.1860567042.187513>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.371.0>,#Ref<0.4282032052.1860567042.187518>,#Ref<0.4282032052.1860567041.186610>,#Ref<0.4282032052.1860567042.187519>,#Ref<0.4282032052.1860567042.187520>,{4000,800}},{client_msstate,<0.363.0>,<<29,63,153,207,161,204,26,168,220,191,176,21,99,81,232,207>>,#{},{state,#Ref<0.4282032052.1860567041.186575>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",<0.364.0>,#Ref<0.4282032052.1860567041.186576>,#Ref<0.4282032052.1860567041.186573>,#Ref<0.4282032052.1860567041.186577>,...}},...}}},...}
> ** Reason for termination ==
> ** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',requeue,[[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,[],[],[],undefined,undefined,[],[],crashed,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},0,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/74KYYG532VV24470DYFGC7T30",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"bulk_uptime_checks">>}},{{client_msstate,<0.366.0>,<<73,146,104,10,126,110,113,0,194,157,242,103,191,238,88,218>>,#{},{state,#Ref<0.4282032052.1860567042.187513>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.371.0>,#Ref<0.4282032052.1860567042.187518>,#Ref<0.4282032052.1860567041.186610>,#Ref<0.4282032052.1860567042.187519>,#Ref<0.4282032052.1860567042.187520>,{4000,800}},{client_msstate,<0.363.0>,<<29,63,153,207,161,204,26,168,220,191,176,21,99,81,232,207>>,#{},{state,#Ref<0.4282032052.1860567041.186575>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",<0.364.0>,#Ref<0.4282032052.1860567041.186576>,#Ref<0.4282032052.1860567041.186573>,#Ref<0.4282032052.1860567041.186577>,#Ref<0.4282032052.1860567041.186578>,{4000,800}}},true,0,4096,0,0,0,0,0,0,infinity,0,0,0,0,0,0,{rates,0.0,0.0,0.0,0.0,-576457896158870550},{0,nil},...}}],...},...]}
> 2019-01-09 11:08:43.338 [error] <0.4203.0> Restarting crashed queue 'bulk_uptime_checks' in vhost '/'.
> 2019-01-09 11:08:43.338 [error] <0.3985.0> CRASH REPORT Process <0.3985.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':requeue([], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,...},...}) line 279 in gen_server2:terminate/3 line 1166
> 2019-01-09 11:08:43.338 [error] <0.846.0> Supervisor {<0.846.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],none,[],[],[],undefined,...}, declare, <0.845.0>) at <0.3985.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':requeue([], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,...},...}) line 279 in context child_terminated
from rabbitmq-message-deduplication.
The loglines:
** Reason for termination ==
** ** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',requeue,[[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"bulk_uptime_checks">>},true,false,none,[],<0.3985.0>,[],[],[],undefined,undefined,[],[],crashed,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},0,{0,nil},{0,nil},{0,nil},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/74KYYG532VV24470DYFGC7T30",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"bulk_uptime_checks">>}},{{client_msstate,<0.366.0>,<<73,146,104,10,126,110,113,0,194,157,242,103,191,238,88,218>>,#{},{state,#Ref<0.4282032052.1860567042.187513>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.371.0>,#Ref<0.4282032052.1860567042.187518>,#Ref<0.4282032052.1860567041.186610>,#Ref<0.4282032052.1860567042.187519>,#Ref<0.4282032052.1860567042.187520>,{4000,800}},{client_msstate,<0.363.0>,<<29,63,153,207,161,204,26,168,220,191,176,21,99,81,232,207>>,#{},{state,#Ref<0.4282032052.1860567041.186575>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",<0.364.0>,#Ref<0.4282032052.1860567041.186576>,#Ref<0.4282032052.1860567041.186573>,#Ref<0.4282032052.1860567041.186577>,#Ref<0.4282032052.1860567041.186578>,{4000,800}}},true,0,4096,0,0,0,0,0,0,infinity,0,0,0,0,0,0,{rates,0.0,0.0,0.0,0.0,-576457896158870550},{0,nil},...}}],...},...]}
Suggests me this is a duplicate of issue #22 which was fixed in commit 046a36a and released in version 0.3.6
. Could you please give it a try?
from rabbitmq-message-deduplication.
I've just upgraded to 0.3.6
(confirmed through rabbitmq-plugins list
, restarted server) but it still appears to crash the queue. This is happening when it should be routing to a dead letter through a policy or defined in arguments when declaring queue:
> 2019-01-09 19:17:09.400 [error] <0.900.0> Restarting crashed queue 'results' in vhost '/'.
> 2019-01-09 19:17:09.401 [error] <0.419.0> ** Generic server <0.419.0> terminating
> ** Last message in was {'$gen_cast',{reject,false,[0],<0.863.0>}}
> ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.419.0>,[],[],[],[{vhost,<<"/">>},{name,<<"pol">>},{pattern,<<"results">>},{'apply-to',<<"queues">>},{definition,[{<<"alternate-exchange">>,<<"dead">>},{<<"dead-letter-exchange">>,<<"dead">>},{<<"dead-letter-routing-key">>,<<"dead">>}]},{priority,10}],undefined,[],[],live,3,[],<<"/">>,#{user => <<"guest">>}},none,true,rabbit_priority_queue,{passthrough,'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.419.0>,[],[],[],undefined,undefined,[],undefined,live,2,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{1,{0,{msg_status,0,<<194,109,141,110,41,204,17,93,130,28,141,247,79,232,27,6>>,{basic_message,{resource,<<"/">>,exchange,<<>>},[<<"results">>],{content,60,{'P_basic',undefined,undefined,[],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"red">>]},<<194,109,141,110,41,204,17,93,130,28,141,247,79,232,27,6>>,false},false,true,false,false,queue_index,{message_properties,undefined,false,3}},nil,nil}},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.371.0>,<<208,22,179,222,16,110,126,58,79,51,86,112,176,48,135,237>>,#{},{state,#Ref<0.3491105379.333316097.155864>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",...},...},...}}},...}
> ** Reason for termination ==
> ** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',ackfold,[#Fun<rabbit_amqqueue_process.42.129944743>,[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.419.0>,[],[],[],undefined,undefined,[],undefined,live,2,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{1,{0,{msg_status,0,<<194,109,141,110,41,204,17,93,130,28,141,247,79,232,27,6>>,{basic_message,{resource,<<"/">>,exchange,<<>>},[<<"results">>],{content,60,{'P_basic',undefined,undefined,[],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"red">>]},<<194,109,141,110,41,204,17,93,130,28,141,247,79,232,27,6>>,false},false,true,false,false,queue_index,{message_properties,undefined,false,3}},nil,nil}},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.371.0>,<<208,22,179,222,16,110,126,58,79,51,86,112,176,48,135,237>>,#{},{state,#Ref<0.3491105379.333316097.155864>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.374.0>,#Ref<0.3491105379.333316097.155865>,#Ref<0.3491105379.333316097.155854>,#Ref<0.3491105379.333316097.155866>,#Ref<0.3491105379.333316097.155867>,{4000,800}},{client_msstate,<0.368.0>,<<"¨¾Ó~÷ãû4\bà*©¸ÐD">>,#{},{state,#Ref<0.3491105379.333316097.155819>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9..."},...}},...}},...],...},...]}
> 2019-01-09 19:17:09.401 [error] <0.419.0> CRASH REPORT Process <0.419.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':ackfold(#Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.419.0>,[],[],[],...},...}, [0]) line 299 in gen_server2:terminate/3 line 1166
> 2019-01-09 19:17:09.402 [error] <0.418.0> Supervisor {<0.418.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.5264.0>,[],[],[],undefined,...}, recovery, <0.417.0>) at <0.419.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':ackfold(#Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[],<0.419.0>,[],[],[],...},...}, [0]) line 299 in context child_terminated
With routing declared in arguments for queue directly:
> 2019-01-09 19:22:35.295 [error] <0.1600.0> Restarting crashed queue 'results' in vhost '/'.
> 2019-01-09 19:22:35.296 [error] <0.1558.0> ** Generic server <0.1558.0> terminating
> ** Last message in was {'$gen_cast',{reject,false,[0],<0.1572.0>}}
> ** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"dead">>}],<0.1558.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},none,true,rabbit_priority_queue,{passthrough,'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"dead">>}],<0.1558.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{1,{0,{msg_status,0,<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,{basic_message,{resource,<<"/">>,exchange,<<>>},[<<"results">>],{content,60,{'P_basic',undefined,undefined,[],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"red">>]},<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,false},false,true,false,false,queue_index,{message_properties,undefined,false,3}},nil,nil}},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.371.0>,<<191,22,70,16,151,172,19,132,174,192,73,230,200,123,233,245>>,#{},{state,#Ref<0.3491105379.333316097.155864>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.374.0>,#Ref<0.3491105379.333316097.155865>,#Ref<0.3491105379.333316097.155854>,#Ref<0.3491105379.333316097.155866>,#Ref<0.3491105379.333316097.155867>,...},...},...}}},...}
> ** Reason for termination ==
> ** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',ackfold,[#Fun<rabbit_amqqueue_process.42.129944743>,[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"dead">>}],<0.1558.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{1,{0,{msg_status,0,<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,{basic_message,{resource,<<"/">>,exchange,<<>>},[<<"results">>],{content,60,{'P_basic',undefined,undefined,[],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"red">>]},<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,false},false,true,false,false,queue_index,{message_properties,undefined,false,3}},nil,nil}},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.371.0>,<<191,22,70,16,151,172,19,132,174,192,73,230,200,123,233,245>>,#{},{state,#Ref<0.3491105379.333316097.155864>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",<0.374.0>,#Ref<0.3491105379.333316097.155865>,#Ref<0.3491105379.333316097.155854>,#Ref<0.3491105379.333316097.155866>,#Ref<0.3491105379.333316097.155867>,{4000,800}},{client_msstate,<0.368.0>,<<212,29,121,30,237,113,35,124,139,202,131,111,67,177,106,154>>,#{},{state,#Ref<0.3491105379.333316097.155819>,...},...}},...}},...],...},...]}
> 2019-01-09 19:22:35.296 [error] <0.1558.0> CRASH REPORT Process <0.1558.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':ackfold(#Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exc...">>,...}],...},...}, [0]) line 299 in gen_server2:terminate/3 line 1166
> 2019-01-09 19:22:35.297 [error] <0.1557.0> Supervisor {<0.1557.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exchange">>,...}],...}, declare, <0.1556.0>) at <0.1558.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':ackfold(#Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exc...">>,...}],...},...}, [0]) line 299 in context child_terminated
from rabbitmq-message-deduplication.
What version of Erlang and RabbitMQ are you using?
from rabbitmq-message-deduplication.
I'm on RMQ 3.7.8
and Erlang OTP 20 (erts-9.2)
.
from rabbitmq-message-deduplication.
Note that this time the reason for the crash is way different:
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',ackfold,[#Fun<rabbit_amqqueue_process.42.129944743>,[],{dqstate,{amqqueue,{resource,<<"/">>,queue,<<"results">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"dead">>}],<0.1558.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"guest">>}},{vqstate,{0,{[],[]}},{0,{[],[]}},{delta,undefined,0,0,undefined},{0,{[],[]}},{0,{[],[]}},1,{0,nil},{0,nil},{1,{0,{msg_status,0,<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,{basic_message,{resource,<<"/">>,exchange,<<>>},[<<"results">>],{content,60,{'P_basic',undefined,undefined,[],1,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},none,none,[<<"red">>]},<<9,249,79,179,215,72,252,252,18,94,150,209,58,81,226,54>>,false},false,true,false,false,queue_index,{message_properties,undefined,false,3}},nil,nil}},{qistate,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/3MOP8SFTC1FGOYT1OQBIU1T50",{#{},[]},undefined,0,32768,#Fun<rabbit_variable_queue.10.44586449>,#Fun<rabbit_variable_queue.11.44586449>,{0,nil},{0,nil},[],[],{resource,<<"/">>,queue,<<"results">>}},{{client_msstate,<0.371.0>,<<191,22,70,16,151,172,19,132,174,192,73,230,200,123,233,245>>,#{},{state,#Ref<0.3491105379.333316097.155864>,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent"},rabbit_msg_store_ets_index,"/var/lib/rabbitmq/mnesia/rabbit@deathstar/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent",0.374.0>,#Ref<0.3491105379.333316097.155865>,#Ref<0.3491105379.333316097.155854>,#Ref<0.3491105379.333316097.155866>,#Ref<0.3491105379.333316097.155867>,{4000,800}},{client_msstate,<0.368.0>,<<212,29,121,30,237,113,35,124,139,202,131,111,67,177,106,154>>,#{},{state,#Ref<0.3491105379.333316097.155819>,...},...}},...}},...],...},...]}
I had a quick look at the code and did not notice anything evident. Therefore, I'll need to reproduce this issue to dig further.
Could you please provide a sequence of commands to reproduce the issue?
- Resources creations via
rabbitmqadmin
/rabbitmqctl
- Problematic message scheduling/consuming
That would really simplify the investigations.
from rabbitmq-message-deduplication.
Sure, here is how I'm reproducing:
$ rabbitmqadmin declare exchange name=dead-letter type=fanout
=> exchange declared
$ rabbitmqadmin declare exchange name=my-exchange type=direct
=> exchange declared
$ rabbitmqadmin declare queue name=my-queue durable=true
=> queue declared
$ rabbitmqadmin declare binding source="my-exchange" destination_type="queue" destination="my-queue" routing_key="my-route"
=> binding declared
$ rabbitmqctl set_policy DLX "my-queue" '{"dead-letter-exchange":"dead-letter"}' --priority 1 --apply-to queues
=> Setting policy "DLX" for pattern "my-queue" to "{"dead-letter-exchange":"dead-letter"}" with priority "1" for vhost "/" ...
$ rabbitmqadmin publish exchange=my-exchange routing_key="my-route" payload="hello"
=> Message published
Error (different from above -- sadly cannot seem to reproduce that error):
2019-01-09 22:28:31.520 [error] <0.7211.0> CRASH REPORT Process <0.7211.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':fetchwhile(#Fun<rabbit_amqqueue_process.36.129944743>, #Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"my-queue">>},true,false,none,[],<0.7211.0>,[],[],[],...},...}) line 234 in gen_server2:terminate/3 line 1166
2019-01-09 22:28:31.521 [error] <0.6764.0> Supervisor {<0.6764.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"my-queue">>},true,false,none,[],none,[],[],[],undefined,undefined,...}, declare, <0.6763.0>) at <0.7211.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':fetchwhile(#Fun<rabbit_amqqueue_process.36.129944743>, #Fun<rabbit_amqqueue_process.42.129944743>, [], {dqstate,{amqqueue,{resource,<<"/">>,queue,<<"my-queue">>},true,false,none,[],<0.7211.0>,[],[],[],...},...}) line 234 in context child_terminated
As an aside, if I apply the policy to the exchange, I receive no errors when nacking:
$ rabbitmqctl set_policy DLX "my-exchange" '{"dead-letter-exchange":"dead-letter"}' --priority 1 --apply-to exchanges
=> Setting policy "DLX" for pattern "my-exchange" to "{"dead-letter-exchange":"dead-letter"}" with priority "1" for vhost "/" ...
$ rabbitmqadmin publish exchange=my-exchange routing_key="my-route" payload="hello"
=> Message published
$ rabbitmqadmin get queue=my-queue ackmode=reject_requeue_false
+-------------+-------------+---------------+---------+---------------+------------------+------------+-------------+
| routing_key | exchange | message_count | payload | payload_bytes | payload_encoding | properties | redelivered |
+-------------+-------------+---------------+---------+---------------+------------------+------------+-------------+
| my-route | my-exchange | 0 | hello | 5 | string | | False |
..though no errors are produced when nacking, messages do not end up in my dead letter exchange (though unsure if that is misconfiguration on my part based on commands above).
from rabbitmq-message-deduplication.
I finally have some messages routing to dead letter by disabled rabbitmq_message_deduplication
and declaring x-dead-letter-exchange
in args when declaring queue:
$ rabbitmqadmin declare queue name=queue-with-args durable=true \
'arguments={"x-dead-letter-exchange":"dead-letter"}'
=> queue declared
When nacking messages on this queue they are, as expected, sent to the dead letter exchange.
However, when I enable rabbitmq_message_deduplication
I'm back to the error in my previous reply.
from rabbitmq-message-deduplication.
Found the issue which affects several functions.
def ackfold(function, A, state = dqstate(queue_state: qs), acks) do
...
end
Note how the second parameter of the above function is a capital letter A
. It is a copy-paste from rabbitmq-common
function definitions which are in Erlang. In Erlang variables are supposed to start with capital letter. But in Elixir Capital letters have completely different meaning. Hence, the function_clause
error.
Will fix it and make a new release.
from rabbitmq-message-deduplication.
Good find! I'll give it a try when I get hands on release.
from rabbitmq-message-deduplication.
Working great now! 👍 🎉
Thanks :)
from rabbitmq-message-deduplication.
np, thanks for reporting this!
from rabbitmq-message-deduplication.
Related Issues (20)
- Queue level message deduplication failed sometimes. HOT 1
- Server crash HOT 2
- Support for quorums? HOT 1
- Deduplication doesn't consider the unack messages in the queue HOT 2
- Are there anyway for duplicated message to be dead letter upon published? [Question]
- Celery lost connection to broker HOT 1
- Queue crashes when is created with x-message-deduplication=true parameter on RabbitMQ 3.13.0, Plugin version 0.6.2 HOT 6
- 0.5.3 queues not working under rabbitmq 3.10.0 HOT 3
- Auto generate deduplication checksum base on payload or configurable properties
- 0.5.3 queues not working under rabbitmq 3.8.17 HOT 1
- Can key be removed from cache, after BasicAck and some time has ellapsed? HOT 3
- {:plugins_not_found, [:rabbitmq_message_deduplication]} HOT 2
- Policies support HOT 4
- Bug - when the rabbitmq is restarted the messages in a persistent queue are not deduplicated HOT 2
- Exchange level doesn't work but queue level does. What's differrence on using these two?
- Queue crashed with reason timeout to call RabbitMQMessageDeduplication.CacheManager HOT 7
- `x-cache-size` seems to be manadatory, not optional. HOT 1
- Queue level deduplication seems broken HOT 8
- Exchange info crash after upgrade to 0.6.0 HOT 5
- Debugging exchanges HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-message-deduplication.