Comments (24)
urkl
commented on July 19, 2024
1
@atetzner, @urkl a new release
0.3.6is out which should address the issues withrequeueandinitfunctions.When you have a spare moment, could you please try and confirm if you still suffer from crashes when creating queues?
For me its working now. Thanks @noxdafox . This was very fast release.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
A message is considered within the queue until is fetched and acknowledged. How are you getting the message from the queue from the console?
EDIT: I just noticed an issue when consuming a message with nack and requeue enabled. Is this the way you are publishing it within the queue?
from rabbitmq-message-deduplication.
commented on July 19, 2024
I just made my tests again:
When I tested just with the web ui, I could not reproduce the problem today - although I am pretty sure, it didn't work yesterday ...
I can reproduce the issue when publishing and receiving the messages from a Spring Boot application with Spring Cloud streaming.
I will try to make a simple "proof of concept" and post it here.
from rabbitmq-message-deduplication.
commented on July 19, 2024
I have built up a sample application, that demonstrates my problem.
Result
The output for me is:
Sending event: Fri Oct 26 10:07:15 CEST 2018
Processing event: Fri Oct 26 10:07:15 CEST 2018
Sending event: Fri Oct 26 10:07:30 CEST 2018
Processed event: Fri Oct 26 10:07:15 CEST 2018
Sending event: Fri Oct 26 10:07:45 CEST 2018
Sending event: Fri Oct 26 10:08:00 CEST 2018
Sending event: Fri Oct 26 10:08:15 CEST 2018
Sending event: Fri Oct 26 10:08:30 CEST 2018
...
Here is my queue/exchange configuration
Sample App Code
build.gradle
buildscript {
repositories {
maven { url "https://repo1.maven.org/maven2/" }
maven { url "https://repo.spring.io/plugins-release/" }
maven { url "https://plugins.gradle.org/m2/" }
}
dependencies {
classpath "org.springframework.boot:spring-boot-gradle-plugin:2.0.6.RELEASE"
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.2.71"
classpath "org.jetbrains.kotlin:kotlin-allopen:1.2.71"
classpath "org.jetbrains.kotlin:kotlin-noarg:1.2.71"
}
}
apply plugin: 'kotlin'
apply plugin: 'io.spring.dependency-management'
apply plugin: 'kotlin-spring'
apply plugin: 'org.springframework.boot'
repositories {
maven { url "https://repo1.maven.org/maven2/" }
maven { url "https://repo.spring.io/plugins-release/" }
maven { url "https://repo.spring.io/milestone" }
}
compileKotlin {
kotlinOptions.jvmTarget = JavaVersion.VERSION_1_8
kotlinOptions.allWarningsAsErrors = true
}
compileTestKotlin {
kotlinOptions.jvmTarget = JavaVersion.VERSION_1_8
kotlinOptions.allWarningsAsErrors = true
}
dependencyManagement {
imports {
mavenBom "org.springframework.boot:spring-boot-dependencies:2.0.6.RELEASE"
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Finchley.SR1"
}
}
dependencies {
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.2.71"
implementation "org.jetbrains.kotlin:kotlin-reflect:1.2.71"
implementation "org.springframework.cloud:spring-cloud-starter-stream-rabbit"
}application.yml
base-package: "com.myapp"
spring:
application:
name: myapp
rabbitmq:
host: ${RABBITMQ_HOST:localhost}
port: ${RABBITMQ_PORT:32500}
username: ${RABBITMQ_USER:root}
password: ${RABBITMQ_PASSWORD:root}
virtual-host: ${RABBITMQ_VHOST:/}
cloud:
stream:
bindings:
input:
destination: myqueue
binder: rabbitmq
group: group.myapp
consumer:
max-attempts: 10
output:
destination: myexchange
binder: rabbitmq
partitionKeyExpression: payload.name
partitionCount: ${SERVICE_REPLICAS:1}
binders:
rabbitmq:
type: rabbit
environment:
spring:
rabbitmq:
host: ${RABBITMQ_HOST:localhost}
port: ${RABBITMQ_PORT:32500}
username: ${RABBITMQ_USER:root}
password: ${RABBITMQ_PASSWORD:root}
virtual-host: ${RABBITMQ_VHOST:/}RabbitQueueDeduplication.kt
package com.myapp.streaming
import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.cloud.stream.annotation.EnableBinding
import org.springframework.cloud.stream.annotation.StreamListener
import org.springframework.cloud.stream.binding.BinderAwareChannelResolver
import org.springframework.cloud.stream.messaging.Sink
import org.springframework.messaging.MessageHeaders
import org.springframework.messaging.support.MessageBuilder
import org.springframework.scheduling.annotation.EnableScheduling
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Component
import java.util.Date
@SpringBootApplication(scanBasePackages = ["com.myapp"])
@EnableBinding(
Sink::class
)
@EnableScheduling
class RabbitQueueDeduplication
fun main(args: Array<String>) {
SpringApplication.run(RabbitQueueDeduplication::class.java, *args)
}
@Component
class Sender(
private val resolver: BinderAwareChannelResolver
) {
@Scheduled(cron = "*/15 * * * * *")
fun sendEvent() {
val payload = Date().toString()
System.out.println("Sending event: $payload")
val message = MessageBuilder
.withPayload(payload)
.setHeader(MessageHeaders.CONTENT_TYPE, "text/plain")
.setHeader("x-deduplication-header", "dedup-me")
.build()
resolver
.resolveDestination("output")
.send(message)
}
}
@Component
class Receiver {
@StreamListener(
Sink.INPUT
)
fun onEvent(payload: String) {
System.out.println("Processing event: $payload")
Thread.sleep(20000)
System.out.println("Processed event: $payload")
}
}
from rabbitmq-message-deduplication.
commented on July 19, 2024
I was unsure, if you are familiar with Spring Boot and kotlin, so here is my sample application as ZIP. Simply uncompress the archive and execute ./gradlew bootRun
If you want to customize the RabbitMQ connection information, simply set them as environment variables, e.g.: RABBITMQ_HOST=my-rabbitmq-host ./gradlew bootRun
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
Thanks for providing a reproducible demo. I tried to run as suggested but it seems it cannot resolve the domain name of a dependency.
noxdafox@mobnox:~/rabbitmq-deduplication$ ./gradlew bootRun
Downloading https://nexus.detss.corpintra.net/repository/gradle-distributions/gradle-4.10.2-all.zip
Exception in thread "main" java.net.UnknownHostException: nexus.detss.corpintra.net
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at org.gradle.wrapper.Download.downloadInternal(Download.java:66)
at org.gradle.wrapper.Download.download(Download.java:51)
at org.gradle.wrapper.Install$1.call(Install.java:62)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
If I try to resolve the domain nexus.detss.corpintra.net.
noxdafox@mobnox:~/rabbitmq-deduplication$ nslookup nexus.detss.corpintra.net
Server: 192.168.32.1
Address: 192.168.32.1#53
** server can't find nexus.detss.corpintra.net: NXDOMAIN
EDIT: replaced with distributionUrl in gradle-wrapper.properties file with https://services.gradle.org/distributions/gradle-4.10.2-all.zip to get it working.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
I tried the provided demo code and could not get it to work.
It does not create correct bindings for the queue, therefore the messages never get correctly routed. Moreover, the queue does not have deduplication enabled.
I am not familiar with Kotlin but I don't see neither in the code nor in the configuration file any hint of setting correct bindings or enabling deduplication on the queue.
from rabbitmq-message-deduplication.
commented on July 19, 2024
Sorry for the wrong URL ...
It is right, the Kotlin code does not create any bindings. I have some other deployment scripts for RabbitMQ, that create all the exchanges, queues and bindings for my application. The Spring application will use exchanges/queues/bindings, if they are already there - and otherwise create them as good as possible. But as you already found out, the application is missing some information to create a completely functional RabbitMQ configuration.
Please see my first post for my exchange/queue/binding configuration.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
I think I managed to reproduce the issue you are suffering from. What confuses me is the fact the tests should have covered this case but they seem to succeed nonetheless.
I produced an experimental build which I am attaching. Could you please verify whether it solves the problem?
rabbitmq_message_deduplication-0.3.5_issue-22.ez.gz
NOTE: I had to gzip the plugin as GitHub enforces a limited amount of file types when uploading to issues.
from rabbitmq-message-deduplication.
commented on July 19, 2024
Sorry for the late response - I've been on vacation for a long weekend ...
I tested the plugin, but unfortunately it seems to have a bug: When I try to create my queue, the plugin seems to crash - even if I declare the queue without x-message-deduplication.
Here is the log I get:
Declaring queue myqueue
2018-11-05 09:17:27.915 [error] <0.839.0> ** Generic server <0.839.0> terminating
** Last message in was {init,new}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"DLX">>},{<<"x-dead-letter-routing-key">>,longstr,<<"myqueue">>}],<0.839.0>,[],[],[],[{vhost,<<"/">>},{name,<<"ha-general">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,2},{<<"ha-promote-on-failure">>,<<"when-synced">>},{<<"ha-sync-batch-size">>,200},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460710858795,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"DLX">>},{<<"x-dead-letter-routing-key">>,longstr,<<"myqueue">>}],<0.839.0>,[],[],[],[{vhost,<<"/">>},{name,<<"ha-general">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,2},{<<"ha-promote-on-failure">>,<<"when-synced">>},{<<"ha-sync-batch-size">>,200},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_mirror_queue_master,init,3,[{file,"src/rabbit_mirror_queue_master.erl"},{line,98}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-05 09:17:27.915 [warning] <0.836.0> FORMAT ERROR: "Declare queue error: ~s" [{'EXIT',{{function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"DLX">>},{<<"x-dead-letter-routing-key">>,longstr,<<"myqueue">>}],<0.839.0>,[],[],[],[{vhost,<<"/">>},{name,<<"ha-general">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,2},{<<"ha-promote-on-failure">>,<<"when-synced">>},{<<"ha-sync-batch-size">>,200},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_mirror_queue_master,init,3,[{file,"src/rabbit_mirror_queue_master.erl"},{line,98}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server2,call,[<0.839.0>,{init,new},infinity]}}}]
2018-11-05 09:17:27.915 [error] <0.839.0> CRASH REPORT Process <0.839.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-05 09:17:27.915 [error] <0.841.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-05 09:17:27.917 [error] <0.838.0> Supervisor {<0.838.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, declare, <0.837.0>) at <0.839.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
*** Request to node [email protected] failed with {'EXIT',
{{function_clause,
[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',
init,
[{amqqueue,
{resource,
<<"/">>,
queue,
<<"myqueue">>},
true,
false,
none,
[{<<"x-dead-letter-exchange">>,
longstr,
<<"DLX">>},
{<<"x-dead-letter-routing-key">>,
longstr,
<<"myqueue">>}],
<0.839.0>,
[],
[],
[],
[{vhost,
<<"/">>},
{name,
<<"ha-general">>},
{pattern,
<<".*">>},
{'apply-to',
<<"all">>},
{definition,
[{<<"ha-mode">>,
<<"exactly">>},
{<<"ha-params">>,
2},
{<<"ha-promote-on-failure">>,
<<"when-synced">>},
{<<"ha-sync-batch-size">>,
200},
{<<"ha-sync-mode">>,
<<"automatic">>}]},
{priority,
0}],
undefined,
[],
[],
live,
0,
[],
<<"/">>,
#{user =>
<<"root">>}},
new,
#Fun<rabbit_amqqueue_process.11.129944743>],
[{file,
"lib/rabbit_message_deduplication_queue.ex"},
{line,
125}]},
{rabbit_priority_queue,
init,
3,
[{file,
"src/rabbit_priority_queue.erl"},
{line,
151}]},
{rabbit_mirror_queue_master,
init,
3,
[{file,
"src/rabbit_mirror_queue_master.erl"},
{line,
98}]},
{rabbit_amqqueue_process,
init_it2,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
207}]},
{rabbit_amqqueue_process,
handle_call,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
2018-11-05 09:17:27.924 [error] <0.841.0> ** Generic server <0.841.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"DLX">>},{<<"x-dead-letter-routing-key">>,longstr,<<"myqueue">>}],<0.841.0>,[],[],[],[{vhost,<<"/">>},{name,<<"ha-general">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,2},{<<"ha-promote-on-failure">>,<<"when-synced">>},{<<"ha-sync-batch-size">>,200},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460710853344,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
1153}]},
{gen_server2,
handle_msg,
2,
[{file,
"src/gen_server2.erl"},
{line,
1029}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
247}]}]},
{gen_server2,
call,
[<0.839.0>,
{init,
new},
infinity]}}}
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-dead-letter-exchange">>,longstr,<<"DLX">>},{<<"x-dead-letter-routing-key">>,longstr,<<"myqueue">>}],<0.841.0>,[],[],[],[{vhost,<<"/">>},{name,<<"ha-general">>},{pattern,<<".*">>},{'apply-to',<<"all">>},{definition,[{<<"ha-mode">>,<<"exactly">>},{<<"ha-params">>,2},{<<"ha-promote-on-failure">>,<<"when-synced">>},{<<"ha-sync-batch-size">>,200},{<<"ha-sync-mode">>,<<"automatic">>}]},{priority,0}],undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},non_clean_shutdown,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_mirror_queue_master,init,3,[{file,"src/rabbit_mirror_queue_master.erl"},{line,98}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1321}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-05 09:17:27.924 [error] <0.841.0> CRASH REPORT Process <0.841.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-05 09:17:27.925 [error] <0.838.0> Supervisor {<0.838.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, declare, <0.837.0>) at <0.841.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[...],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
2018-11-05 09:17:27.924 [error] <0.843.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-05 09:17:27.932 [error] <0.845.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-05 09:17:27.933 [error] <0.843.0> ** Generic server <0.843.0> terminating
With the latest release of your plugin, my container comes up and the deployment scripts can create all exchanges / queues / bindings.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
Looking at the error it seems you are creating a different queue from the one you initially reported (you are binding it to a dead lettering exchange, using high availability and a lot more). The error you are getting is completely different from the initial one.
I would like to focus on one issue at a time. Can you please test the plugin with the initial premises and confirm the issue is solved?
We can then concentrate to the new issue at hand.
from rabbitmq-message-deduplication.
commented on July 19, 2024
I deleted my whole rabbitmq data directory to have a clean setup. After that, I created only the objects as mentioned in comment 1:
echo "Declaring queue myqueue"
$RABBITMQADMIN declare queue \
name=myqueue \
durable=true \
arguments="{\"x-message-deduplication\":true}"
Nevertheless, the plugin crashes, when creating the queue:
Declaring queue myqueue
2018-11-05 10:01:07.587 [error] <0.821.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-05 10:01:07.587 [warning] <0.816.0> FORMAT ERROR: "Declare queue error: ~s" [{'EXIT',{{function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.819.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server2,call,[<0.819.0>,{init,new},infinity]}}}]
2018-11-05 10:01:07.588 [error] <0.819.0> ** Generic server <0.819.0> terminating
** Last message in was {init,new}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.819.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460714227981,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.819.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-05 10:01:07.588 [error] <0.819.0> CRASH REPORT Process <0.819.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-05 10:01:07.589 [error] <0.818.0> Supervisor {<0.818.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.817.0>) at <0.819.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
*** Request to node [email protected] failed with {'EXIT',
{{function_clause,
[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',
init,
[{amqqueue,
{resource,
<<"/">>,
queue,
<<"myqueue">>},
true,
false,
none,
[{<<"x-message-deduplication">>,
bool,
true}],
<0.819.0>,
[],
[],
[],
undefined,
undefined,
[],
[],
live,
0,
[],
<<"/">>,
#{user =>
<<"root">>}},
new,
#Fun<rabbit_amqqueue_process.11.129944743>],
[{file,
"lib/rabbit_message_deduplication_queue.ex"},
{line,
125}]},
{rabbit_priority_queue,
init,
3,
[{file,
"src/rabbit_priority_queue.erl"},
{line,
151}]},
{rabbit_amqqueue_process,
init_it2,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
207}]},
{rabbit_amqqueue_process,
handle_call,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
1153}]},
{gen_server2,
handle_msg,
2,
[{file,
"src/gen_server2.erl"},
{line,
1029}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
247}]}]},
{gen_server2,
call,
[<0.819.0>,
{init,
new},
infinity]}}}
2018-11-05 10:01:07.600 [error] <0.823.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-05 10:01:07.601 [error] <0.821.0> ** Generic server <0.821.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.821.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460714223374,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.821.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},non_clean_shutdown,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1321}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-05 10:01:07.601 [error] <0.821.0> CRASH REPORT Process <0.821.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-05 10:01:07.602 [error] <0.818.0> Supervisor {<0.818.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.817.0>) at <0.821.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
2018-11-05 10:01:07.614 [error] <0.823.0> ** Generic server <0.823.0> terminating
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
The error is really weird. You said you are using Erlang 20.x right? I might have compiled the plugin with the wrong Erlang version. Once in front of my workstation I will double-check. Thanks for your patience.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
I rebuilt both the dependencies with Erlang 20.3. Could you please try these versions?
elixir-1.7.3.ez.gz
rabbitmq_message_deduplication-0.3.5.ez.gz
from rabbitmq-message-deduplication.
commented on July 19, 2024
I am sorry, but the error during startup seems to stay the same:
Declaring queue myqueue
2018-11-06 06:59:32.098 [error] <0.843.0> ** Generic server <0.843.0> terminating
** Last message in was {init,new}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.843.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460686078507,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.843.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-06 06:59:32.099 [error] <0.843.0> CRASH REPORT Process <0.843.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-06 06:59:32.099 [warning] <0.840.0> FORMAT ERROR: "Declare queue error: ~s" [{'EXIT',{{function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.843.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},new,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_call,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,1153}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1029}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server2,call,[<0.843.0>,{init,new},infinity]}}}]
2018-11-06 06:59:32.099 [error] <0.845.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-06 06:59:32.100 [error] <0.842.0> Supervisor {<0.842.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.841.0>) at <0.843.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
*** Request to node [email protected] failed with {'EXIT',
{{function_clause,
[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',
init,
[{amqqueue,
{resource,
<<"/">>,
queue,
<<"myqueue">>},
true,
false,
none,
[{<<"x-message-deduplication">>,
bool,
true}],
<0.843.0>,
[],
[],
[],
undefined,
undefined,
[],
[],
live,
0,
[],
<<"/">>,
#{user =>
<<"root">>}},
new,
#Fun<rabbit_amqqueue_process.11.129944743>],
[{file,
"lib/rabbit_message_deduplication_queue.ex"},
{line,
125}]},
{rabbit_priority_queue,
init,
3,
[{file,
"src/rabbit_priority_queue.erl"},
{line,
151}]},
{rabbit_amqqueue_process,
init_it2,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
207}]},
{rabbit_amqqueue_process,
handle_call,
3,
[{file,
"src/rabbit_amqqueue_process.erl"},
{line,
1153}]},
{gen_server2,
handle_msg,
2,
[{file,
"src/gen_server2.erl"},
{line,
1029}]},
{proc_lib,
init_p_do_apply,
3,
[{file,
"proc_lib.erl"},
{line,
2018-11-06 06:59:32.107 [error] <0.845.0> ** Generic server <0.845.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.845.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460686072030,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
** {function_clause,[{'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue',init,[{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.845.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},non_clean_shutdown,#Fun<rabbit_amqqueue_process.11.129944743>],[{file,"lib/rabbit_message_deduplication_queue.ex"},{line,125}]},{rabbit_priority_queue,init,3,[{file,"src/rabbit_priority_queue.erl"},{line,151}]},{rabbit_amqqueue_process,init_it2,3,[{file,"src/rabbit_amqqueue_process.erl"},{line,207}]},{rabbit_amqqueue_process,handle_cast,2,[{file,"src/rabbit_amqqueue_process.erl"},{line,1321}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1050}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
2018-11-06 06:59:32.107 [error] <0.847.0> Restarting crashed queue 'myqueue' in vhost '/'.
247}]}]},
{gen_server2,
call,
[<0.843.0>,
{init,
new},
infinity]}}}
2018-11-06 06:59:32.108 [error] <0.845.0> CRASH REPORT Process <0.845.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in gen_server2:terminate/3 line 1166
2018-11-06 06:59:32.108 [error] <0.842.0> Supervisor {<0.842.0>,rabbit_amqqueue_sup} had child rabbit_amqqueue started with rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, declare, <0.841.0>) at <0.845.0> exit with reason no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, non_clean_shutdown, #Fun<rabbit_amqqueue_process.11.129944743>) line 125 in context child_terminated
2018-11-06 06:59:32.119 [error] <0.849.0> Restarting crashed queue 'myqueue' in vhost '/'.
2018-11-06 06:59:32.119 [error] <0.847.0> ** Generic server <0.847.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {q,{amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,bool,true}],<0.847.0>,[],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,#{user => <<"root">>}},none,false,undefined,undefined,{state,{queue,[],[],0},{active,-576460686064407,1.0}},undefined,undefined,undefined,undefined,{state,fine,5000,undefined},{0,nil},undefined,undefined,undefined,{state,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},delegate},undefined,undefined,undefined,undefined,'drop-head',0,0,running}
** Reason for termination ==
Just to be sure: I am using RabbitMQ 3.7.8 on Erlang 20.3.4. The md5 hashes of the *.ez files you gave me are:
md5sum elixir-erl-20.ez rabbitmq_message_deduplication-erl-20.ez
a8324397a1b7a31464cbbae91071c453 elixir-erl-20.ez
c216dc0d3e17573d8d547b6dfc9dc85e rabbitmq_message_deduplication-erl-20.ez
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
Thanks for giving it a second try. I cannot reproduce the issue on my local machine. I've seen this problem in the past when starting the plugin on a "dirty" broker.
Are you sure your setup is clean? Have you tried reinstalling the broker? Did you clear the data folder?
from rabbitmq-message-deduplication.
commented on July 19, 2024
I have double-checked, that my data directory was clean before startup. Additionally I removed all other configuration steps, that are not absolutely necessary. Unfortunately the problem persists.
I have the additional plugins rabbitmq_peer_discovery_k8s and rabbitmq_management enabled. Perhaps the issue is connected to these?
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
The additional plugins should not be an issue as they are not related. I myself always use the rabbitmq_management plugin.
I tried to reproduce the issues in several ways but with no luck.
The crash report:
CRASH REPORT Process <0.819.0> with 0 neighbours exited with reason: no function clause matching 'Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue':init({amqqueue,{resource,<<"/">>,queue,<<"myqueue">>},true,false,none,[{<<"x-message-deduplication">>,...}],...}, new, #Fun<rabbit_amqqueue_process.11.129944743>) line 125
Indicates that when trying to call the Elixir.RabbitMQ.MessageDeduplicationPlugin.Queue.init() function at line 125, Erlang did not find a matching function.
Looking at the parameters being passed to the init function they seem correct: a amqqueue record, the atom new and a function callback. These are the parameters which the init function is expecting. Hence I am very confused when seeing that error.
Do you think you could provide me a copy of your environment which would reproduce the issue? A Docker container for example would be perfect.
from rabbitmq-message-deduplication.
commented on July 19, 2024
While creating simple scripts for you for testing, I think I found the cause of the error. Unfortunately, this piece of code is from my colleague and I need to ask him, why he did this ...
I will report back, when I know more
from rabbitmq-message-deduplication.
commented on July 19, 2024
After getting deeper into it, the command I thought was the problem is actually required / sensible.
To be sure that my Dockerfile is not the problem, I modified the official one and adapted only some necessary things. So here is the Dockerfile with which I can reproduce the issue:
FROM alpine:3.8
# add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
RUN addgroup -S rabbitmq -g 101 && \
adduser -S -h /var/lib/rabbitmq -G rabbitmq -u 100 rabbitmq
# grab su-exec for easy step-down from root
RUN apk add --no-cache 'su-exec>=0.2'
RUN apk add --no-cache \
# Bash for docker-entrypoint
bash \
# Procps for rabbitmqctl
procps \
# Erlang for RabbitMQ
erlang-asn1 \
erlang-hipe \
erlang-crypto \
erlang-eldap \
erlang-inets \
erlang-mnesia \
erlang \
erlang-os-mon \
erlang-public-key \
erlang-sasl \
erlang-ssl \
erlang-syntax-tools \
erlang-xmerl \
# Python for rabbitmqadmin
python
# get logs to stdout (thanks @dumbbell for pushing this upstream! :D)
ENV RABBITMQ_LOGS=- RABBITMQ_SASL_LOGS=-
# https://github.com/rabbitmq/rabbitmq-server/commit/53af45bf9a162dec849407d114041aad3d84feaf
ENV RABBITMQ_HOME /opt/rabbitmq
ENV PATH $RABBITMQ_HOME/sbin:$PATH
# gpg: key 6026DFCA: public key "RabbitMQ Release Signing Key <[email protected]>" imported
ENV RABBITMQ_GPG_KEY 0A9AF2115F4687BD29803A206B73A36E6026DFCA
ENV RABBITMQ_VERSION 3.7.8
ENV RABBITMQ_GITHUB_TAG v3.7.8
RUN set -ex; \
\
apk add --no-cache --virtual .build-deps \
ca-certificates \
gnupg \
libressl \
wget \
xz \
; \
\
wget -O rabbitmq-server.tar.xz.asc "https://github.com/rabbitmq/rabbitmq-server/releases/download/$RABBITMQ_GITHUB_TAG/rabbitmq-server-generic-unix-${RABBITMQ_VERSION}.tar.xz.asc"; \
wget -O rabbitmq-server.tar.xz "https://github.com/rabbitmq/rabbitmq-server/releases/download/$RABBITMQ_GITHUB_TAG/rabbitmq-server-generic-unix-${RABBITMQ_VERSION}.tar.xz"; \
\
export GNUPGHOME="$(mktemp -d)"; \
rm -rf "$GNUPGHOME"; \
\
mkdir -p "$RABBITMQ_HOME"; \
tar \
--extract \
--verbose \
--file rabbitmq-server.tar.xz \
--directory "$RABBITMQ_HOME" \
--strip-components 1 \
; \
rm -f rabbitmq-server.tar.xz*; \
\
# update SYS_PREFIX (first making sure it's set to what we expect it to be)
grep -qE '^SYS_PREFIX=\$\{RABBITMQ_HOME\}$' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
sed -ri 's!^(SYS_PREFIX=).*$!\1!g' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
grep -qE '^SYS_PREFIX=$' "$RABBITMQ_HOME/sbin/rabbitmq-defaults"; \
\
apk del .build-deps; \
\
wget -O "$RABBITMQ_HOME/plugins/elixir-erl-20.ez.gz" "https://github.com/noxdafox/rabbitmq-message-deduplication/files/2550344/elixir-1.7.3.ez.gz" ; \
gunzip "$RABBITMQ_HOME/plugins/elixir-erl-20.ez.gz" ; \
wget -O "$RABBITMQ_HOME/plugins/rabbitmq_message_deduplication-erl-20.ez.gz" "https://github.com/noxdafox/rabbitmq-message-deduplication/files/2550345/rabbitmq_message_deduplication-0.3.5.ez.gz" ; \
gunzip "$RABBITMQ_HOME/plugins/rabbitmq_message_deduplication-erl-20.ez.gz"
# set home so that any `--user` knows where to put the erlang cookie
ENV HOME /var/lib/rabbitmq
RUN mkdir -p /var/lib/rabbitmq /etc/rabbitmq /var/log/rabbitmq /tmp/rabbitmq-ssl \
&& chown -R rabbitmq:rabbitmq /var/lib/rabbitmq /etc/rabbitmq /var/log/rabbitmq /tmp/rabbitmq-ssl \
&& chmod -R 777 /var/lib/rabbitmq /etc/rabbitmq /var/log/rabbitmq /tmp/rabbitmq-ssl
VOLUME /var/lib/rabbitmq
# add a symlink to the .erlang.cookie in /root so we can "docker exec rabbitmqctl ..." without gosu
RUN ln -sf /var/lib/rabbitmq/.erlang.cookie /root/
RUN ln -sf "$RABBITMQ_HOME/plugins" /plugins
COPY docker-entrypoint.sh /usr/local/bin/
# enables the necessary plugins
RUN rabbitmq-plugins enable rabbitmq_peer_discovery_k8s && \
rabbitmq-plugins enable rabbitmq_management && \
rabbitmq-plugins enable rabbitmq_message_deduplication
ENTRYPOINT ["docker-entrypoint.sh"]
EXPOSE 4369 5671 5672 25672
CMD ["rabbitmq-server"]Hope this helps to find the issue ...
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
You did not provide the docker-entrypoint.sh script. Is it this one?
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
I managed to reproduce the issue and eventually understood the core problem.
Last Monday I was lucky enough to meet few core RabbitMQ developers. Special thanks to @dumbbell for explaining me some quirks when building plugins.
In short, I was building the plugin without setting any restriction on the dependencies version. As a consequence, the plugin was built against the latest RabbitMQ code (3.8.0.beta1) which introduces few incompatibilities. In particular, the amqqueue record has been changed causing the init function to fail matching.
I will soon release a 0.3.6 version built with the correct dependencies.
from rabbitmq-message-deduplication.
noxdafox
commented on July 19, 2024
@atetzner, @urkl a new release 0.3.6 is out which should address the issues with requeue and init functions.
When you have a spare moment, could you please try and confirm if you still suffer from crashes when creating queues?
from rabbitmq-message-deduplication.
commented on July 19, 2024
@noxdafox , @urkl : I can also confirm, that 0.3.6 works and fixes the problem described in this issue.
Many thanks!
from rabbitmq-message-deduplication.
Related Issues (20)
- Queue level message deduplication failed sometimes. HOT 1
- Server crash HOT 2
- Support for quorums? HOT 1
- Deduplication doesn't consider the unack messages in the queue HOT 2
- Are there anyway for duplicated message to be dead letter upon published? [Question]
- Celery lost connection to broker HOT 1
- Queue crashes when is created with x-message-deduplication=true parameter on RabbitMQ 3.13.0, Plugin version 0.6.2 HOT 6
- 0.5.3 queues not working under rabbitmq 3.10.0 HOT 3
- Auto generate deduplication checksum base on payload or configurable properties
- 0.5.3 queues not working under rabbitmq 3.8.17 HOT 1
- Can key be removed from cache, after BasicAck and some time has ellapsed? HOT 3
- {:plugins_not_found, [:rabbitmq_message_deduplication]} HOT 2
- Policies support HOT 4
- Bug - when the rabbitmq is restarted the messages in a persistent queue are not deduplicated HOT 2
- Exchange level doesn't work but queue level does. What's differrence on using these two?
- Queue crashed with reason timeout to call RabbitMQMessageDeduplication.CacheManager HOT 7
- `x-cache-size` seems to be manadatory, not optional. HOT 1
- Queue level deduplication seems broken HOT 8
- Exchange info crash after upgrade to 0.6.0 HOT 5
- Debugging exchanges HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-message-deduplication.