Too many queue and rqueue don't working anymore about rqueue HOT 15 CLOSED

Hi @j0nathan33
I hope there's no bug as such, please check if you can do this

I will try to give you feedback this week.

Thanks

from rqueue.

OMG! Why do you need 72 queues on one machine? I think it's having starvation issue, can you please add debugger in the

from rqueue.

The other to way solve this problem is run Rqueue on sub set of queues and you've multiple clusters for this something like this

Cluster 1 => [Q1, Q2, Q3, ... Q20 ]
Cluster 2 => [Q21, Q22, ..., Q40]
Cluster 3 => [Q41, Q42, ..., Q60]
Clustern 4 => [Q61, Q62, .. Q80]

Etc, you can use activate=false flag to disable/enable in respective clusters.

from rqueue.

If you think 73 queues can be handled by one system then you can try to group these queues using prirorityGroup and priority field.

Read more details here https://github.com/sonus21/rqueue/wiki/Queue-Priority

from rqueue.

@j0nathan33 I would love to hear your feedback about Rqueue, if you have got some time please provide your feedback here.

Feedback

from rqueue.

I try to create another cluster and problem is solve. If I compare with dev enviromnent with new cluster and prod enviromnent, cpu is too high in prod. In prometheus, I receive alarm of "CPUThrottlingHigh". In prod right now, It have 0 job in queue. In dev enviromnent, I have 1 job in queue. Tomorrow, I will check class DefaultRqueuePoller.

FYI: Application is not deployed in only one machine. In prod, we use 3 pod(Kubernetes) for this app. In prod, I don't know why, but sometime, rqueue stop working maybe it's related to that bug. When we have this bug, application is responding with health check and we have some scheduler(spring) was run.

I will try to give you feedback this week.

from rqueue.

Thanks for the update, do you know when Rqueue stops processing messages, any error around that time?
Are you exporting queue metrics into prometheus, if so can you please share processing message queue statistics?

processing.queue.size : Currently being processed message count
queue.size : Number of items waiting to be processed

You can group this data based on the instance info, if you see queue.size > 0 for one minute and processing.queue.size == 0 means something is not working.

Running Rqueue on 3 machine/PODs won't help if you run 72 queues on each machine, this happens since Rqueue will always have 72 jobs in thread pool to poll messages (classical problem with consumer) from queue and each of them will sleep for polling interval if there're no message in the queue. It seems most of the CPU is consumed by these 72 jobs and you have no or small CPU units to process message(execute listener method call). Since you've 72 queues, its default thread pool size would be 3 * 72 => 216, 2*72 => 144 (core pool size). At peak you're running 216 jobs and that's where CPU fight occurs and your jobs are getting throttled.

Also there're other jobs fighting for CPU, like scheduled worker that moves messages from processing-queue to main-queue due to job failure, also we've other jobs that moves messages from scheduled-queue to main-queue, but we run handful number of jobs here.

Also, please search for TaskRejectedException error in the log, if you see such errors please report, also see if there're any logs from org.springframework.scheduling.concurrent, java.util.concurrent, org.springframework.core.task and org.springframework.scheduling packages.

from rqueue.

I check in Kibana(logs) for last 4 months and It have 0 log for TaskRejectedException, org.springframework.scheduling.concurrent, java.util.concurrent, org.springframework.core.task and org.springframework.scheduling. In past when I see this problem, It have 0 logs. The only log I have, It's for start of spring.

For queue statistics, right now in prod, we don't have this metrics on prometheurs and change will be apply on next release.

In general, we don't have all queue with 1 job at same time. Some queue, we have only job at midnight for do some synchronization with external tool.

from rqueue.

I debug class DefaultRqueuePoller and I see CorePoolSize is too lower in ThreadPoolExecutor.execute . Locally, I change number to greater number for CorePoolSize and problem is solve.

from rqueue.

Did you set task executor or Rqueue's some bug that had computed corePoolSize to be low?
As I explained above it's core pool size = 2*queue count (2*73=146 in your case) and max pool size should be3* queue count(3*73=219).

from rqueue.

Ah, I see problem in my case. I created new ThreadPoolTaskExecutor because I need to set taskDecorator and I can't set in default taskExecutor. Do you have idea for set taskDecorator in default taskExecutor? If is ok for you, I can add new function for set value.

from rqueue.

What's your use-case for taskDecorator? There're two types of tasks 1. Long Running Message Poller 2. Message Executor.

Currently there's no support of task decorator though it can be added.

from rqueue.

I use task decorator for initialise session and transaction for db. After check my code, I can add annotation Transactional on each function with rqueue annotation.

from rqueue.

After check my code, I can add annotation Transactional on each function with rqueue annotation.

Yes, that's correct you can add @Transactional on method or on class and this would work. You can also add other annotation like @NewSpan from sleuth and others.

Also, you can use Middleware to start/end transaction(s) see https://github.com/sonus21/rqueue/wiki/Middleware

from rqueue.

It's ok :). I fix it last week :)

from rqueue.