Choosing only a subset of metrics to publish about rabbitmq-cloudwatch-exporter HOT 11 CLOSED

Issue is still valid.

In our case we have a lot of Queues in Bus, which are satisfiex by regex filter. The autoscaling for consumes is used just only one Metric, whcih is Queue.Messages. But with current version we have a lot all Queue Metrics which increase CloudWatch price a lot for business case with a lot of Messages in the Bus.

Still suggest to have something like this to manage cost in CloudWatch properly:
cloudwatch_exporter.metrics.queue = ["messages"] - just specify exactly metrics which are used for AutoScaling.

Thanks you,
Alex R.

from rabbitmq-cloudwatch-exporter.

I was thinking about something similar while developing this plugin. The main challenge is finding a proper syntax for inclusions/exclusions. The above example you provided for example would not work as RMQ sysctl syntax is not supporting lists.

Your above example would rather look like:

cloudwatch_exporter.metrics.queue.1 = messages_ready
cloudwatch_exporter.metrics.queue.2 = messages_unacknowledged

You can imagine how tedious would become the above syntax when extended to all metrics available.

from rabbitmq-cloudwatch-exporter.

I believe that the default should be to include all of them.
If the cloudwatch_exporter.metrics.queue list is defined, only those messages will be used.

from rabbitmq-cloudwatch-exporter.

Oh yes, we just published 400K metrics (stomp websocket) by mistake with a cost of ~$440. I am disabling queue metrics now – filter would help a lot. Some important queues always have the same name, I just need to get rid of dynamic queues appearing disappearing.

from rabbitmq-cloudwatch-exporter.

Sorry to hear that.

Avoiding the above is the reason why I opted for not publishing anything by default. Nevertheless, there's definitely room for improvement.

What about a regex based whitelist for entities such as queues, exchanges, connections and channels?

Something like

cloudwatch_exporter.metrics.1 = queue

cloudwatch_exporter.export_regex.queue = "foo.*|bar.*|.*baz"

If the filter is provided, only those entities which name matches the regex will be published into CW.
Omitting the filter would be pretty much the equivalent of:

cloudwatch_exporter.export_regex.queue = ".*"

from rabbitmq-cloudwatch-exporter.

@noxdafox I think that would be very helpful, yes!

from rabbitmq-cloudwatch-exporter.

Release 0.3.0 adds support for regex based entities selection. You can see a description with examples in the README.

from rabbitmq-cloudwatch-exporter.

from rabbitmq-cloudwatch-exporter.

Closing the issue, please re-open if the solution is not satisfying.

from rabbitmq-cloudwatch-exporter.

Hi,

I am facing the same issue - having a lot of exchanges/queues, but need only 1-2 metrics from each. Would it be possible to add also a regex filter for metrics names?

IMO it would be fine also as proposed before with the list:

cloudwatch_exporter.metrics.queue.1 = messages_ready
cloudwatch_exporter.metrics.queue.2 = messages_unacknowledged

It's something that you would set up once and not touch that often, I think it's totally acceptable with extra lines in the sake of reduced messages posted to CW.

P.S. Thanks for the great plugin, this is the best solution I've seen so far.

from rabbitmq-cloudwatch-exporter.

Hello,

I created issue #16 to reflect your issue. Please feel free to provide feedback regarding the configuration format proposal.

from rabbitmq-cloudwatch-exporter.