scribd / elasticache-slowlog-to-datadog Goto Github PK

If (SLOWLOG events per flush interval) > 10, then source data is being truncated, because the SLOWLOG get method returns 10 entries by default, configurable up to the slowlog-max-len parameter.

By default on AWS, slowlog_max_len is 128, however this is tuneable. Therefore, we want this to be tunable or dynamic here as well. CONFIG generally isn't available in Elasticache, so slowlog-max_len might need to be discovered through another means, perhaps by dynamically increasing the get length until the last entry index value (the first element of an entry) == 0.

if a SLOWLOG reset was performed, index 0 won't be available to us, so we'll have to watch for this case as well, and have a fallback mechanism for this, perhaps watching for the first entry to fall off. In addition, since we are a lambda job, we may startup after a rolloff has already occurred.

Recommendation: start with SLOWLOG get 128; look for index 0, or length < 128; else increase get length. If response length increases, repeat until it does not, or index 0 is found.

If the (SLOWLOG events per flush interval) exceed slowlog_max_len, then the source data will still be truncated. There are two approaches to resolve this. One is to reduce the flush interval. The flush interval is currently designed to be invoked from without, as a cloudwatch timer event. Adjusting this timer calls for re-architecture as a container. The other option is to create an independent collection interval and store flushable data in a buffer until the call is received.
Both approaches means adding an internal clock and call into question the need for the external clock (lambda).

Recommendation: add an internal clock and buffer to collect at a more frequent interval. Possibly, forgo the external clock entirely. Possibly rearchitect as a container.

Critical exception from handler
{
"errorMessage": "undefined method `[]' for nil:NilClass",
"errorType": "Function<NoMethodError>",
"stackTrace": [
"/var/task/lib/slowlog_check.rb:119:in `did_i_get_it_all?'",
"/var/task/lib/slowlog_check.rb:126:in `redis_slowlog'",
"/var/task/lib/slowlog_check.rb:132:in `slowlogs_by_flush_interval'",
"/var/task/lib/slowlog_check.rb:210:in `ship_slowlogs'",
"/var/task/lambda_function.rb:64:in `lambda_handler'"
]
}

did_i_get_it_all? should return true when []

Two things, redis-rb needs to be connect with cluster mode, and parsing of the endpoint parameter needs to identify which mode to connect with.

An example of cluster-mode enabled configuration endpoint is:
blahblah-redis.abcdef.ab.1234.use2.cache.amazonaws.com

scribd/terraform-elasticache-slowlog-to-datadog#2 removed leading or trailing slashes from the terraform module, but this is a confusing interface.

It would be better to accept both forms and chop as appropriate.

Rather than send nothing at all, when the bucket period has been passed by a reasonable amount of time, send zeros.