noahhl / batsd Goto Github PK

We see lots of errors in the log like this:
2015-10-21 11:54:32 +0300: Uncaught error Infinity
2015-10-21 11:54:32 +0300: Uncaught error Infinity
2015-10-21 11:54:32 +0300: Uncaught error Infinity
2015-10-21 11:54:32 +0300: Uncaught error Infinity
2015-10-21 11:54:32 +0300: Uncaught error Infinity

What can the issue be ?

I'm using batsd to store instrumentation form a Rails app. Naturally, some of the methods have a ! in their name. It looks like this line in the receiver prevents those measurements from being stored:

key, value, type, sample = row.split(/\||:|!/)

1.9.3p194 :010 > "test!:1|c".split(/\||:|!/)
 => ["test", "", "1", "c"] 
1.9.3p194 :011 > "test:1|c".split(/\||:|!/)
 => ["test", "1", "c"] 

Is it necessary to split on bangs too? I looked into the statsd protocol and cound't find any reason for that. I can make a quick pull request with the changes if this turns out to be a bug.

I am seeing a large number of CLOSE_WAIT connections to redis (on the same box) left open by the batsd-server process. I don't see anything in the code that would explicitly call @redis.quit on the redis connections that are opened in Server#post_init.

Under what circumstances would the server process clean up these connections. We've been seeing the server hang after periods of a day or two of uptime. I'm wondering if this is related.

Thanks.

Hi,

I'm currently trying to integrate batsd into our metrics infrastructure but can't seem to get it up and running 100%.

The redis 'datapoints' set is only updated for the first metric that comes in after start of the batsd receiver, the following metrics are added into redis (I see them using redis-cli KEYS *) but the names are not added to datapoints. (SMEMBERS datapoints)

Therefore batsd only 'exposes' a few keys to batsd-dash.

I first tried master, then switched to dev branch on jruby using redis 2.6.12.

Were am I going wrong ?

regards

Bart

Using RVM off a fresh jruby install I get a LoadError on the json gem when running:

bundle exec ruby --1.9 bin/batsd -c config.yml

This is using jruby-1.6.5 but I've also tried jruby-head with the same results.

Hi,

Using the ruby client I get the following weird results:

1.9.3p327 :024 > Time.now
=> 2013-06-26 18:32:31 +0200
1.9.3p327 :025 > batsd = BatsdReportingClient.new
=> #<BatsdReportingClient:0x0000000382ef30 @ipaddr="127.0.0.1", @PORT=8127, @timeout=2000, @Remote=#<TCPSocket:fd 11>>
1.9.3p327 :026 > batsd.values("counters:pushes.830.opened", 10.days.ago.to_i, Time.now)
=> [{:timestamp=>2013-06-26 16:20:00 +0200, :value=>1.0}]
1.9.3p327 :027 > batsd.values("counters:pushes.830.opened", 1.days.ago.to_i, Time.now)
=> []

With a retention configuration file looking like:

retentions:
1: 600 #1 hour
10: 360 #1 hour
60: 10080 #1 week
600: 52594 #1 year

Requesting data with a retention within the 2nd retention interval doesn't return anything, while asking for one within the 3rd interval does. The point is about 2 hours old so both should work.

Any idea why could that happen, and how to debug this? Thank you.

I'm trying batsd, but whenever I send a stat, after a while it will pop an exception in the console (but data is saved to redis)

Thread #<Thread:0x000000015e0de8> error: ERR unknown command 'eval' ERR unknown command 'eval' /home/luca/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis/client.rb:43:in `call'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:1647:in `block in method_missing'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:55:in `block in synchronize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:49:in `call'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:49:in `block (2 levels) in initialize'
~/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:49:in `block in initialize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:55:in `call'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:55:in `synchronize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0.rc1/lib/redis.rb:1646:in `method_missing'
~/Desktop/batsd/lib/batsd/redis.rb:74:in `get_and_clear_key'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:100:in `block (5 levels) in flush'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:98:in `each'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:98:in `block (4 levels) in flush'
~/Desktop/batsd/lib/batsd/threadpool.rb:17:in `call'
~/Desktop/batsd/lib/batsd/threadpool.rb:17:in `block (3 levels) in initialize'
~/Desktop/batsd/lib/batsd/threadpool.rb:14:in `loop'
~/Desktop/batsd/lib/batsd/threadpool.rb:14:in `block (2 levels) in initialize'

if I update redis gem to the latest 3.0.0 stable, it will give me this instead:

Thread #<Thread:0x00000002d8c7d0> error: undefined method `length' for 1:Fixnum undefined method `length' for 1:Fixnum ~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0/lib/redis.rb:2122:in `block in _eval'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0/lib/redis.rb:36:in `block in synchronize'
~/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0/lib/redis.rb:36:in `synchronize'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0/lib/redis.rb:2121:in `_eval'
~/.rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.0/lib/redis.rb:2148:in `eval'
~/Desktop/batsd/lib/batsd/redis.rb:74:in `get_and_clear_key'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:100:in `block (5 levels) in flush'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:98:in `each'
~/Desktop/batsd/lib/batsd/handlers/counter.rb:98:in `block (4 levels) in flush'
~/Desktop/batsd/lib/batsd/threadpool.rb:17:in `call'
~/Desktop/batsd/lib/batsd/threadpool.rb:17:in `block (3 levels) in initialize'
~/Desktop/batsd/lib/batsd/threadpool.rb:14:in `loop'
~/Desktop/batsd/lib/batsd/threadpool.rb:14:in `block (2 levels) in initialize'

e.g., If a timer is very sparse, you can end up with a situation where it gets recorded in its short duration intervals correctly, but then zeros are recorded until the next time it gets the longest duration flush. The crude solution (which statsd-server used to do) is to just not write zero values for timers. Maybe there's a better way.

Can just check for redis version and use multi-command version for Batsd::Redis#get_and_clear_key and Batsd::Redis#extract_values_from_string. Will be slower (requires more round trips to Redis, string split is slower in JRuby than Lua), but allows for more flexibility.

I think it should, if you mimic his interface and idea, it should be one of the first references you make, right?

I'm trying to test some tools built around batsd and I want to load up the database with data from a long time period quickly. Is there a way to force data in at certain times quickly to simulate a long run of data collection? Thanks

First of all: Great work! I have been using batsd for some months now and it works great for me.

I just wanted to clean up in my batsd database and remove old metrics that are not interesting to me anymore and noticed that that's actually harder than I thought: I would have to delete the corresponding keys in redis-cli (DEL command), and then figure out what data files belong to that metric. Therefore I would have to build MD5 hashes for the metric name and all rentention keys, check if each file exists, and if it does delete it. If the containing directory is empty then, I'd like to delete that too.

I think something like ./bin/batsd delete <key> would be really nice to simplify things. Another way might be to extend the management interface on port 8126 by this 'delete' function. What do you guys think about that?

I'm getting a 404 when I hit any of the documentation pages - is it just me?