hiredis 0.5.0 introduces memory leak about hiredis-rb HOT 14 CLOSED

FYI, here's the core event loop from the process on that box:
https://github.com/mroth/emojitrack/blob/e12080311e79c6acbe4af50d472a314c860ab309/feeder.rb#L72-L89

5 redis commands in a pipeline. This loop is consistently executed approximately ~250-500 times per second.

from hiredis-rb.

That's not good... Let me look at the diffs again and see if I overlooked something initially.

from hiredis-rb.

Which version of Ruby are you using?

from hiredis-rb.

ruby 2.0.0p451 (2014-02-24 revision 45167) [x86_64-linux]

from hiredis-rb.

Thanks, I'm taking a look.

from hiredis-rb.

I cannot reproduce. Memory consumption of the Ruby process is perfectly stable, while its executing more than 100k commands per second. Could you try running the script below in the same environment to check that we're using the same environment and see the same result?

My Ruby: ruby 2.0.0p451 (2014-02-24 revision 45167) [x86_64-linux]

My test script:

require "hiredis"

c = Hiredis::Connection.new
c.connect("localhost", 6379)

loop do
  16.times do |i|
    c.write(["SET", "key:%010d" % i, i])
  end

  16.times do |i|
    c.read
  end
end

from hiredis-rb.

Thanks for looking into this. I'll try to get a new environment set up to test in soon (that one is from my production environment so I can't throw the extra things on, need to get a new box provisioned). In the meantime, the main differences I can see between my production code and your test script that might be significant(?) are the following:

• I'm including hiredis by setting it as the driver for the redis gem, e.g.

require 'redis'
Redis.new(
   :host => REDIS_URI.host,
   :port => REDIS_URI.port, 
   :password => REDIS_URI.password, 
   :driver => :hiredis)

• My commands are wrapped in a PIPELINE
• I'm using PUBLISH to send medium sized JSON blobs as strings
• I also use ZINCRBY, LPUSH, and LTRIM, but the above two strike me as potentially more memory intensive?

from hiredis-rb.

The script I posted does show a memory leak on a different machine where I ran it with the same Ruby version. It's something. Now to find where the leak was introduced and what's going on.

from hiredis-rb.

The leak was introduced in f85e8c6.

from hiredis-rb.

Confirmed that the leak exists if I use Ruby's rb_fdset_t API. If I use the previous fd_set standard library API, all is well. The same behavior shows on MRI 1.9.2, 1.9.3, and 2.0.0.

from hiredis-rb.

Fixed in f7d9b1e. I pushed version 0.5.1 to RubyGems.

Many thanks for kicking the tires on 0.5.0 and reporting this bug!

from hiredis-rb.

👍 fast fix! I'm going to re-apply the update on my production server and will monitor and the memory usage and report back in a few hours.

from hiredis-rb.

Rock solid, you can see the instance restart from the upgrade at around 10:30. All good!

from hiredis-rb.

Thanks again for testing and confirming the fix works for you, I appreciate it!

from hiredis-rb.