shopify / memcached_store Goto Github PK

View Code? Open in Web Editor NEW

36.0 456.0 21.0 203 KB

ActiveSupport MemcachedStore

License: MIT License

Ruby 100.00%

memcached_store's Introduction

ActiveSupport MemcachedStore

This gem includes two memcached stores.

MemcachedStore

ActiveSupport memcached store. This wraps the memcached gem into a ActiveSupport::Cache::Store, so it could be used inside Rails.

MemcachedSnappyStore

ActiveSupport cache store that adds snappy compression at the cost of making the incr, decr operations unavailable.

Installation

Add this line to your application's Gemfile:

gem 'memcached_store'

And then execute:

$ bundle

Or install it yourself as:

$ gem install memcached_store

Usage

In your environment file:

# for memcached store
config.cache_store = :memcached_store,

# for snappy store
config.cache_store = :memcached_snappy_store,
  Memcached.new(['memcached1.foo.com', 'memcached2.foo.com'])

Benchmarks

For benchmarks please refer to https://github.com/basecamp/memcached_bench.

Code status

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Added some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

memcached_store's People

Contributors

Stargazers

Watchers

Forkers

atteeela pvijay127 beiersi jdm-twtr rsouza-carecloud hirocaster ursm alexsnaps bluemutedwisdom saiqulhaq jwcooper isabella232 byroot ktsalz qpc-github

memcached_store's Issues

release the gem

seems like this gem is not release to rubygems yet..
We probably should.
Maybe even change the name to something like: memcached_stores

Enable travis ci

We need to enable travis for this project:

go to https://travis-ci.org/profile/Shopify.
tick the button besides the project name.

@jduff can you do that?

thanks

TypeError after upgrading to 1.1.0

I thought it's related to the removal of Memcached::Rails. Will it be fixed by flushing the memcached server?

gems/memcached-1.8.0/lib/memcached/memcached.rb:728:in `memcached_cas': no implicit conversion of ActiveSupport::Cache::Entry into String (TypeError)
    from gems/memcached-1.8.0/lib/memcached/memcached.rb:728:in `single_cas'
    from gems/memcached-1.8.0/lib/memcached/memcached.rb:451:in `cas'
    from gems/memcached-1.8.0/lib/memcached/rails.rb:90:in `cas'
    from gems/memcached_store-1.1.0/lib/active_support/cache/memcached_store.rb:80:in `block (2 levels) in cas'
    from gems/activesupport-5.0.2/lib/active_support/cache.rb:562:in `block in instrument'
    from gems/activesupport-5.0.2/lib/active_support/notifications.rb:166:in `instrument'
    from gems/activesupport-5.0.2/lib/active_support/cache.rb:562:in `instrument'
    from gems/memcached_store-1.1.0/lib/active_support/cache/memcached_store.rb:79:in `block in cas'
    from gems/memcached_store-1.1.0/lib/active_support/cache/memcached_store.rb:248:in `handle_exceptions'
    from gems/memcached_store-1.1.0/lib/active_support/cache/memcached_store.rb:78:in `cas'
    from gems/identity_cache-0.5.1/lib/identity_cache/cache_fetcher.rb:30:in `fetch'
    from gems/identity_cache-0.5.1/lib/identity_cache/memoized_cache_proxy.rb:57:in `fetch'
    from gems/identity_cache-0.5.1/lib/identity_cache.rb:115:in `fetch'
    from gems/identity_cache-0.5.1/lib/identity_cache/configuration_dsl.rb:259:in `attribute_dynamic_fetcher'
    from gems/identity_cache-0.5.1/lib/identity_cache/configuration_dsl.rb:199:in `fetch_id_by_token'
    from gems/identity_cache-0.5.1/lib/identity_cache/configuration_dsl.rb:54:in `fetch_by_token'

Add session store implementation

As we already have the cache_store implementation, will be nice to implement the session_store one too, So people can just plugin in this gem and use it for session and cache.

Add benchmarks

We should add some benchmarks to compare this store, against Dalli, and MemCache stores.

Problem with offical Ruby docker images connecting to Heroku's Memcached (Memcached::FailedToSendAuthenticationToServer)

Reproduction:

Dockerfile:

FROM ruby:2.7.1

RUN apt-get update \
    && apt-get -y upgrade \
    && apt-get -y install build-essential libsasl2-dev

RUN gem install memcached_store pry

Build and run pry using docker build -t ruby-memcache-lab . && docker run --rm -it ruby-memcache-lab pry
Run the following to connect to a Heroku memcached instance:

require 'memcached_store'
m = Memcached.new(["DOMAIN:PORT"], credentials: [ "USER", "PASSWORD" ])
m.get('name')

I get the following error:

Memcached::FailedToSendAuthenticationToServer: Key {"name"=>"DOMAIN:PORT:8"}
from /usr/local/bundle/gems/memcached-1.8.0/lib/memcached/memcached.rb:616:in `reraise'

While doing the same steps with the following Dockerfile will work without any issue:

FROM ubuntu:20.04

RUN apt-get update \
    && apt-get -y upgrade \
    && apt-get -y install build-essential ruby-full libsasl2-dev

RUN gem install memcached_store pry

I tried other official Ruby docker images getting the same error (FailedToSendAuthenticationToServer). It seems to be related to the native libmemcached native library but I wonder if the gem builds the module locally how it behaves differently on different distros?

Update readme

Need to update the readme to have instructions on how to add the 2 different cache_stores that this gem has now.

Memcached-store with `compress: true` is giving terrible response time

We used to use dalli-2.7.x gem for storing and fetching cache for our rails app.
With Dalli-3.2.x there were sever performance degradation, and thats why we decided to switched to memcached-store for now.
But what we have observed that, memcached-store with compress: true is consuming a lot heap memory and increasing response time by double or triple, and we can see from trace that it's stuck in the cache for 75% or the response time.

And temporary to make it work, we have turned of compress by compress: false, by which the response time is back to normal, but we don't understand why. We store a lot in memcache, most of them are also with complex structure.

Can anyone please let me understand what kind of next steps we can take, because with compress: false it will consume more cache memory and can store less ?

The local store implementation is causing a double serialization

When fetching a value from Memcached we deserialize it with the actual coder, but then it is put into the local cache and there it's serialized again but with Marshal to make a copy

This is wasteful, we should be able to store the payload we got from Memcached directly.

Ref: rails/rails#48302

cc @felix-d @Shopify/rails

Version 1.2.2 can't work with Bootsnap

my app works fine with 1.2.1

/Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/active_support.rb:43:in `load_missing_constant': uninitialized constant ActiveSupport::Cache::MemcachedStore (NameError)
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/memcached_store-1.2.2/lib/memcached_store/railtie.rb:5:in `block in <class:Railtie>'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:347:in `each'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:347:in `call'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing'
        from /Users/saiqulhaq/.code/repos/hh/rails-app/config/environment.rb:6:in `<main>'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/bootsnap-1.3.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application.rb:102:in `preload'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application.rb:153:in `serve'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application.rb:141:in `block in run'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application.rb:135:in `loop'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application.rb:135:in `run'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/gems/2.3.0/gems/spring-2.0.2/lib/spring/application/boot.rb:19:in `<top (required)>'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /Users/saiqulhaq/.rbenv/versions/2.3.7/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from -e:1:in `<main>'

Ruby 3.2.0 warnings from memcached gem (used in this gem memcached_store)

When using memcached_store on Ruby 3.2.0, we get the message:

/Users/phillipmuldoon/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/memcached-1.8.0/lib/rlibmemcached.bundle: warning: undefining the allocator of T_DATA class swig_runtime_data

The only consumer of the Memcached gem in our Gemfile.lock is memcached_store. It might seem to need an update in dependencies to a later version?

Support local cache read to use before attempting a CAS operation

cc @casperisfine

When using CAS to fill a cache entry, we want to read from the local cache to try to get a cache hit without going to the remote cache. With #66 the local cache is updated at the end of the CAS methods, but it isn't used for reading, which does make sense in other cases where we want to make sure the remote cache entry is updated to the correct value. We also can't seamlessly add local cache reading to the CAS methods, since the given block is also used to compute the updated value which can be expensive when involving database loads where we only want to do that after first reading from the remote cache.

Currently, Active Support doesn't support getting the local cache to read from directly. I also can't think of a good reason to use that with the methods Active Support supports, since it doesn't support CAS operations.

Adding support for local cache reads would avoid this problem and given the application flexibility in how it handles the local cache value. Currently, I don't see a need to expose local cache writes so I think we could avoid exposing the local cache object or encouraging the application to manage the updating of the local cache directly.

How to get get a default expiration in?

I think this is an issue of intercommunication problems between identity cache, memcached_store, and the memcached gem itself.

Given a config like this:

config.identity_cache_store = :memcached_snappy_store, Memcached.new(
  "127.0.0.1",
  support_cas: true,
  auto_eject_hosts: false,
  default_ttl: 13,
)
IdentityCache.cache_backend = ActiveSupport::Cache.lookup_store(*Rails.configuration.identity_cache_store)

the memcached instance will have a default ttl of 13, however in IdentityCache I see this:

def add(key, value)
  @cache_backend.write(key, value, :unless_exist => true) if IdentityCache.should_fill_cache?
end

(https://github.com/Shopify/identity_cache/blob/60f42585a9057a159a39ef09ee22343a3a5fa849/lib/identity_cache/cache_fetcher.rb#L84)

Ultimately this ends up in ActiveSupport:: Cache:: MemcachedStore#expiration:

def expiration(options)
  expires_in = options[:expires_in].to_i
  if expires_in > 0 && !options[:raw]
    # Set the memcache expire a few minutes in the future to support race condition ttls on read
    expires_in += 5.minutes.to_i
  end
  expires_in
end

a value for :expires_in is never passed along, so this is 0.

I think this is an interaction bug and not one of misconfiguration. Does that sound possible? Not sure best how to approach a fix on that (introduce a ttl to IdentityCache, perhaps?)

[Rails 5.1] TypeError: Expected argument 5 of type time_t, but got ActiveSupport::Duration XXX in SWIG method 'memcached_set'

When using mmemcached_store with IdentityCache in a Rails 5.1 application, the app fails with the exceptions:

TypeError: Expected argument 5 of type time_t, but got ActiveSupport::Duration X minutes and XX seconds in SWIG method 'memcached_set'

I quickly scanned the code and I found https://github.com/Shopify/memcached_store/blob/master/lib/active_support/cache/memcached_store.rb#L242 which will convert the expiration in a ActiveSupport::Duration and the coercion to integer will fail raising the exception above.

shopify / memcached_store Goto Github PK

memcached_store's Introduction

ActiveSupport MemcachedStore

MemcachedStore

MemcachedSnappyStore

Installation

Usage

Benchmarks

Code status

Contributing

memcached_store's People

Contributors

Stargazers

Watchers

Forkers

memcached_store's Issues

Recommend Projects

Recommend Topics

Recommend Org