linsomniac / python-memcached Goto Github PK

A python memcached client library.

Makefile 0.18% Python 99.82%

python-memcached's Introduction

Overview

This software is a 100% Python interface to the memcached memory cache daemon. It is the client side software which allows storing values in one or more, possibly remote, memcached servers. Search google for memcached for more information.

This library is stable and largely in maintenance mode. Another library that is getting more active enhancements is pymemcache and they have links and a good set of comparisons between them on their page.

This package was originally written by Evan Martin of Danga. Please do not contact Evan about maintenance. Sean Reifschneider of tummy.com, ltd. has taken over maintenance of it.

Please report issues and submit code changes to the github repository at:

https://github.com/linsomniac/python-memcached

For changes prior to 2013-03-26, see the old Launchpad repository at:

Historic issues: https://launchpad.net/python-memcached

Testing

Test patches locally and easily by running tox:

pip install tox
tox -e py312

Test for style by running tox:

tox -e pep8

python-memcached's People

Stargazers

Watchers

Forkers

eguven ptsakiri sunliwen saschpe hughdbrown saily nghitran giocalitri soukingang wnyc underrun userrl matheuspatury fktrcfylh ahussein bgyss queertypes oremj zewt erankor msabramo cgc1983 katomaso idning alex aaron218 allen-smithee vlogic harlowja muhammadsa x-eye tylerdave jerith stalkerg charsyam nicholasserra xuziyan001 chop-dbhi ephemerallabs lexqt haridas gilgamezh honsys ehrmann identityleak aurthes itxaka gpulost caperea charlievrettos canadahelps jeff h-huihui suzaku jdaaaaaavid kmaglione markzhang90 degroat ibrahimgunduz34 pipermerriam ebas ymt2 saboia dieselmachine slam ltd-beget sheepcat dims deconevidya awingu scrapinghub jeltef mylokin rebelmouseteam charliemalicay youxixin888 dinabelova oreza free-free grg350 octupszhang vinicius-ronconi teej-minted stackdriver soleaf michael-iclinic hunza frewsxcv mthpower nashuiliang dzhuang mawkee tayfun romuald helkhalfi preston-landers pratikvasa sseg timgraham kevinfrommelt-wf

python-memcached's Issues

Discussion: Moving to a new maintainer.

For a variety of reasons that are ultimately unimportant, I have not been able to find any attention for maintaining this package. I've recently had an offer from Morgan Fainberg to take over maintenance of the package. That proposal includes importing it into the OpenStack CI infrastructure for testing and development. That would remove it from Github, as I understand it, and use the review.openstack.org gerrit instance, and releasing it on pypi.

If anyone has an opinion on this, please comment here.

delete returns nonzero on NOT_FOUND

It appears that delete will return nonzero even on failure (NOT_FOUND server response). This appears to violate the api description that says "@return: Nonzero on success." The memecached text protocol defines only 'DELETED' as "success", and 'NOT_FOUND' to indicate the server did not have the key at the time the delete was requested -- and records this in server stats as 'delete_misses'.

The fix is trivial (simply remove NOT_FOUND from the expect list for delete), and I was going to just open a pull req for it, but there appears to be some history behind the decision.

It appears this change was part of 81933bc and related to an old-bug (at least this is the bug referenced in the commit).

Is there historical/deeper meaning surrounding this change?
Is it a bug or just a kludge to maintain some pre-existing behavior (e.g. api stability)?

set_multi fails with tens of thousands of keys

version 1.53

Error from memcached: Failed to write, and not due to blocking: Connection reset by peer

data = zip(range(1000000,1200000), range(1, 200000))
set_multi_results = set_multi(dict(data))
get_many_results = get_multi(range(1000000, 1200000))

set_multi_results contains all of the keys, because the server died.
get_many_results will contain some arbitrary fraction of keys that were actually set, testing on my system was about 30,000.

Installation of PyPI package is really slow.

Installing python-memcached over PyPI takes ages.

I suspect some url configuration on the PyPI package causes this.
Could you fix this issue?

Below is my console output for debugging this:

$ pypi-show-urls -v -p python-memcached 

Download candidates for python-memcached
========================================

  Candidates from https://pypi.python.org/simple/python-memcached/
  ----------------------------------------------------------------
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.51.tar.gz#md5=580603a82b7078642ed18ea000f825c5
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.48.tar.gz#md5=58f8c328304df6aca1f8b60170e98932
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.31.tar.gz#md5=e1de1fa7d504bb9ce0f9e295fd77adb0
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.47.tar.gz#md5=e4e9d65e5721a1bb01f8d657ddf3f03e
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.53.tar.gz#md5=89570d26e7e7b15caa668a6b2678bd3c
    https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.52.tar.gz#md5=b80e2d024470e9baf693d016c9e96d6a
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.45.tar.gz
    http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.46.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.44.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.43.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.38.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.41.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.42.tar.gz
    http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.47.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.40.tar.gz
    ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz
    http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.48.tar.gz

  Candidates from http://www.tummy.com/Community/software/python-memcached/
  -------------------------------------------------------------------------
    ftp://ftp.tummy.com/pub/python-memcached/python-memcached-latest.tar.gz

  Versions only available externally
  ----------------------------------
    1.38
    1.42
    1.45
    1.44
    1.46
    1.41
    1.40
    1.43
    latest


$ pip install -vUI python-memcached
Downloading/unpacking python-memcached
  Could not fetch URL ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz (from http://f.pypi.python.org/simple/python-memcached/): <urlopen error ftp error: [Errno ftp error] 550 Failed to change directory.>
  Will skip URL ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz when looking for download links for python-memcached
  Using version 1.53 (newest of versions: 1.53, 1.52, 1.51, 1.48, 1.48, 1.47, 1.47, 1.46, 1.45, 1.44, 1.43, 1.42, 1.41, 1.40, 1.38, 1.31, latest, latest)
  Using download cache from /Users/diederik/Library/Caches/pip-downloads/http%3A%2F%2Ff.pypi.python.org%2Fpackages%2Fsource%2Fp%2Fpython-memcached%2Fpython-memcached-1.53.tar.gz
  Running setup.py egg_info for package python-memcached
    running egg_info
    creating pip-egg-info/python_memcached.egg-info
    writing pip-egg-info/python_memcached.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/python_memcached.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/python_memcached.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.rst'
    warning: no files found matching '*.txt'
    warning: no files found matching 'MakeFile'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.gitignore' found anywhere in distribution
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    writing manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
Installing collected packages: python-memcached
  Found existing installation: python-memcached 1.53
    Uninstalling python-memcached:
      Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.py
      Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.pyc
      Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/python_memcached-1.53-py2.7.egg-info
      Successfully uninstalled python-memcached
  Running setup.py install for python-memcached
    running install
    running build
    running build_py
    creating build
    creating build/lib
    copying memcache.py -> build/lib
    running install_lib
    copying build/lib/memcache.py -> /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages
    byte-compiling /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.py to memcache.pyc
    running install_egg_info
    running egg_info
    writing python_memcached.egg-info/PKG-INFO
    writing top-level names to python_memcached.egg-info/top_level.txt
    writing dependency_links to python_memcached.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'python_memcached.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.rst'
    warning: no files found matching '*.txt'
    warning: no files found matching 'MakeFile'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.gitignore' found anywhere in distribution
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    writing manifest file 'python_memcached.egg-info/SOURCES.txt'
    Copying python_memcached.egg-info to /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/python_memcached-1.53-py2.7.egg-info
    running install_scripts
    writing list of installed files to '/var/folders/24/5y_gshzd7vg9f5mv_hpxf1y40000gn/T/pip-JNRJb5-record/install-record.txt'
Successfully installed python-memcached
Cleaning up...
  Removing temporary dir /Users/diederik/Sites/virtualenvs/demo-env/build...

Catching _ConnectionDeadError in _deletetouch

It seems that handler of _ConnectionDeadError is missed.

        try:
            server.send_cmd(cmd)
            line = server.readline()
            if line and line.strip() in expected: return 1
            self.debuglog('%s expected %s, got: %s'
                    % (cmd, ' or '.join(expected), repr(line)))
        except socket.error, msg:
            if isinstance(msg, tuple): msg = msg[1]
            server.mark_dead(msg)
        return 0

Google code style formatted docstrings

I'd like to update the codebase to use Google's coding style for docstrings:

Args:
    path (str): The path of the file to wrap
    field_storage (FileStorage): The :class:`FileStorage` instance to wrap
    temporary (bool): Whether or not to delete the file when the File
       instance is destructed

Returns:
    BufferedFileStorage: A buffered writable file descriptor

Doing so would enable it to work with Sphinx using the Napoleon extension, but did not want to put forth the work if it would not be accepted.

Would you be willing to accept such a pull-request if it is clean, doesn't break anything, etc? I could also add a docs dir with sphinx autodoc so a readthedocs.org site could be setup.

python-memcached 1.53 wheel package has 1.55 code in it

Hi,

I just noticed that some of my TravisCI tests started failing. They depend on python-memcached==1.5.3, and the failure looks like this:

ERROR: test_cache_is_working (bulk_sms.tests.test_sending.MultiprocSendingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/<projectname>/bulk_sms/tests/test_sending.py", line 330, in test_cache_is_working
    self.assertTrue(cache.get('CACHETEST', default=False),
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 82, in get
    val = self._cache.get(key)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1089, in get
    return self._get('get', key)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1073, in _get
    return _unsafe_get()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1061, in _unsafe_get
    value = self._recv_value(server, flags, rlen)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1235, in _recv_value
    val = int(buf)
ValueError: invalid literal for int() with base 10: 'True'

When I look at this page, the source package has different code than the wheel package:

Specifically, line 93 of memcache.py inside the wheel package says this:

__version__ = "1.55"

Would it be possible to upload a new wheel package with 1.53 code in it? Thanks for your work modernizing this package!

problem setting a value

hi,
Im working on linux 12.04 64 bits
and i getting the following error:

File "build/bdist.linux-x86_64/egg/memcache.py", line 704, in set
    return self._set("set", key, val, time, min_compress_len, noreply)
  File "build/bdist.linux-x86_64/egg/memcache.py", line 1024, in _set
    return _unsafe_set()
  File "build/bdist.linux-x86_64/egg/memcache.py", line 998, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "build/bdist.linux-x86_64/egg/memcache.py", line 962, in _val_to_store_info
    pickler.dump(val)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

when i run tests i get this :

py27 develop-inst-nodeps: /var/www/boorar.git
py27 installed: The directory '/home/ricardo/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,coverage==4.0.3,flake8==2.2.4,hacking==0.10.2,mccabe==0.2.1,nose==1.3.7,pbr==1.8.1,pep8==1.5.7,pyflakes==0.8.1,-e git+https://github.com/linsomniac/python-memcached.git@37f55ca4ad94ca4ade30d6be28e1facb79ac3182#egg=python_memcached-master,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='4142489713'
py27 runtests: commands[0] | nosetests
...MemCached: while expecting 'DELETED', got unexpected response 'NOT_FOUND'
MemCached: while expecting 'DELETED', got unexpected response 'NOT_FOUND'
....MemCached: while expecting 'STORED', got unexpected response 'SERVER_ERROR object too large for cache'
......MemCached: MemCache: inet:127.0.0.1:11211: test.  Marking dead.
.....MemCached: MemCache: inet:memcached:11211: connection closed in readline().  Marking dead.
MemCached: MemCache: inet:memcached:11211 (dead until 1450219647): connection closed in readline().  Marking dead.
.
----------------------------------------------------------------------
Ran 19 tests in 0.261s

OK
py27 runtests: commands[1] | python -c import memcache; memcache._doctest()
Doctests: TestResults(failed=0, attempted=19)
_______________________________________________ summary ________________________________________________
  py27: commands succeeded
  congratulations :)

and this is my code:

import traceback
from memcache import Client

mc = Client(servers=['localhost:8000'])

try:
    # key = "0_acf2c7795a8d7fc8c6b321ef9e3103960a0d699b6e3df12b428bebff"
    # value = "0"
    return mc.set(key, value, time, min_compress_len, noreply)
except Exception, e:
    traceback.print_exc()

can you help me to figure out what is happening?

EDIT:

value was a result from psycopg2, i converted to json and worked:

import json

value = json.loads(json.dumps(value))
...

Changing SERVER_MAX_VALUE_SIZE doesn't work

Apparently, changing the value of server_max_value_length in the memcache module does NOT change the value inherited by the clients.

From looking at the source, I can not for the life of me figure out why!

This example was run on Ubuntu 13.04, and I've also observed it on a remote machine running Ubuntu 12.04 (on Amazon cloud).

Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> memcache.SERVER_MAX_VALUE_LENGTH
1048576
>>> memcache.SERVER_MAX_VALUE_LENGTH *= 10
>>> memcache.SERVER_MAX_VALUE_LENGTH
10485760
>>> c = memcache.Client(['localhost'])
>>> c.server_max_value_length
1048576
>>> memcache.SERVER_MAX_VALUE_LENGTH
10485760

Failover support?

Is it possible to make client automatically skip a server in case it went down?
We have a cluster of 4 machines. One of them is down which cause application servers to to pause (Default config) and fill up the queue.

I found failover config in PHP Client. Is there counterpart in this client? What's the best way to deal with this?

can't install python-memcached1.58 in Python 3.6.0b2

here is the traceback:

Traceback (most recent call last):
File "setup.py", line 8, in
version=get_module_constant('memcache', 'version'),
File "/home/ubuntu/.myvenv/lib/python3.6/site-packages/setuptools/depends.py", line 164, in get_module_constant
return extract_constant(code, symbol, default)
File "/home/ubuntu/.myvenv/lib/python3.6/site-packages/setuptools/depends.py", line 195, in extract_constant
const = code.co_consts[arg]
IndexError: tuple index out of range

Consistent Hashing - Ketama

Hi,

I have the working implementation of ketama consistent hashing algorithm on top of your library. May I send you pull request ? Or you are keeping the library as minimal as possible ? Please let me know, I would love to share my implementation.

Warm Regards,
Haridas N.

ImportError on six when running tests

Following the docs on running tests and hitting an ImportError while running. Six is definitely installed, not sure if this is a venv issue, or something that's been brought up before?

(memcached)nick@localhost python-memcached (master) $ tox -e py27
py27 create: /Users/nick/Code/python-memcached/.tox/py27
py27 installdeps: -r/Users/nick/Code/python-memcached/requirements.txt, -r/Users/nick/Code/python-memcached/test-requirements.txt
py27 develop-inst: /Users/nick/Code/python-memcached
py27 installed: argparse==1.3.0,coverage==3.7.1,flake8==2.2.4,hacking==0.10.2,mccabe==0.2.1,nose==1.3.7,pbr==0.10.8,pep8==1.5.7,pyflakes==0.8.1,-e git+https://github.com/nicholasserra/python-memcached.git@664bd3e23fe500fbde4c70636e2d24c8fd2f35af#egg=python_memcached-master,six==1.9.0,stevedore==1.2.0,virtualenv==12.0.2,virtualenv-clone==0.2.5,virtualenvwrapper==4.3.1,wsgiref==0.1.2
py27 runtests: PYTHONHASHSEED='3267181802'
py27 runtests: commands[0] | nosetests
EE
======================================================================
ERROR: Failure: ImportError (No module named six)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/nick/Code/python-memcached/tests/test_memcache.py", line 5, in <module>
    import six
ImportError: No module named six

======================================================================
ERROR: Failure: ImportError (No module named six)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/nick/Code/python-memcached/tests/test_setmulti.py", line 17, in <module>
    import memcache
  File "/Users/nick/Code/python-memcached/memcache.py", line 60, in <module>
    import six
ImportError: No module named six

----------------------------------------------------------------------
Ran 2 tests in 0.004s

FAILED (errors=2)
ERROR: InvocationError: '/Users/nick/Code/python-memcached/.tox/py27/bin/nosetests'

Catching _ConnectionDeadError in get_multi

  cache.get_multi(new_keys)
  File "site-packages/memcache.py", line 930, in get_multi
    line = server.readline()
  File "site-packages/memcache.py", line 1129, in readline
    raise _ConnectionDeadError()

It currently catches _Error and socket.error, but not _ConnectionDeadError.

I noticed this while doing a restart of a memcached machine.

Python3 exception when Client can't connect

In [1]: import memcache

In [2]: mc = memcache.Client(['somenonexistenthost'])

In [3]: mc.get('qwe')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-bb201bdf51b9> in <module>()
----> 1 mc.get('qwe')

/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in get(self, key)
   1087         @return: The value or None.
   1088         '''
-> 1089         return self._get('get', key)
   1090 
   1091     def gets(self, key):

/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in _get(self, cmd, key)
   1032         if self.do_check_key:
   1033             self.check_key(key)
-> 1034         server, key = self._get_server(key)
   1035         if not server:
   1036             return None

/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in _get_server(self, key)
    411                 # print("(using server %s)" % server,)
    412                 return server, key
--> 413             serverhash = serverHashFunction(str(serverhash) + str(i))
    414         return None, None
    415 

/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in cmemcache_hash(key)
     63 def cmemcache_hash(key):
     64     return (
---> 65         (((binascii.crc32(key) & 0xffffffff)
     66           >> 16) & 0x7fff) or 1)
     67 serverHashFunction = cmemcache_hash

TypeError: a bytes-like object is required, not 'str'

In [4]: mc = memcache.Client(['localhost'])

In [6]: mc.set('qwe', 'it works')
Out[6]: True

In [7]: mc.get('qwe')
Out[7]: 'it works'

No such file README.md in Version 1.50

$ pip install python-memcached==1.50
Downloading/unpacking python-memcached==1.50
  Running setup.py egg_info for package python-memcached
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/home/lakr/.venv/mtk_1.5/build/python-memcached/setup.py", line 9, in <module>
        long_description=open("README.md").read(),
    IOError: [Errno 2] No such file or directory: 'README.md'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/home/lakr/.venv/mtk_1.5/build/python-memcached/setup.py", line 9, in <module>

    long_description=open("README.md").read(),

IOError: [Errno 2] No such file or directory: 'README.md'

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/lakr/.venv/mtk_1.5/build/python-memcached
Storing complete log in /home/lakr/.pip/pip.log

Get this when installing via pypi. Surely related to 17fc525.

Failed operations should raise errors

Right now, failed operations in python-memcached either return 0 or None. For example, get returns None if it cannot connect to the server, but set returns 0.

It is more pythonic to raise exceptions when an unexpected condition has occurred.

See for example how redis-py raises a ConnectionError exception on invoking get when the server is down.

How do I check if the connection is closed?

Is it possible to programmatically check if the connection is really working after firing a memcache.cas request? Can I check the response code?

I could do a dumb:

memcache.set('Connection') == 'True'
if memcache.get('Connection') == 'True':
    continue

However is there a better way to do that?

Python 3 crash on storing binary strings

This test (adapted from Django's test suite) passes on Python 2, but fails on Python 3:

diff --git a/tests/test_memcache.py b/tests/test_memcache.py
index 321dc7b..0415de9 100644
--- a/tests/test_memcache.py
+++ b/tests/test_memcache.py
@@ -137,6 +137,30 @@ class TestMemcache(unittest.TestCase):
         value = self.mc.get(key)
         self.assertEqual(value, 5)

+    def test_binary_string(self):
+        # Binary strings should be cacheable
+        from zlib import compress, decompress
+        value = 'value_to_be_compressed'
+        compressed_value = compress(value.encode())
+
+        # Test set
+        self.mc.set('binary1', compressed_value)
+        compressed_result = self.mc.get('binary1')
+        self.assertEqual(compressed_value, compressed_result)
+        self.assertEqual(value, decompress(compressed_result).decode())
+
+        # Test add
+        self.mc.add('binary1-add', compressed_value)
+        compressed_result = self.mc.get('binary1-add')
+        self.assertEqual(compressed_value, compressed_result)
+        self.assertEqual(value, decompress(compressed_result).decode())
+
+        # Test set_many
+        self.mc.set_multi({'binary1-set_many': compressed_value})
+        compressed_result = self.mc.get('binary1-set_many')
+        self.assertEqual(compressed_value, compressed_result)
+        self.assertEqual(value, decompress(compressed_result).decode())
+
     def test_ignore_too_large_value(self):
         # NOTE: "MemCached: while expecting[...]" is normal...
         key = 'keyhere'

======================================================================
ERROR: test_binary_string (__main__.TestMemcache)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_memcache3.py", line 53, in test_binary_string
    compressed_result = self.mc.get('binary1')
  File "/home/tim/code/python-memcached/memcache.py", line 1093, in get
    return self._get('get', key)
  File "/home/tim/code/python-memcached/memcache.py", line 1077, in _get
    return _unsafe_get()
  File "/home/tim/code/python-memcached/memcache.py", line 1065, in _unsafe_get
    value = self._recv_value(server, flags, rlen)
  File "/home/tim/code/python-memcached/memcache.py", line 1235, in _recv_value
    val = buf.decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte

"'str' does not support the buffer interface" in serverHashFunction()

We found a problem in versions 1.56 and 1.57. The code below (memcache.py:399) sometimes (maybe always) store str into serverhash variable and cmemcache_hash/serverHashFunction function is failing because of that. The version 1.54 is ok.

def _get_server(self, key):
    if isinstance(key, tuple):
        serverhash, key = key
    else:
        serverhash = serverHashFunction(key)
...

Traceback (most recent call last):
  File "/srv/venv/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/srv/venv/lib/python3.4/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/srv/venv/lib/python3.4/site-packages/newrelic-2.54.0.41/newrelic/hooks/component_djangorestframework.py", line 8, in _nr_wrapper_APIView_dispatch_
    return wrapped(*args, **kwargs)
  File "/srv/venv/lib/python3.4/site-packages/rest_framework/views.py", line 456, in dispatch
    response = self.handle_exception(exc)
  File "/srv/venv/lib/python3.4/site-packages/rest_framework/views.py", line 453, in dispatch
    response = handler(request, *args, **kwargs)
  File "/srv/app/api/views/base.py", line 163, in post
    return self.data_valid(serializer.validated_data)
  File "/srv/app/api/views/get_suppliers_list.py", line 50, in data_valid
    suppliers = cache.get(key)
  File "/srv/venv/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 85, in get
    val = self._cache.get(key)
  File "/srv/venv/lib/python3.4/site-packages/newrelic-2.54.0.41/newrelic/api/datastore_trace.py", line 43, in _nr_datastore_trace_wrapper_
    return wrapped(*args, **kwargs)
  File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 1092, in get
    return self._get('get', key)
  File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 1037, in _get
    server, key = self._get_server(key)
  File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 416, in _get_server
    serverhash = serverHashFunction(str(serverhash) + str(i))
  File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 66, in cmemcache_hash
    (((binascii.crc32(key) & 0xffffffff)
TypeError: 'str' does not support the buffer interface

# note that the key used for incr/decr must be a string.

In the source code, I found the following comment:

    mc.set("key", "1") # note that the key used for incr/decr must be
                       # a string.
    mc.incr("key")
    mc.decr("key")

However, I tested incr using both integer and string. Both seem to work:

In [14]: cache.add('foo', 1)
Out[14]: True

In [15]: cache.incr('foo')
Out[15]: 2

In [16]: cache.get('foo')
Out[16]: 2

In [17]: cache.add('bar', '1')
Out[17]: True

In [18]: cache.incr('bar')
Out[18]: 2

In [19]: cache.get('bar')
Out[19]: '2'

So is the comment still relevant?

get fails with UnboundLocalError: local variable 'val' referenced before assignment

I receive the above exception when issuing a get request for a key "groovy-1" inserted by a Java client.

The key in question (inserted by a Java client) that causes a problem.

echo "get key groovy-1" | nc 127.0.0.1 11211
VALUE groovy-1 32 13
1377311483802
END

Does a flag value of "32" affect this python client's ability to read the value?

Thank you for looking into the matter.

The code that causes the Exception is as follows:

#!/usr/bin/env python

import memcache

mc = memcache.Client( [ '127.0.0.1:11211'] )

# access a non-existent key in memcache, should throw
# UnboundLocalError: local variable 'val' referenced before assignment
# At least that exception is thrown for me using python_memcached-1.48-py2.7.egg
#    installed using easy_install on Aug 23rd 2013.

print "groovy-1 = %s" % mc.get( "groovy-1" )

MemcachedKeyCharacterError should confess the bad key

We got a bunch of MemcachedKeyCharacterError exceptions from inside our django app. We've been unable to reproduce the problem. Debugging these would be a lot easier if the exception included the offending key. Something like the following should do the trick:

--- /home/roy/deploy/current/python/lib/python2.7/site-packages/memcache.py 2013-10-10 16:43:47.484586967 -0400
+++ /tmp/memcache.py    2013-11-02 11:41:36.316624255 -0400
@@ -1059,7 +1059,7 @@
                          % self.server_max_key_length)
             if not valid_key_chars_re.match(key):
                 raise Client.MemcachedKeyCharacterError(
-                        "Control characters not allowed")
+                        "Control characters not allowed (key = %r)" % key)

 class _Host(object):

Sporadic KeyError when using get_multi

# some web code here
# keys = [key, key+'_queued']
 File "./services/cache.py", line 33, in mget
    results = self.get_multi(keys)
  File "/var/www/heydayserver/env/local/lib/python2.7/site-packages/memcache.py", line 972, in get_multi
    retvals[prefixed_to_orig_key[rkey]] = val   # un-prefix returned key.
KeyError: '-34.9832:146.2685_queued'

I am not sure why this is happening though, all I use the memcache for is to store json strings with keys. It happens sporadically, around redeploy of a Flask server.

Can't get, or set 'NFL::CAR_TB'

After checking a whole loada things, I've found that for some keys, python-memcached just won't get, or set them on my machine (homebrew'd 1.4.15, pip installed python-memcached 1.53 on Mac OSX 10.9). My first issue was a shot in the dark, not having a clear idea as to what was going on, but after more digging I now definitely now.

It all hinges around
def _get_server(self, key):

If we add two debug printout lines,

def _get_server(self, key):
    if isinstance(key, tuple):
        serverhash, key = key
    else:
        serverhash = serverHashFunction(key)

    for i in range(Client._SERVER_RETRIES):
        server = self.buckets[serverhash % len(self.buckets)]
        if server.connect():
            #print "(using server %s)" % server,
            print 'got server {} for {}'.format(serverhash % len(self.buckets),
                                                key)
            return server, key
        print 'server {} failed for {}'.format(serverhash % len(self.buckets),
                                            key)
        serverhash = serverHashFunction(str(serverhash) + str(i))
    return None, None

Trying to get or set the key 'NFL::CAR_TB',

import memcache
mc_cl = memcache.Client('127.0.0.1')
mc_cl.set('NFL::CAR_TB', 1)
mc_cl.get('NFL::CAR_TB')
mc_cl.set('NFL::CAR_UB', 1)
mc_cl.get('NFL::CAR_UB')

results, on my machine with,

server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 3 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 3 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
got server 6 for NFL::CAR_UB
got server 6 for NFL::CAR_UB

If I alter _get_server to

def _get_server(self, key):
    choices = range(len(self.buckets) - 1)
    random.shuffle(choices)

    if isinstance(key, tuple):
        choice, key = key
    else:
        choice = choices.pop()

    for _ in range(Client._SERVER_RETRIES):
        server = self.buckets[choice]
        if server.connect():
            #print "(using server %s)" % server,
            print 'got server {} for {}'.format(choice,
                                                key)
            return server, key
        print 'server {} ({}) failed for {}'.format(choice, server, key)
        choice = choices.pop()
    return None, None

Then everything works much better,

got server 4 for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 2 (inet:7:11211 (dead until 1384820883)) failed for NFL::CAR_TB
got server 4 for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_UB
server 3 (inet:.:11211 (dead until 1384820883)) failed for NFL::CAR_UB
got server 4 for NFL::CAR_UB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_UB
got server 6 for NFL::CAR_UB

~~
multi_get and multi_set not getting / setting all keys

I was getting very inconsistent results when trying to set multiple keys in my python program,

import memcache

TWO_HOURS = 2 * 60 * 60
mc_cl = memcache.Client('127.0.0.1')
mapping = {...}

mc_cl.flush_all()
ret = mc_cl.set_multi(mapping=mapping, time=TWO_HOURS)
getret = mc_cl.get_multi(mapping.keys())
if len(mapping) != len(getret):
    print 'not set\n\t{}'.format('\n\t'.join([str((k, mapping[k])) for k in
                                              [a for a in mapping.keys()
                                               if a not in getret.keys()]]))

And after analysing the raw memcache output, it seems like not all the keys are being set, and not all the keys are being requested thereafter. All things done and up to date on homebrew and pip, on Mac OSX 10.9.

Memcache output below and reading it states that only 101 keys were attempted to be written and read, whereas there were 228 items.

import re

gamesSet = []
gamesGet = []
with open('memout.log') as f:
    for line in f.read().split('\n'):
        match = re.match('^<21 set ([^ ]+) .*$', line)
        if match is not None:
            gamesSet.append(match.group(1))
            continue
        match = re.match('^<21 get (.*)$', line)
        if match is not None:
            gamesGet += match.group(1).split(' ')
            continue
print len(mapping), mapping
print len(gamesSet), gamesSet
print len(gamesGet), gamesGet

That log is below.

<21 new auto-negotiating client connection
21: Client using the ascii protocol
<21 flush_all
>21 OK
<22 new auto-negotiating client connection
22: Client using the ascii protocol
<22 flush_all
>22 OK
<21 set NCAAF::BUFF_KENTST 1 7200 34
>21 STORED
<21 set NCAAF::ND_USC 1 7200 29
>21 STORED

Performance drop between v1.53 and v1.54

We've been using python-memcached v1.48 with Python 2.7.8 and Django 1.7.7.

I've just tried deploying python-memcached v1.54 and it increased average Memcached get response times from ~4.5ms to ~16ms, and sets from ~0.3ms to ~0.4ms (as reported by New Relic).

Before I try bisecting further, is this expected?

(If it helps, our version of six is 1.9.0).

set_multi() raises AttributeError if connection is closed between readline() calls

We ran into an issue on our live memcached servers where set_multi() was raising an error like the following (this is test code, but it was basically the same traceback):

Traceback (most recent call last):
  File "C:\dev\main\python\test\setmultibug.py", line 41, in <module>
    main()
  File "C:\dev\main\python\test\setmultibug.py", line 37, in main
    bad_keys = client.set_multi(mapping)
  File "C:\dev\main\tools\python\modules\memcache.py", line 735, in set_multi
    line = server.readline()
  File "C:\dev\main\tools\python\modules\memcache.py", line 1163, in readline
    recv = self.socket.recv
AttributeError: 'NoneType' object has no attribute 'recv'

I'm not sure the root cause of the connection issue in our case, but it shouldn't cause an AttributeError -- it should mark the server dead and move along. What I believe is happening is the connection was being forcibly closed between readline() calls when reading the per-key responses in set_multi().

readline() actually handles this case with its if not data: ... clause, but it does it by calling self.close_socket() directly, which sets self.socket = None.

Here's a test program that stubs out socket.socket to pretend it closed the connection between recv() calls to repro the issue:

"""Repro memcache set_multi() AttributeError:
AttributeError: 'NoneType' object has no attribute 'recv'
"""

import socket

import memcache

RECV_CHUNKS = ['chunk1']

class FakeSocket(object):
    def __init__(self, *args):
        print 'FakeSocket{0!r}'.format(args)
        self._recv_chunks = list(RECV_CHUNKS)

    def connect(self, *args):
        print 'FakeSocket.connect{0!r}'.format(args)

    def sendall(self, *args):
        print 'FakeSocket.sendall{0!r}'.format(args)

    def recv(self, *args):
        if self._recv_chunks:
            data = self._recv_chunks.pop(0)
        else:
            data = ''
        print 'FakeSocket.recv{0!r} -> {1!r}'.format(args, data)
        return data

    def close(self):
        print 'FakeSocket.close()'

def main():
    socket.socket = FakeSocket
    client = memcache.Client(['memcached'], debug=True)
    mapping = {'foo': 'FOO', 'bar': 'BAR'}
    bad_keys = client.set_multi(mapping)
    print 'set_multi({0!r}) -> {1!r}'.format(mapping, bad_keys)

if __name__ == '__main__':
    main()

python-memcached should fail gracefully when the server does not exist.

I recently ran into a situation where our build server did not have a running instance of memcached. I would have expected things to fail gracefully. (Clearly, there wouldn't be any caching happening. Each request would simply have to re-fetch any cached data.) But instead of failing gracefully, the caching code failed after the connect attempt to the memcached server failed.

It looks like once the server is marked dead there are some additional calls to cmemcache_hash that get made with strings as keys. To reproduce this, find a server without a memcached instance and try this:

python
Python 3.5.1 (default, Apr  6 2016, 16:48:42) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> cache = memcache.Client(['127.0.0.1:11211'], debug=1)
>>> cache.set("working?",1,1)
MemCached: MemCache: inet:127.0.0.1:11211: connect: [Errno 111] Connection refused.  Marking dead.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 700, in set
    return self._set("set", key, val, time, min_compress_len, noreply)
  File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 983, in _set
    server, key = self._get_server(key)
  File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 413, in _get_server
    serverhash = serverHashFunction(str(serverhash) + str(i))
  File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 65, in cmemcache_hash
    (((binascii.crc32(key) & 0xffffffff)
**TypeError: a bytes-like object is required, not 'str'**

So, I temporarily "fixed" the issue with something like this:

--- /<snipped>/lib/python3.5/site-packages/memcache.py  2016-04-07 17:23:41.428028602 -0500
+++ /<snipped>/lib/python3.5/site-packages/memcache.py_fixed    2016-04-07 17:23:13.606003863 -0500
@@ -61,6 +61,9 @@


 def cmemcache_hash(key):
+    if isinstance(key, six.text_type):
+        key = key.encode('utf-8')
+
     return (
         (((binascii.crc32(key) & 0xffffffff)
           >> 16) & 0x7fff) or 1)

I'm not sure if this is the right approach to correct the issue or not. Hopefully we can get back to a point where python-memcached can fail gracefully when the underlying service is absent.

Thanks,

Matthew Lee

installing with easy-install or pip

Hi,
I am getting errors while trying to load python-memcached. With easy-install

Best match: python-memcached 1.53
Downloading https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.53.tar.gz#md5=89570d26e7e7b15caa668a6b2678bd3c
Processing python-memcached-1.53.tar.gz
Writing /var/folders/_b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/setup.cfg
Running python-memcached-1.53/setup.py -q bdist_egg --dist-dir /var/folders/b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/egg-dist-tmp-ibs24u
Traceback (most recent call last):
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/bin/easy_install", line 9, in
load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1992, in main
with_ei_usage(lambda:
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1979, in with_ei_usage
return f()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1996, in
distclass=DistributionWithoutHelpCommands, **kw
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 929, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 948, in run_command
cmd_obj.run()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 380, in run
self.easy_install(spec, not self.no_deps)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 653, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
run_setup(setup_script, args)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 69, in run_setup
lambda: execfile(
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 120, in run
return func()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 71, in
{'file':setup_script, 'name':'main'}
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/compat.py", line 92, in execfile
exec(compile(source, fn, 'exec'), globs, locs)
File "setup.py", line 4, in
File "/var/folders/_b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/memcache.py", line 410
except socket.error, msg:
^
SyntaxError: invalid syntax

Support for usernames/passwords?

Hi,

I'm trying to use memcache on Heroku with "Memcached Cloud". The connection credentials I'm given include a username and a password, however it doesn't seem like there's anywhere to provide that information in the library. Is there any plan to support usernames/passwords in the library, or is that beyond the scope of this library?

Thanks!

let's please revisit CAS support

I realize this will be controversial given how long the Python memcache API has been around. Relative to the memcached protocol, Python's API doesn't support CAS beyond very trivial use cases.

The Python API forces implementations to manage CAS ID's implicitly. This results in several problems:

CAS ID lifetime management

Since an implementation doesn't know if a CAS ID will subsequently be needed, it must hold on to all ID's indefinitely. Granted, this is not much of a problem for applications with a fixed number of items involved in CAS. However there are use cases which can yield an unbounded number of such items, for example when implementing concurrent collections on memcache using an item per node. (I have https://github.com/google/memcache-collections which implements a deque this way.)

While the API does provide reset_cas(), it is of limited use given that the retained cache ID's are global state. This makes it very hard to combine different libraries employing CAS into one application since they will inevitably have different CAS ID lifetime requirements. Furthermore, the Python memcache implementation makes bugs related to CAS ID lifetime exceedingly hard to detect since it silently succeeds cas() calls for which a CAS ID couldn't be found.

aside: The current API limits CAS use to single reads via gets()-- it isn't possible to use get_multi() with CAS. For this, App Engine extended get_multi() with a for_cas parameter.

Inability to share CAS ID's

A very useful building block for distributed data structures is the MCAS operation, i.e. atomic CAS across multiple, independent addresses. It's possible in general to implement MCAS on top of the single CAS primitive as described in "Practical lock freedom", Keir Fraser, 2004 (http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf). This would be interesting to implement on top of memcache, as it would allow atomic mutation of multiple items, even when they exist on different memcached servers in a cluster. However the lock free algorithm requires other processes to "help" when they find transactions in an intermediate state. To implement this on memcache, the CAS ID's would need to be stored, shared, and used among clients (which may be on different threads, processes, or machines). The memcached protocol supports this, but the Python API does not.

I don't have a proposal yet, but I'd like to investigate extending the Python memcache API to support explicit management of CAS ID's.

Alive-Check

Hi,

how can I figure out if a host is alive or not without using expensive timestamp set/get checks?

set_multi wants 'STORED' even for values it refuses to store

When set_multi decides that a value is too big to be stored, it still wants a 'STORED' from the server... the server is finally marked as dead.

Here is how to reproduce:

import memcache
c = memcache.Client(('localhost:11211', ))
assert not any(server.deaduntil for server in c.servers), 'servers are marked dead before set_multi!'
assert 'a' in c.set_multi({'a': 'a' * 1024 * 1025}), 'we were able to write more than 1Mb of data!'
assert not any(server.deaduntil for server in c.servers), 'servers are marked dead after set_multi!'

raises an AssertionError: servers are marked dead after set_multi! but it shouldn't

Package is unmaintained - recommend using pylibmc instead

In case anyone about to start using python-memcached sees this.

This package is not actively maintained at the moment. For example with the latest release there is a 400% performance slowdown under Python 2.7 (#71) that would be fixed by the unmerged #86.

As such for now I'd strongly recommend using pylibmc instead, which in most cases is a drop-in replacement:
https://github.com/lericson/pylibmc
http://sendapatch.se/projects/pylibmc/misc.html#differences-from-python-memcached

Python 3 TypeError: 'str' does not support the buffer interface - memcache.py line 65

If memcached isn't running Python3 gets the following error on set(): TypeError: 'str' does not support the buffer interface. Should this be handled better?

We're hitting this in python-keystoneclient test code that checks if memcached is available. See: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/tests/unit/test_auth_token_middleware.py#L78

Here's the difference between python 2 and 3 behavior:

ubuntu@w1:$ sudo service memcached stop
ubuntu@w1:$ python
Python 2.7.10 (default, Jun 1 2015, 16:21:46)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import memcache
c = memcache.Client(['localhost:11211'])
c.set('ping', 'pong', time=1)
0
quit()
ubuntu@w1:~$ python3
Python 3.4.3+ (default, Jun 2 2015, 14:09:35)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
import memcache
c = memcache.Client(['localhost:11211'])
c.set('ping', 'pong', time=1)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3/dist-packages/memcache.py", line 700, in set
return self._set("set", key, val, time, min_compress_len, noreply)
File "/usr/lib/python3/dist-packages/memcache.py", line 983, in _set
server, key = self._get_server(key)
File "/usr/lib/python3/dist-packages/memcache.py", line 413, in _get_server
serverhash = serverHashFunction(str(serverhash) + str(i))
File "/usr/lib/python3/dist-packages/memcache.py", line 65, in cmemcache_hash
(((binascii.crc32(key) & 0xffffffff)
TypeError: 'str' does not support the buffer interface
quit()

TypeError: 'str' does not support the buffer interface under Python 3

I get this error under Python 3.4 using 2251916:

Error handling request
Traceback (most recent call last):
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = middleware_method(request, response)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/core/middleware.py", line 192, in process_response
    context = RequestContext(request)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/template/context.py", line 169, in __init__
    self.update(processor(request))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/conf/context_processors.py", line 40, in settings
    settings_dict = cache_get(cache_key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/utils/cache.py", line 48, in cache_get
    packed = cache.get(_hashed_key(key))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 75, in get
    val = self._cache.get(key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1002, in get
    return self._get('get', key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 986, in _get
    return _unsafe_get()
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 957, in _unsafe_get
    server.send_cmd("%s %s" % (cmd, key))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1299, in send_cmd
    self.socket.sendall(cmd + '\r\n')
TypeError: 'str' does not support the buffer interface

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 93, in handle
    self.handle_request(listener, req, client, addr)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 134, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
    response = self.get_response(request)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 203, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 236, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/utils/decorators.py", line 99, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/core/views.py", line 220, in server_error
    context = RequestContext(request, {"STATIC_URL": settings.STATIC_URL})
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/template/context.py", line 169, in __init__
    self.update(processor(request))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/conf/context_processors.py", line 40, in settings
    settings_dict = cache_get(cache_key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/utils/cache.py", line 48, in cache_get
    packed = cache.get(_hashed_key(key))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 75, in get
    val = self._cache.get(key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1002, in get
    return self._get('get', key)
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 986, in _get
    return _unsafe_get()
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 957, in _unsafe_get
    server.send_cmd("%s %s" % (cmd, key))
  File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1299, in send_cmd
    self.socket.sendall(cmd + '\r\n')
TypeError: 'str' does not support the buffer interface

Python 3 support

Hi!
I ran code via 2to3 tool and it seems quite simple to port it. Maybe you have some thoughts about it, what Python versions might be supported?

git tags are missing

It would be useful to have git tags corresponding to pypi release tarballs:
https://pypi.python.org/packages/source/p/python-memcached/

Speed up gets/sets by 30% by simplifying check_key

I don't know if anyone's profiled where memcache.py is spending most of its time, but in my recent benchmarks, cProfile tells me it was spending far and away most of its time in check_key(). This is quite sad, as once you've tested your code, you're not going to be sending through bad keys, so these expensive checks are basically worthless in production.

The most expensive check is looping through all the chars in the key, calling ord(c), and ensuring each char is not a control character. This is the most expensive check, and also the least likely to actually happen in practice (in my experience it's very rare to build a key containing control chars).

I'm using a subclass of memcache.Client that overrides check_key() and speeds up my multi_get/multi_set benchmark by literally 30%. I still perform the most basic sanity checks on the key:

class Client(memcache.Client):
    def check_key(self, key, key_extra_len=0):
        assert key, "Can't use None or empty string as key"
        assert isinstance(key, str), 'Keys must be str instances'

Even eliminating the for char in key: loop gets you 80% of this speed improvement.

What about an option to the Client() constructor check_keys that bypasses what's now check_key(), or just performs the basic checks I show above? It could be check_keys=True by default to emulate current behaviour.

Status of 1.54?

From the CHANGELOG and commit messages, I gather that a 1.54 release is imminent.

People may be interested in it because of the presumed ability to set SERVER_MAX_VALUE_LENGTH at import time.

local variable 'val' referenced before assignment

From time to time when getting some data from memcache i get this error:

ERR = local variable 'val' referenced before assignment

Regression storing non-ASCII values on Python 2

Bisected to 04f1c78

Failing test case (adapted from Django's test suite):

diff --git a/tests/test_memcache.py b/tests/test_memcache.py
index 321dc7b..03147d6 100644
--- a/tests/test_memcache.py
+++ b/tests/test_memcache.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 from __future__ import print_function

 import unittest
@@ -137,6 +138,14 @@ class TestMemcache(unittest.TestCase):
         value = self.mc.get(key)
         self.assertEqual(value, 5)

+    def test_unicode_value(self):
+        key = 'key'
+        value = six.u('Iñtërnâtiônàlizætiøn2')
+
+        self.mc.set(key, value)
+        cached_value = self.mc.get(key)
+        self.assertEqual(value, cached_value)
+
     def test_ignore_too_large_value(self):
         # NOTE: "MemCached: while expecting[...]" is normal...
         key = 'keyhere'

======================================================================
FAIL: test_unicode_value (__main__.TestMemcache)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_memcache.py", line 147, in test_unicode_value
    self.assertEqual(value, cached_value)
AssertionError: u'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti\xc3\xb4n\xc3\xa0liz\xc3\xa6ti\xc3\xb8n2' != 'I\xc3\x83\xc2\xb1t\xc3\x83\xc2\xabrn\xc3\x83\xc2\xa2ti\xc3\x83\xc2\xb4n\xc3\x83\xc2\xa0liz\xc3\x83\xc2\xa6ti\xc3\x83\xc2\xb8n2'

String 'True' treated as Int

Context: memcached + python-memcached + django-memcached

Error:
/usr/local/lib/python2.7/site-packages/memcache.py in _recv_value
1235 val = int(buf)

Local vars:

buf 'True'
self    <memcache.Client object at 0x7f926d7590b8>
rlen    6
flags   2
server  <memcache._Host object at 0x7f926d826190>

This error does not occur in v <= 1.54

1.54 release?

Looks like a lot of work has been done since 1.53 — what's the status on a new release?

Is it possible to reformat Changelog according to the best practices proposed by keepachangelog.com?

It will be easier to track what happened between two versions if information will be grouped by version.

I could reformat the Changelog file and make a pull-request if you wish.

New release with python 3 support

Hi,

The latest release is now over a year old and doesn't include any of the python 3 compatibility. Can you release a new version please?

Setting/Getting boolean values yields a ValueError

In Python Memcached 1.56 (w/ six-1.9.0 & Python 2.7.10)

doing

cache.set('foo', False)
cache.get('foo')

generates a ValueError: invalid literal for int() with base 10: 'False'

line 1235 memcache.py

When the method _val_to_store_info is called the path code that it is followed with a boolean is

   elif isinstance(val, int):
        flags |= Client._FLAG_INTEGER
        val = str(val)
        if six.PY3:
            val = val.encode('ascii')
        # force no attempt to compress this silly string.
        min_compress_len = 0

Since the flags bits have been set to an integer, the boolean value is changed to a string ('False'). So in the method _recv_value the code tries to change that string to an int which fails .

In Python Memcached 1.54

the original code in _val_to_store_info is

elif isinstance(val, int):
flags |= Client._FLAG_INTEGER
val = "%d" % val
# force no attempt to compress this silly string.
min_compress_len = 0

the issue also affect when checking for long values

Incorrect "Key is None" during multi_delete()

The code below shows this behaviour:

import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)

keys = ["", "userA", "userB"]
prefix = "settings_"
mc.delete_multi(keys, key_prefix=prefix)

In my case I use empty string to signify the default for a collection of keys. Memcache will store it fine but python-memcache incorrectly reports I'm using a key of None. I think this is the cause:
In _map_and_prefix_keys() we check validity of the key before adding the prefix: https://github.com/linsomniac/python-memcached/blob/master/memcache.py#L725

Release management confusion

Hi there, I am a bit confused about the current latest release on PyPI (1.31) whereas I used to package 1.47 (https://pypi.python.org/pypi/python-memcached/1.47) for openSUSE. However, it seems like the contents of 1.31 is newer than 1.47. The issue here is that distributions already packaged and ship 1.47 (or 1.48) and downgrading would be a huge pain (involving RPM epoch). Also, when comparing both versions, 1.467has more than 100000 downloads compared to ~500 of 1.37. So I suspect we're not the only ones hitting this issue.

So first of all thank you for taking over maintainership of this module. But in order to solve the issue, could you please create a release with a more appropriate version number? Maybe a 2.0 is appropriate even ;-)