jamesls / fakeredis Goto Github PK
View Code? Open in Web Editor NEWFake implementation of redis API (redis-py) for testing purposes
Fake implementation of redis API (redis-py) for testing purposes
When commands allow a variable number of arguments, would it be possible to implement something similar to what redis-py uses, where it allows you to pass multiple keys to a command either via a list, or as an *args list.
See: https://github.com/andymccurdy/redis-py/blob/master/redis/client.py#L23
Examples:
hmget('key', 'hashkey1', 'hashkey1')
hmget('key', ['hashkey1', 'hashkey1'])
Comparison:
https://github.com/andymccurdy/redis-py/blob/master/redis/client.py#L1409
https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L620
I use the variable length lists at the moment, but I'd have to convert them in order to seamlessly use fakeredis. I'm happy to do so, but I thought I'd put in a feature request anyway!
Thanks for the great package!
fakeredis.FakeStrictRedis
doesn't implement redis-py's .lock() method. Are you open to a pull request that uses the standard lib's threading.Lock
to fake the same behavior?
(Of course, I'll follow the contribution guidelines :)
Maybe flushall()
should be mentioned in the readme, as it's quite essential for using FakeRedis in unit tests.
Example:
def setUp(self):
# setup fake redis for testing
self.r = fakeredis.FakeStrictRedis()
def tearDown(self):
# clear data in fake redis
self.r.flushall()
When using StrictRedis as a base class, and when using a pipeline to commit commands, in the subclassed pipeline uses the original StrictRedis implementation of the command, not the subclassed command.
Here is an implementation of a pop
operation using both a subclassed StrictRedis
and FakeStrictRedis
.
class PrintRedis(redis.StrictRedis):
def get(self, key):
printme = super(PrintRedis, self).get(key)
print 'PrintRedis: {}'.format(printme)
return printme
def pop(self, key):
pipe = self.pipeline()
printme, _ = pipe.get(key).delete(key).execute()
print 'PrintRedis: {}'.format(printme)
return printme
class FakePrintRedis(fakeredis.FakeStrictRedis):
def get(self, key):
printme = super(FakePrintRedis, self).get(key)
print 'FakePrintRedis: {}'.format(printme)
return printme
def pop(self, key):
pipe = self.pipeline()
printme, _ = pipe.get(key).delete(key).execute()
print 'FakePrintRedis: {}'.format(printme)
return printme
Ideally, these both would operate the same, since fakeredis is a fake. However, that's not the case--
Print Redis Test
>>> p = PrintRedis()
>>> p.set('printredis', 'testdata')
True
>>> p.get('printredis')
PrintRedis get: testdata
'testdata'
>>> p.pop('printredis')
PrintRedis pop: testdata
'testdata'
>>> p.get('printredis')
PrintRedis get: None
Fake Print Redis Test
>>> f = FakePrintRedis()
>>> f.set('fakeprintredis', 'testdata')
True
>>> f.get('fakeprintredis')
FakePrintRedis get: testdata
'testdata'
>>> f.pop('fakeprintredis')
FakePrintRedis get: testdata
FakePrintRedis pop: testdata
'testdata'
>>> f.get('fakeprintredis')
FakePrintRedis get: None
As you can see, the fake print redis prints to stdout twice-- the .get
call in the pipeline is being called from the subclassed FakePrintRedis version of the function.
Incidentally, my use case for this is in testing--
class TestRedisPrint(unittest.TestCase):
def test_redis_pop(self):
patcher = patch.object(PrintRedis, '__bases__',
(fakeredis.FakeStrictRedis,))
with patcher:
patcher.is_local = True
redis_set('pop_testkey', 'pop_testvalue')
with self.assertPrinted("PrintRedis pop: pop_testvalue"):
testvalue = redis_pop('pop_testkey')
Which is a failing test.
On the other hand, I kind of prefer the pipeline suing the subclassed function... but since this is the fake, we should make it match the real one.
Hi,
Great library, thanks.
I noticed that smembers returns None instead of an empty list sometimes which is different to what redis does. I haven't looked at other similar functions.
Cheers,
Hadley
Code example:
redis = FakeStrictRedis()
redis.set("key", "value", ex=1)
redis.delete("key")
redis.set("key", "value")
sleep(2)
print redis.get("key")
Should return "value", but return None.
Version 0.4.2
According to the redis documentation:
If key already exists and is a string, this command appends the value at the end of the string. If key does not exist it is created and set as an empty string, so APPEND will be similar to SET in this special case.
If a non-existent key is appended to fakeredis it returns a KeyError.
Here's what happens when I try to use 'get' on a key that I've used to store a list:
>>> from redis_cache import get_redis_connection
>>> client = get_redis_connection('persistent')
>>> client.rpush('foo', 1)
1L
>>> client.get('foo')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 551, in get
return self.execute_command('GET', name)
File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 361, in execute_command
return self.parse_response(connection, command_name, **options)
File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 371, in parse_response
response = connection.read_response()
File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/connection.py", line 311, in read_response
raise response
ResponseError: WRONGTYPE Operation against a key holding the wrong kind of value
Using fakeredis, I don't get any error:
>>> import fakeredis
>>> client = fakeredis.FakeStrictRedis()
>>> client.rpush('foo', 1)
1
>>> client.get('foo')
"['1']"
I think the fix to this might involve tracking the type of item at each key, which could get involved. Is this of interest to fakeredis?
I think the problem here is that the ranges are treated categorically as numbers instead of preprocessing (see: https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L888) which causes a string / int comparison in the case of using exclusive bound notation.
https://pypi.python.org/pypi/lupa is a bridge from python to lua.
It can be used to provide the same environment as redis provides so that lua scripts will be able to be tested without using a real redis instance.
What do you think?
Hi,
It seems that some arguments (time and value) order are inverted for the setex function.
I got an error when using fakeredis version 0.7.0
Using redis-py:
import redis
# ...
r.setex('mykey', 'myvalue', 10)
Using fakeredis version 0.7.0
import fakeredis
r = fakeredis.FakeStrictRedis()
r.setex('mykey', 10, 'myvalue')
Can actually be overcome by naming the args:
import redis
r.setex(name='mykey', value='myvalue', time=10)
But it would be cool if it can have the same order for the args.
What do you think?
Regards.
When using a tuple as a key, redis will use the string form of the tuple. Fakeredis returns the key as the original tuple.
Example:
import redis/fakeredis
import time
client = redis.StrictRedis()
client.zadd('test_set', time.time(), (1, 2, 3))
result = client.zrange('test_set', 0, 0)
result returns tuple in array [(1, 2, 3)] in fakeredis, but string in array ['(1, 2, 3)'] in Redis
Redis returns the results of mget as a list of strings. In fakeredis, if you use incr and then mget, you get a list of integers, not strings.
Here's the bug in action (note the last item is a list containing an integer):
>>> import fakeredis
>>> cache = fakeredis.FakeStrictRedis()
>>> cache.incr('a', 1)
1
>>> cache.get('a')
'1'
>>> cache.incr('a', 1)
2
>>> cache.get('a')
'2'
>>> cache.mget(['a'])
[2]
Here's how it works against my redis instance:
>>> from django.core.cache import get_cache
>>>
>>> redis_cache = get_cache('persistent').raw_client
>>> redis_cache.incr('a',1)
1
>>> redis_cache.get('a')
'1'
>>> redis_cache.incr('a',1)
2
>>> redis_cache.get('a')
'2'
>>> redis_cache.mget(['a'])
['2']
It looks like incr always transforms the item into an integer, but get works around it by calling to_bytes on the value before returning it. Perhaps the solution is to do the same on each item in mget.
redis-py always returns hash values as bytes, even when they are valid floating point numbers. fakeredis is consistent with that when you set the value with HSET:
In [1]: import redis, fakeredis
In [2]: real = redis.StrictRedis()
In [3]: real.hset('h', 'x', 1.5)
Out[3]: 1
In [4]: real.hgetall('h')
Out[4]: {b'x': b'1.5'}
In [5]: fake = fakeredis.FakeStrictRedis()
In [6]: fake.hset('h', 'x', 1.5)
Out[6]: 1
In [7]: fake.hgetall('h')
Out[7]: {b'x': b'1.5'}
However, it breaks when you use HINCRBYFLOAT:
# Standard behavior
In [8]: real.hincrbyfloat('h2', 'x', 1.5)
Out[8]: 1.5
In [9]: real.hgetall('h2')
Out[9]: {b'x': b'1.5'}
# Incompatible behavior
In [10]: fake.hincrbyfloat('h2', 'x', 1.5)
Out[10]: 1.5
In [11]: fake.hgetall('h2')
Out[11]: {b'x': 1.5}
In [12]: fake.hget('h2', 'x')
Out[12]: 1.5
HINCRBY has the same problem, it changes the type to int:
In [13]: real.hincrby('h3', 'x', 15)
Out[13]: 15
In [14]: real.hgetall('h3')
Out[14]: {b'x': b'15'}
In [15]: fake.hincrby('h3', 'x', 15)
Out[15]: 15
In [16]: fake.hgetall('h3')
Out[16]: {b'x': 15}
In [17]: fake.hget('h3', 'x')
Out[17]: 15
I think it would be useful to mock the errors thrown when you are unable to connect to the redis server. It appears that redis-py throws a ConnectionError
with these args ('Error 61 connecting to localhost:6379. Connection refused.',)
when it cannot reach the server.
Example test that fails:
def test_sort_with_set(self):
self.redis.sadd('foo', '3')
self.redis.sadd('foo', '1')
self.redis.sadd('foo', '2')
self.assertEqual(self.redis.sort('foo'), ['1', '2', '3'])
This appears simple to add. If I created a pull request, would you be interested in merging it in?
feeds/cache.py:128: in store_feed_content
self.redis_client.set(feed_key, raw_content)
.tox/py27/lib/python2.7/site-packages/fakeredis.py:386: in set
self._db[name] = to_bytes(value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
x = '<feedán/>', charset = 'ascii', errors = 'strict'
def to_bytes(x, charset=sys.getdefaultencoding(), errors='strict'):
if x is None:
return None
if isinstance(x, (bytes, bytearray, buffer)) or hasattr(x, '__str__'):
> return bytes(x)
E UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 5: ordinal not in range(128)
There is an issue with how keys are stored in _StrKeyDict#_ex_key
and _StrKeyDict#_dict
. Consider the following case
import fakeredis
from time import sleep
r = fakeredis.FakeStrictRedis()
class Key(object):
def __init__(self, k):
self.k = k
def __str__(self):
return str(self.k)
r.set(Key("a"), "value", 1)
r.set(Key("a"), "value", 10)
sleep(2)
print r.get(Key("a")) # None instead of value
Here is redis-py's transaction()
As you can see, it takes keyword arguments: def transaction(self, func, *watches, **kwargs)
fakeredis' transaction()
does not expect keyword arguments: def transaction(self, func, *keys)
Naturally this results in a TypeError
when trying to use keyword arguments.
The three current kwargs are shard_hint=None
, value_from_callable=False
, and watch_delay=None
. The fakeredis implementation, however, does not support any of these.
I think I've discovered a discrepancy between redis's watch statement implementation and fakeredis's mocking of that implementation.
Code that runs fine on actual redis throws a watch error every time when using fakeredis.
I reduced the code down into the minimal example, below.
key = 'testkey'
fake_redis = fakeredis.FakeStrictRedis()
redis = RedisProxy() # however you connect to your redis
with fake_redis.pipeline(transaction=True) as pl:
pl.watch(key)
pl.hset(key, 'blah1', "blah2")
pl.execute()
if you use redis.pipeline, this code works fine, if you use fake_redis.pipeline, you get the following
In [8]: pl.execute()
---------------------------------------------------------------------------
WatchError Traceback (most recent call last)
<ipython-input-8-622a4d39c733> in <module>()
----> 1 pl.execute()
/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/fakeredis-0.5.1-py2.7.egg/fakeredis.pyc in execute(self)
1247 'Watched key%s %s changed' % (
1248 '' if len(mismatches) == 1 else
-> 1249 's', ', '.join(k for (k, _, _) in mismatches)))
1250 ret = [getattr(self.owner, name)(*args, **kwargs)
1251 for name, args, kwargs in self.commands]
WatchError: Watched key testkey changed
A watch shouldn't throw an error if it's modified by the same connection that set the watch - it should only error if it's modified by another connection, to prevent race conditions.
Hi, it may just be my machine but the current release on PyPi doesn't install correctly:
$ pip install fakeredis
Downloading/unpacking fakeredis
Running setup.py egg_info for package fakeredis
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/Users/foo/dev/something/venv/build/fakeredis/setup.py", line 11, in <module>
'README.rst')).read(),
IOError: [Errno 2] No such file or directory: '/Users/foo/something/venv/build/fakeredis/README.rst'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/Users/foo/dev/something/venv/build/fakeredis/setup.py", line 11, in <module>
'README.rst')).read(),
However the current master does:
$ pip install git+git://github.com/jamesls/fakeredis.git
Downloading/unpacking git+git://github.com/jamesls/fakeredis.git
Cloning git://github.com/jamesls/fakeredis.git to /var/folders/m_/xrysd28j79s9mpdc180tsnzc0000gn/T/pip-4q3Ooz-build
Running setup.py egg_info for package from git+git://github.com/jamesls/fakeredis.git
Requirement already satisfied (use --upgrade to upgrade): redis in ./venv/lib/python2.7/site-packages (from fakeredis==0.1.1)
Installing collected packages: fakeredis
Running setup.py install for fakeredis
Successfully installed fakeredis
Cleaning up...
anychance you could update pypi?
StrictRedis' ttl and pttl methods return -1 when expire is not set and -2 when key does not exist (documentation of python redis, documentation of redis), but FakeStrictRedis' methods return None instead.
In [1]: r = redis.StrictRedis(db=0)
In [2]: r.ttl("foo")
Out[2]: -2
In [3]: f = fakeredis.FakeStrictRedis()
In [4]: f.ttl("foo")
Out[4]: None
In redis-py
, StrictRedis
accepts a decode_responses
argument, set to False
by default, but when set to True
, it will decode all bytes values using its charset
argument, default to utf-8.
When my team uses Redis, we set this argument to True
. So we expect strings (we use python 3), not bytes.
But fakeredis
always returns bytes. It encodes everything in bytes but never decodes.
So in our tests we have inconsistent behavior if we swap between a real redis server and fakeredis
.
Viewing the code of fakeredis
it's not so easy to support it because not everything goes through a single method, as does redis-py
with read_response
.
Do you think it's possible to have this feature? Would you accept a pull-request on this subject?
Problem:
The code assumes that min
and max
range query parameters will be numerically comparable values.
e.g. https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L888
redis-py passes on the pieces
as arguments to redis pools, as can be shown at: https://github.com/andymccurdy/redis-py/blob/26c56b9d816c9d1cc1393c04b04b4f1d688f7353/redis/client.py#L1696
This allows for the redis query parameters -inf
and +inf
to be passed transparently, for queries that expect a range.
The implementation should fill-in the ranges in case those strings are passed.
More on the parameters: http://redis.io/commands/zrangebyscore
SADD, HDEL, SREM, ZREM, ZADD, LPUSH, and RPUSH
If you set a key to a value in a hash (thereby provisioning it in Fakeredis' internal storage), but then access it as if it were a simple key, fakeredis
responds by stringifying an internal object and passing it back, where it should raise a TypeError
.
Steps to reproduce:
fakeredis.hset('hash', 'key', 'value')
fakeredis.get('hash')
produces '<fakeredis._StrKeyDict object at 0x10ad47d90>'
How Redis handles this:
HSET blab bleebs 5
HGET blab bleebs
produces 5
GET blab
produces (error) WRONGTYPE Operation against a key holding the wrong kind of value
It would be nice if fakeredis supported LUA scripts like redis does. LUA bindings for python are available so it might not be that hard - never used them though.
Why would it be useful? Well, chances are good that the usage of lua scripts means there is some more complex logic involved that cannot be easily done with simply redis commands. Chances are good that this logic should be well-tested and thus having lua support in fakeredis would be helpful.
It looks like RQ doesn't work with fakeredis. This is probably a concern of RQ, but thought I'd ask here to see what you think.
The problem is that RQ does something different when the connection is Redis
vs StrictRedis
, and uses isinstance
to differentiate.
FakeRedis
and FakeStrictRedis
inherit from object
directly, so this obviously won't work. Since all the methods are there, would it be possible to inherit from Redis
and StrictRedis
instead? All the behavior would still be masked, but now these classes can be dropped in wherever the superclasses are explicitly expected.
Sure, using isinstance
isn't very Pythonic, but doing this would allow the fake classes to fit in more real-world cases. Mock
objects do this too: isinstance(Mock(int), int) # => True
.
pyredis allows for passing in a string of arguments specifying an unlimited number of key/value pairs. I can also pass in **kwargs as a mapping. Fakeredis forces me to pass a dict so the two APIs differ causing testing to guide my implementation.
I'd like to add support for redis lexicographical ordering functions, which are currently missing.
Apparently it's just adding the first key/value of **kwargs
to the ZSet...
Lines 1300 to 1301 in e7e0069
Any reason?
Also not returning the number of items added...
In [21]: f=fakeredis.FakeRedis()
In [22]: f.keys()
Out[22]: []
TypeError Traceback (most recent call last)
in ()
----> 1 f.keys('qwer*')
TypeError: keys() takes exactly 1 argument (2 given)
redis-py supports python3.2 and python3.3, so fakeredis should support those versions as well.
The latest version of redis-py has a "score_cast_func" parameter on the sorted set functions that accept a withscores parameter.
As fakeredis doesn't have a matching parameter, attempts to use with such calls fail
ZRANGE and ZREVRANGE both break score ties by ordering alphabetically on keys, but ZREVRANGE should do so in reverse order
I noticed that hmset
seems to mutate the hash that is passed in for storage. It looks like everything is being cast to a string on the input dict itself, instead of on a copy.
In [1]: conn = fakeredis.FakeStrictRedis()
In [2]: to_store = {'key': [123, 456]}
In [3]: conn.hmset('fake-key', to_store)
Out[3]: True
In [4]: to_store
Out[4]: {'key': '[123, 456]'}
This differs from the behavior of actual redis, which does not mutate the input dictionary
In [5]: conn = get_redis_connection() # wired up to real redis
In [6]: to_store = {'key': [123, 456]}
In [7]: conn.hmset('fake-key', to_store)
Out[7]: True
In [8]: to_store
Out[8]: {'key': [123, 456]}
As you might imagine, this resulted in some very confusing test results.
The test_sort_alpha
fails with the latest redis 2.6 RC:
ResponseError: One or more scores can't be converted into double
Real Redis: ResponseError: wrong number of arguments for 'mget' command
Fake redis: no issues
In the contrived example below, there is a type difference for the returned value in
some circumstances. hset/hget with fake returns a long whereas real returns a string.
mimicking the return type of real redis is probably desired here.
Basically, whatever is shipped off into real redis returns as a string. However, only some content that goes into fake redis returns as a string. Some return as the native object type without pickling.
--code--
import redis
import fakeredis
real = redis.StrictRedis()
fake = fakeredis.FakeStrictRedis()
fake.set("blah", 0L)
real.set("blah", 0L)
print "fake:", type(fake.get("blah"))
print "real:", type(real.get("blah"))
fake.hset("blah1", "key", 0L)
real.hset("blah1", "key", 0L)
print "fake:", type(fake.hget("blah1", "key"))
print "real:", type(real.hget("blah1", "key"))
--output--
fake: <type 'str'>
real: <type 'str'>
fake: <type 'long'>
real: <type 'str'>
While trying to mock a list, I found that trying to return the type returns None.
import fakeredis
redis=fakeredis.FakeStrictRedis()
redis.lpush('bar', 2)
2
print redis.type('bar')
None
Currently fakeredis only looks for libc when searching for an implementation of strtod. On Windows, libc is not available and this function is found in msvcrt.lib.
The fix is to check for both libraries and use whichever is present.
Relevant code:
https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L356
Relevant documentation:
https://github.com/jamesls/fakeredis#unimplemented-commands
There's a discrepancy here. Would it be better to implement the randomkey
method, or just add it to the list of unimplemented commands?
In some cases, people may use redis.Redis.execute_command directly
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.