jlhutch / pylru Goto Github PK
View Code? Open in Web Editor NEWA least recently used (LRU) cache for Python
License: GNU General Public License v2.0
A least recently used (LRU) cache for Python
License: GNU General Public License v2.0
@lrudecorator(100)
def s(x):
return x * x
In [1]: s(1)
Out[1]: 1
In [2]: s(x=1)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-197169c752a0> in <module>()
----> 1 s(x=1)
TypeError: wrapped() got an unexpected keyword argument 'x'
Hello!
I am working with a lot of large numpy.ndarray and posting them over a network. I want to use a LRU cache so that if I make a call with the same numpy.ndarray I will get the same answer. numpy.ndarray take up a lot of memory and are unhashable, so I would need to use id(object)
to identfiy the images.
I would be happy to do the work to enable some kind of id(object)
mode (or md5 etc) for caching if the feature would be accepted. This is very relevant to my use cases and I imagine would help other people working with large unhashable objects.
To be clear this is talking about using @lrudecorator(100, mode=id) or something like that to support caching of large unhashable objects. I can use the current library by interacting with the cache directly to solve my problem but would like to make it even simpler in the future. If above in unclear I will be providing code snippets later on to give more detail to my feature idea.
Let me know what you think
Hi, with 1.0.6, I could do this:
class HTTPDecoder(object):
@lrudecorator(10000)
def device(self, ua):
....
httpDecoder = HTTPDecoder()
print httpDecoder.device('a string')
With 1.0.8, I now get this error. I haven't changed anything:
File "/usr/local/lib/python2.7/dist-packages/pylru.py", line 523, in __call__
value = self.func(*args, **kwargs)
TypeError: device() takes exactly 2 arguments (1 given)
$ virtualenv .
Running virtualenv with interpreter /usr/bin/python2
New python executable in ./bin/python2
Also creating executable in ./bin/python
Installing setuptools, pip...done.
$ bin/pip install pylru
Downloading/unpacking pylru
Downloading pylru-1.0.6.tar.gz
Running setup.py (path:/home/chris/play/build/pylru/setup.py) egg_info for package pylru
Installing collected packages: pylru
Running setup.py install for pylru
Successfully installed pylru
Cleaning up...
$ bin/pip uninstall pylru
Can't uninstall 'pylru'. No files were found to uninstall.
$ bin/pip freeze
argparse==1.2.1
pylru==1.0.6
wsgiref==0.1.2
Got error in multi-threading usage.
Found py3 has built-in lru cache. See https://hg.python.org/cpython/file/3.5/Lib/functools.py#l438
A simple demonstration:
import pylru
from copy import deepcopy
herp = pylru.lrucache(size=1024)
derp = deepcopy(herp)
A copy of the lrucache
object, or the lvalue bound to the original
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
<...snip ...>
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/copy.py", line 137, in deepcopy
d = id(x)
RecursionError: maximum recursion depth exceeded while calling a Python object
A simple subclass of pylru.lrucache
that returns a pointer to the original on deepcopy operations gets me where I'd like to be:
class LRUCache(pylru.lrucache):
def __deepcopy__(self, memodict=None):
return self
SIGABRT
crashes are similarly bad form. The software should provide a lrucache.__deepcopy__
method that either:NotImplementedError
, RuntimeError
, or similar, ORlrucache.__init__
)Can you upload new release with improvements to pypi?
I really like your project. I was reading docs, trying stuff and stuck on obtaining control over cache of manager objects, particularly FunctionCacheManager
.
It took me a while to understand that manager objects have cache
property. My IDE didn't show it. I haven't found a mention about it in docs. Reading docs my first though was that FunctionCacheManager
has nothing more than size()
and clear()
.
I think that this a core feature of the project, and clearly writing it in docs could ease the understanding of the project capabilities.
For better security, could you please sign all upcoming git commits?
If required, I can make the case for the usefulness of signing git commits.
See https://docs.python.org/2/library/functools.html#functools.wraps for details.
The following ipython session demonstrates the issue:
In [8]: def s_plain(y):
...: "square a number"
...: return y * y
...:
In [9]: s_plain.__doc__
Out[9]: 'square a number'
In [10]: s_plain.__name__
Out[10]: 's_plain'
In [11]: @lrudecorator(100)
def s(x):
"square a number"
return x * x
....:
In [12]: s.__doc__
In [13]: s.__name__
Out[13]: 'wrapped'
Hi,
I am trying to use pylru to cache methods in a class. Say:
class myclass(object):
def __init__(self,data):
self.data = data
@lrudecorator(100)
def longcomputation(self):
#do some "hard work" depending on self.data and store results in x
print "I am working hard"
x = self.data
return x
def update(self,newdata):
self.data = newdata
Now if I do:
obj = myclass("mydata")
print obj.longcomputation()
obj.update("mychangeddata")
print obj.longcomputation()
I get
I am working hard
mydata
mydata
printed on the console, as it should be.
The question I have is if there is some way to clear the cached method so that if I do:
def update(self,newdata):
self.data = newdata
# "something to clear the cache for longcomputation"
I get
I am working hard
mydata
I am working hard
mynewdata
The function cached.keys returns only partial keys, and many repetitions.
why add the _dlnode when __init__
?
if the size is 100000, then I need Consume the memory of 100000 node before i use it.
Suggestion:
table
and the size is in limit, we can add node dynamic
.size = -1
, when -1, we do not Eliminate or remove cache key.thanks for your work, i will use it in https://github.com/hustcc/wrapcache/blob/master/wrapcache/database/init.py, it is a wrap cache, use it for lru memory cache.
It appears that calling cache.clear() doesn't call the eviction callback... Is this by design or an accident?
I'm working on a project where I have to expose a dictionary interface to a data store so that pylru could wrap it. It is not that much of a trouble but it would be nicer and easier to maintain if pylru allowed for generic interface to a data store (e.g., function calls)
I'm willing to submit a pull request for a change like that if you give the green light.
Hi. I want to use pylru in an application and use the callback option. However, my callback would need additional information in order to execute. I would like to hand that information into the init() function as an optional dictionary. The callback would know what to do with it.
Is this feasible or is there already a way to do this that I have missed?
If this is the wrong place to request a feature (or propose a code mod) please let me know. Thanks,
< david
Hi!
https://github.com/jlhutch/pylru/blob/master/pylru.py#L549
value = func(*args, **kwargs)
For python3.4 need check if is coroutine and add yield from
value = yield from func(*args, **kwargs)
if asyncio.iscoroutinefunction(expect_handler)
But maybe it will anyway raise error on python 2.
Could you please upload your OpenPGP / gpg public key somewhere?
related:
#30
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.