Git Product home page Git Product logo

django-fancy-cache's Introduction

django-fancy-cache

Copyright Peter Bengtsson, [email protected], 2013-2022

License: BSD

About django-fancy-cache

A Django cache_page decorator on steroids.

Unlike the stock django.views.decorators.cache.change_page this decorator makes it possible to set a key_prefix that is a callable. This callable is passed the request and if it returns None the page is not cached.

Also, you can set another callable called post_process_response (which is passed the response and the request) which can do some additional changes to the response before it's set in cache.

Lastly, you can set post_process_response_always=True so that the post_process_response callable is always called, even when the response is coming from the cache.

How to use it

In your Django views:

from fancy_cache import cache_page
from django.utils.decorators import method_decorator
from django.views.generic import TemplateView

@cache_page(60 * 60)
def myview(request):
    return render(request, 'page1.html')

def prefixer(request):
    if request.method != 'GET':
        return None
    if request.GET.get('no-cache'):
        return None
    return 'myprefix'

@cache_page(60 * 60, key_prefix=prefixer)
def myotherview(request):
    return render(request, 'page2.html')

def post_processor(response, request):
    response.content += '<!-- this was post processed -->'
    return response

@cache_page(
    60 * 60,
    key_prefix=prefixer,
    post_process_response=post_processor)
def yetanotherotherview(request):
    return render(request, 'page3.html')


class MyClassBasedView(TemplateView):
    template_name = 'page4.html'

    @method_decorator(cache_page(60*60))
    def get(self, request, *args, **kwargs):
        return super().get(request, *args, **kwargs)

Optional uses

If you want to you can have django-fancy-cache record every URL it caches. This can be useful for things like invalidation or curious statistical inspection.

You can either switch this on on the decorator itself. Like this:

from fancy_cache import cache_page

@cache_page(60 * 60, remember_all_urls=True)
def myview(request):
    return render(request, 'page1.html')

Or, more conveniently to apply it to all uses of the cache_page decorator you can set the default in your settings with:

FANCY_REMEMBER_ALL_URLS = True

Now, suppose you have the this option enabled. Now you can do things like this:

>>> from fancy_cache.memory import find_urls
>>> list(find_urls(['/some/searchpath', '/or/like/*/this.*']))
>>> # or, to get all:
>>> list(find_urls([]))

There is also another option to this and that is to purge (aka. invalidate) the remembered URLs. You simply all the purge=True option like this:

>>> from fancy_cache.memory import find_urls
>>> list(find_urls([], purge=True))

Note: Since find_urls() returns a generator, the purging won't happen unless you exhaust the generator. E.g. looping over it or turning it into a list.

If you are using Memcached, you must enable check-and-set to remember all urls by enabling the FANCY_USE_MEMCACHED_CHECK_AND_SET flag and enabling cas in your CACHES settings:

# in settings.py

FANCY_USE_MEMCACHED_CHECK_AND_SET = True

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
        # This OPTIONS setting enables Memcached check-and-set which is
        # required for remember_all_urls or FANCY_REMEMBER_ALL_URLS.
        'OPTIONS': {
            'behaviors': {
                'cas': True
            }
        }
    }
 }

The second way to inspect all recorded URLs is to use the fancy-cache management command. This is only available if you have added fancy_cache to your INSTALLED_APPS setting. Now you can do this:

$ ./manage.py fancy-cache --help
$ ./manage.py fancy-cache
$ ./manage.py fancy-cache /some/searchpath /or/like/*/this.*
$ ./manage.py fancy-cache /some/place/* --purge
$ # or to purge them all!
$ ./manage.py fancy-cache --purge

Note, it will only print out URLs that if found (and purged, if applicable).

The third way to inspect the recorded URLs is to add this to your root urls.py:

url(r'fancy-cache', include('fancy_cache.urls')),

Now, if you visit http://localhost:8000/fancy-cache you get a table listing every URL that django-fancy-cache has recorded.

Optional uses (for the exceptionally curious)

If you have enabled FANCY_REMEMBER_ALL_URLS you can also enable FANCY_REMEMBER_STATS_ALL_URLS in your settings. What this does is that it attempts to count the number of cache hits and cache misses you have for each URL.

This counting of hits and misses is configured to last "a long time". Possibly longer than you cache your view. So, over time you can expect to have more than one miss because your view cache expires and it starts over.

You can see the stats whenever you use any of the ways described in the section above. For example like this:

>>> from fancy_cache.memory import find_urls
>>> found = list(find_urls([]))[0]
>>> found[0]
'/some/page.html'
>>> found[2]
{'hits': 1235, 'misses': 12}

There is obviously a small additional performance cost of using the FANCY_REMEMBER_ALL_URLS and/or FANCY_REMEMBER_STATS_ALL_URLS in your project so only use it if you don't have any smarter way to invalidate, for debugging or if you really want make it possible to purge all cached responses when you run an upgrade of your site or something.

Running the test suite

The simplest way is to simply run:

$ pip install tox
$ tox

Or to run it without tox you can simply run:

$ export PYTHONPATH=`pwd`
$ export DJANGO_SETTINGS_MODULE=fancy_tests.tests.settings
$ django-admin.py test

Changelog

1.3.1
  • Fix a bug whereby FANCY_COMPRESS_REMEMBERED_URLS setting raises a TypeError upon first implementation.
1.3.0
  • Enable FANCY_COMPRESS_REMEMBERED_URLS setting to compress remembered_urls dictionary when FANCY_REMEMBER_ALL_URLS is True.
  • Bugfix: use correct location for REMEMBERED_URLS when using Memcached.
  • Add support for Python 3.11, Django 4.1 & 4.2
  • Drop support for Python < 3.8, Django < 3.2, Django 4.0
1.2.1
  • Bugfix: conflict between the DummyCache backend when FANCY_USE_MEMCACHED_CHECK_AND_SET is True
1.2.0
  • Restructure the remembered_urls cache dict to clean up stale entries
  • Update FancyCacheMiddleware to match latest Django CacheMiddlware (Also renames to FancyCacheMiddleware)
  • Apply Memcached check-and-set to the delete_keys function if settings.FANCY_USE_MEMCACHED_CHECK_AND_SET = True
  • Drop support for Python <3.6
  • Add support for Python 3.10 and Django 4.0
1.1.0
  • If you use Memcached you can set settings.FANCY_USE_MEMCACHED_CHECK_AND_SET = True so that you can use cache._cache.cas which only workd with Memcached
1.0.0
  • Drop support for Python <3.5 and Django <2.2.0
0.11.0
  • Fix for parse_qs correctly between Python 2 and Python 3
0.10.0
  • Fix for keeping blank strings in query strings. #39
0.9.0
  • Django 1.10 support
0.8.2
  • Remove deprecated way to define URL patterns and tests in python 3.5
0.8.1
  • Ability to specify different cache backends to be used #31
0.8.0
  • Started keeping a Changelog

django-fancy-cache's People

Contributors

aaronvanderlip avatar idealatom avatar justinfay avatar peterbe avatar pigmonkey avatar pyman avatar regadas avatar skorokithakis avatar timbutler avatar ypcrumble avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-fancy-cache's Issues

Add support for Django 3.1

First of all: thank you for this project - I hope it is still somewhat maintained.

According to the Django documentation support for version 3.0 is dropped in one month (and 3.2 LTS should be released soon).

When I upgrade my Django application from v3.0.x to v3.1.x I get the following error when attempting to run the server:

dev@vm:~$ python manage.py runserver

[..]

[..]/venv/lib/python3.9/site-packages/django/utils/decorators.py", line 117, in _decorator
    middleware = middleware_class(view_func, *m_args, **m_kwargs)
TypeError: __init__() got multiple values for argument 'key_prefix'

which is coming from the @cache_page decorator:

from fancy_cache import cache_page
from myproject.utils import cache

[..]

@cache_page(TTL, key_prefix=cache.cache_prefixer)
def my_view(...):
    [..]

It works as expected with Django <= 3.0.x.

Race condition with find_urls' purge=True

I'm starting to see frequent race conditions when using find_urls with purge=True when there is no existing cache entry.

The mechanism is as follows:

Request A (POST object Z) Request B (GET list of objects)
Begin INSERT operation (slow) Check for cache (nothing found, fast)
Begin SELECT operation (fast)
SELECT operation returns a list of objects without object Z
INSERT finishes creating object Z, begin find_urls via the object's post_save signal Begin setting the cache with stale data
find_urls finishes and finds no URLs
Finish setting the cache with stale data

RemovedInDjango110Warning for url patterns in stats views

I integrated the following urls to allow for fancy cache stats views.

url(r'fancy-cache', include('fancy_cache.urls')),

Whenever I access any of the views I get the following deprecation warning.

/vagrant/env/local/lib/python2.7/site-packages/fancy_cache/urls.py:7: RemovedInDjango110Warning: django.conf.urls.patterns() is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of django.conf.urls.url() instances instead.                                                                                                  
  url(r'^$', views.home, name='home'),                                                                                

Client side caching may be overriding cache invalidation

I haven't been able to fully research this as it's quite complex and I'm not an expert on caching headers, but plan to in the future and want to write it down to make sure I'm not on a completely wrong path.

What I'm experiencing is that browsers that are accessing pages that are cached and invalidated using django-fancy-cache are still pulling a resource (from disk cache) on chrome. Hard refresh of the browser solves the issue which makes me believe this is an issue with browser caching results to disk.

Question - should items cached and invalided by fancy-cache include a no-cache; no-store header? Otherwise is there a good way to make sure that the browser doesn't pull a stale response from its local cache?

I've looked into Etag and I suspect that might be a good alternative; as far as I can tell this would allow cache invalidation based on the hashed value of the response.

csrftoken and cookie vary

Got any hints on how django-fancy-cache could be used to cache views without breaking the csrftoken?

The issue is that you could just ignore csrftoken cookie like this, but then the cache would store a fixed csrftoken in the HTML, all users would get the same token and things will not work (csrf token validation would fail)

Without fixing this issue, the cache hit/miss rate plummets

To make things worse, if the user accesses any page on your site (lets say /contact) with a form he'll get this http response header:

csrftoken=GfV...RYTB; expires=Mon, 13-Jul-2015 17:21:16 GMT; 
           Max-Age=31449600; Path=/; secure

Please note the "Path=/", which indicates that all the next requests performed by this user are going to include the csrftoken cookie and won't be retrieved from the cache.

PyPy tests are flaky

PyPy tests are flaky so we commented them out. Chances are this project works in PyPy, but until we have a maintainer for PyPy it is simply not included in the test automation.

See discussion in #63

Consider keeping blank querystring values

I use django-fancy-cache for the only_get_keys functionality.

In one of my views I watch for the pdf querystring and return the page as a PDF if it is present. /foo/bar/ returns the normal HTML web page, /foo/bar/?pdf returns the page as a PDF.

I have set only_get_keys=['pdf'] for this view. However, I noticed that in normal usage (where /foo/bar/ is generally hit first) Django returns the HTML page when visiting /foo/bar/?pdf.

This is caused by the use of cgi.parse_qs, which will not return empty values by default.

>>> import cgi
>>> cgi.parse_qs('pdf')
{}
>>> cgi.parse_qs('pdf=')
{}
>>> cgi.parse_qs('pdf=a')
{'pdf': ['a']}

For my use case, I would like to pass keep_blank_values=True to the parser.

>>> import cgi
>>> cgi.parse_qs('pdf', keep_blank_values=True)
{'pdf': ['']}

It appears that these blank values are valid according to the RFC.

For how django-fancy-cache uses the querystrings, I think it makes sense to use keep_blank_values=True. I would expect /, /?pdf and /?pdf=foo to all have different cache keys. However I admit that my use of /?pdf is sort of lazy, and I could just avoid this problem by watching for ?pdf=true or something. I thought I would open an issue rather than submitting a pull request to see if there are any other opinions on the matter.

I'm happy to submit a pull request for this if people are ok with it.

Setting a cache by periodic management command or asynchronous task

The main problem I have with most caching methods is the first request. My experience on services like Heroku and other virtualized databases is that 'some' queries/pages can be slower than an arthritic snail if the database is not warmed into memory, so that first cold request on a new or expiring cache can be a problem (particularly on Heroku which times-out after 30 seconds). You can fudge it for the user by ajax but I just want to do it right for every user every time not just the second person through the door.

To solve this problem I'm keen to crawl slow pages by scheduled management command at periodic intervals to refresh the cache. The second part of that is that I want to always set the cache when the management command is run so that I can control when the page is refreshed regardless of the timeout. So for example when an object is updated that is part of a complex query I can asynchronously pass off to a job queue the task of setting the cache.

My first thought was to simply create static pages and serve them up via reverse proxy/CDN, but my second thought was to customise the django cache middleware to consume a GET key/value pair like /path/?setcache=some-api-key. Given the key and the correct value the cache is always updated (assuming it uses a cache decorator).

Since this is a lovely fresh iteration on the stock django caching methods I wanted to get your thoughts as someone who has dug into the caching middleware recently as to a) it's feasibility as a feature and b) whether it is a feature you might accept if I did the work on it.

Update CI to use GitHub Actions

I believe Travis removed access to open source repos recently, I'm happy to convert CI to GitHub Actions if that's welcomed!

RemovedInDjango110Warning for OptionParser usage in management command

I'm using Django 1.9 and django-fancy-cache 0.8.0 and got the following warning when I called manage.py fancy-urls to check the stats.

/vagrant/env/local/lib/python2.7/site-packages/django/core/management/__init__.py:345: RemovedInDjango110Warning: OptionParser usage for Django management commands is deprecated, use ArgumentParser instead
  self.fetch_command(subcommand).run_from_argv(self.argv)

OptionParser is deprecated and should be using ArgumentParser instead.

0 / None for cache timeout doesn't work as expected

As of Django 1.7 (to be out soon), setting the cache timeout to None means "never expire." In the fancy cache middleware.py, however, boolean checks against the timeout are done, making setting an infinite timeout impossible.

Current workaround: set timeout to a huge integer.

Incompatible with django 4.1

It is mentioned in the code and it's true: https://github.com/peterbe/django-fancy-cache/blob/master/fancy_cache/middleware.py#L382

Unfortunately the code fails with the following error: AttributeError: can't set attribute

Traceback (most recent call last):
  File "/app/manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.9/site-packages/django/core/management/base.py", line 443, in execute
    self.check()
  File "/opt/venv/lib/python3.9/site-packages/django/core/management/base.py", line 475, in check
    all_issues = checks.run_checks(
  File "/opt/venv/lib/python3.9/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/opt/venv/lib/python3.9/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
  File "/opt/venv/lib/python3.9/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    return check_method()
  File "/opt/venv/lib/python3.9/site-packages/django/urls/resolvers.py", line 494, in check
    for pattern in self.url_patterns:
  File "/opt/venv/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/opt/venv/lib/python3.9/site-packages/django/urls/resolvers.py", line 715, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/opt/venv/lib/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/opt/venv/lib/python3.9/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/config/urls.py", line 12, in <module>
    from mybaze.affiliates import urls as affiliates_urls
  File "/app/mybaze/affiliates/urls.py", line 14, in <module>
    cache_page(timeout=WIDGET_CACHE_TTL)(ProductsShowcaseView.as_view()),
  File "/opt/venv/lib/python3.9/site-packages/django/utils/decorators.py", line 121, in _decorator
    middleware = middleware_class(view_func, *m_args, **m_kwargs)
  File "/opt/venv/lib/python3.9/site-packages/fancy_cache/middleware.py", line 390, in __init__
    self.cache = caches[self.cache_alias]
AttributeError: can't set attribute

Add setting to automatically clear or update the cache when saving models using the admin interface.

It would be very useful to have a setting like FANCY_CLEAR_CACHE_ON_MODEL_ADMIN_SAVE = True|False or FANCY_UPDATE_CACHE_ON_MODEL_ADMIN_SAVE = True|False to automatically clear or update the cache when saving any object using the admin interface.

This can be done easily by attaching a post_save signal to LogEntry model:

from django.contrib.admin.models import LogEntry
from django.db.models.signals import post_save
from django.dispatch import receiver
from fancy_cache.memory import find_urls

@receiver(post_save, sender=LogEntry, dispatch_uid="clear_fancy_cache")
def clear_fancy_cache(sender, instance, **kwargs):
     #clear cache
     list(find_urls([], purge=True))
     #now to update the cache just requests all urls that have been purged

To be more specific it would be better to have the possibility to specify for each decorator a list of model classes that on save will invalidate just the cache specified by key_prefixer.

@cache_page(60 * 60,
    key_prefixer=prefixer,
    post_process_response=post_processor, 
    clear_on_post_save_models=[MyModel1, MyModel2, MyModel3]     
)

Django 1.7 compability

Please see the 1.7 release docs.

It seems django-fancy-cache's partial query parameter matching (achieved via patching the request object) won't work with django 1.7.

[Suggestion] Use stale-bot or just close all issues 2016 and earlier

@peterbe just trying to be helpful keeping this repo up to date! It might be nice to add stale-bot which would ping users on all old issues so they could participate if they're still using the repo and/or experiencing the issue and could provide helpful info on the issue(s). Or you could do this manually - I've taken a look and I'd definitely do that for anything < 2016.

Can't use DummyCache and django-fancy-cache together

During development (runserver) I want to use DummyCache, its simply easier when working on improving templates and views. The problem is that fancy-cache breaks if I set this in my cache configuration:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
    },
}

This is the traceback

Traceback:
File "/home/pablo/virtual-envs/spam/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  115.                         response = callback(request, *callback_args, **callback_kwargs)
File "/home/pablo/virtual-envs/spam/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  83.                     result = middleware.process_request(request)
File "/home/pablo/virtual-envs/spam/local/lib/python2.7/site-packages/fancy_cache/middleware.py" in process_request
  175.             cache.incr(cache_key)
File "/home/pablo/virtual-envs/spam/local/lib/python2.7/site-packages/django/core/cache/backends/base.py" in incr
  142.             raise ValueError("Key '%s' not found" % key)

Exception Type: ValueError at /
Exception Value: Key 'dedb5c6fc930a4bf89f25f5442876c38' not found

FANCY_REMEMBER_ALL_URLS option and performance over time

Are there known limitations to this option? I had enabled it to give more visibility to our system and at about ~22,000 entries, the endpoints that were being tracked stopped being functional. Clearing the fancy-urls key solved the issue immediately.

In [1]: v = cache.get('fancy-urls')

In [2]: len(v.keys())
Out[2]: 22717

Above was the final count before deleting the key.

integrate this to official django

I have spent days trying to do granular invalidation using django built-in cache, there were no way to do it, This package was perfect to achieve it.

Why don't you adapt it and integrate it to official django ?

AttributeError: 'FetchFromCacheMiddleware' object has no attribute 'cache_anonymous_only'

Hi Peter,
first of all, I wanted to thank you for the nice piece of software you wrote.
Django 1.9 and Python 3.5:

I added your code to my project by doing a drop in replacement of the UpdateCacheMiddleware and FetchFromCacheMiddleware classes:
e.g. in my settings:

MIDDLEWARE_CLASSES = (
'fancy_cache.middleware.UpdateCacheMiddleware',
...,
'fancy_cache.middleware.FetchFromCacheMiddleware',
)

I then added this annotation into my 'url.py':
cache_page(cache_timeout= 600, forget_get_keys=['draw'],post_process_response_always=replace_draw_id)(MyClassView.as_view())

This created the following error:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/django/core/handlers/base.py", line 123, in get_response
response = middleware_method(request)
File "/basic/fancy_cache/middleware.py", line 167, in process_request
response = self._process_request(request)
File "/basic/fancy_cache/middleware.py", line 182, in _process_request
if self.cache_anonymous_only:
AttributeError: 'FetchFromCacheMiddleware' object has no attribute 'cache_anonymous_only'

Looking at the code, there is no init function for both classes. Just copying init from
CacheMiddleware fixes the problem.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.