noripyt / django-cachalot Goto Github PK

View Code? Open in Web Editor NEW

1.2K 24.0 145.0 1.57 MB

No effort, no worry, maximum performance.

Home Page: http://django-cachalot.readthedocs.io

License: BSD 3-Clause "New" or "Revised" License

Python 99.74% HTML 0.26%

django python performance cache

django-cachalot's Introduction

Django Cachalot

Caches your Django ORM queries and automatically invalidates them.

Documentation: http://django-cachalot.readthedocs.io

Table of Contents:

Quickstart
Usage
Hacking
Benchmark
Third-Party Cache Comparison
Discussion

Quickstart

Cachalot officially supports Python 3.7-3.11 and Django 3.2, 4.1, 4.2 with the databases PostgreSQL, SQLite, and MySQL.

Note: an upper limit on Django version is set for your safety. Please do not ignore it.

Usage

pip install django-cachalot
Add 'cachalot', to your INSTALLED_APPS
If you use multiple servers with a common cache server, double check their clock synchronisation
If you modify data outside Django – typically after restoring a SQL database –, use the manage.py command
Be aware of the few other limits
If you use django-debug-toolbar, you can add 'cachalot.panels.CachalotPanel', to your DEBUG_TOOLBAR_PANELS
Enjoy!

Hacking

To start developing, install the requirements and run the tests via tox.

Make sure you have the following services:

Memcached
Redis
PostgreSQL
MySQL

For setup:

Install: pip install -r requirements/hacking.txt
For PostgreSQL: CREATE ROLE cachalot LOGIN SUPERUSER;
Run: tox --current-env to run the test suite on your current Python version.
You can also run specific databases and Django versions: tox -e py38-django3.1-postgresql-redis

Benchmark

Currently, benchmarks are supported on Linux and Mac/Darwin. You will need a database called "cachalot" on MySQL and PostgreSQL. Additionally, on PostgreSQL, you will need to create a role called "cachalot." You can also run the benchmark, and it'll raise errors with specific instructions for how to fix it.

Install: pip install -r requirements/benchmark.txt
Run: python benchmark.py

The output will be in benchmark/TODAY'S_DATE/

TODO Create Docker-compose file to allow for easier running of data.

Third-Party Cache Comparison

There are three main third party caches: cachalot, cache-machine, and cache-ops. Which do you use? We suggest a mix:

TL;DR Use cachalot for cold or modified <50 times per minutes (Most people should stick with only cachalot since you most likely won't need to scale to the point of needing cache-machine added to the bowl). If you're an enterprise that already has huge statistics, then mixing cold caches for cachalot and your hot caches with cache-machine is the best mix. However, when performing joins with select_related and prefetch_related, you can get a nearly 100x speed up for your initial deployment.

Recall, cachalot caches THE ENTIRE TABLE. That's where its inefficiency stems from: if you keep updating the records, then the cachalot constantly invalidates the table and re-caches. Luckily caching is very efficient, it's just the cache invalidation part that kills all our systems. Look at Note 1 below to see how Reddit deals with it.

Cachalot is more-or-less intended for cold caches or "just-right" conditions. If you find a partition library for Django (also authored but work-in-progress by Andrew Chen Wang), then the caching will work better since sharding the cold/accessed-the-least records aren't invalidated as much.

Cachalot is good when there are <50 modifications per minute on a hot cached table. This is mostly due to cache invalidation. It's the same with any cache, which is why we suggest you use cache-machine for hot caches. Cache-machine caches individual objects, taking up more in the memory store but invalidates those individual objects instead of the entire table like cachalot.

Yes, the bane of our entire existence lies in cache invalidation and naming variables. Why does cachalot suck when stuck with a huge table that's modified rapidly? Since you've mixed your cold (90% of) with your hot (10% of) records, you're caching and invalidating an entire table. It's like trying to boil 1 ton of noodles inside ONE pot instead of 100 pots boiling 1 ton of noodles. Which is more efficient? The splitting up of them.

Note 1: My personal experience with caches stems from Reddit's: https://web.archive.org/web/20210803213621/https://redditblog.com/2017/01/17/caching-at-reddit/

Note 2: Technical comparison: https://django-cachalot.readthedocs.io/en/latest/introduction.html#comparison-with-similar-tools

Discussion

Help? Technical chat? It's here on Discord.

Legacy chats:

django-cachalot's People

Contributors

Stargazers

Watchers

Forkers

vishen jansonzhou ithacadream magarcia felsen pahpa helenwarren relekang josephdrose dan-passaro dalberto pombredanne jjmontalbo devoto13 karnebeck jcugat bennylope zenotech arunaharsa dougmacnerland ryanmrubin androidsx chadmasso xino12 lukaszkoziara eriktelepovsky kapilratnani jaredlewis ambitioninc loggi anhzhi harrypotterismyname avorio slurms izimobil frasern yourcelf koleror uncovertruth thecardcheat agateblue hugoren mwvolo mighty-justice manarah-aswaq ramaprv tawanda kunalgrover05 artenepo slaskis skio-music leetcode-opensource sidebarchats jcass77 mbjerkness centriam mediamachineslab federicobond glogiotatidis acu anmolakhilesh pekopt oppianmatt whoppah langal joshlabs strogo amrlotfy77 heyhugo jnoortheen dmippolitov ridecell altimore infernonet allenjseb petrdlouhy abbad euginelethal eliziario beda42 benjfield susoo shield007 naartti aemitos lucasmoeskops natureshadow ruan-stint salqueng milos-u shenek jjorissen52 dbartenstein debdolph shitalluitel sumpfralle djaizz sohaib-dev sonsandco janovalaska

django-cachalot's Issues

Add documentation on time syncing

Since the current time is used to determine validity of table cache keys, it's important that when deploying against multiple application servers, they all have time synchronized. I'd like to suggest adding this to the docs somewhere prominently.

Even a few seconds difference was enough to cause issues for us in production.

Cachalot is making too many calls to memcache server

We are using Cachalot in one of our projects and and we are observer too many get_multi calls to our memcache servers on all our environments. On my local environment when I run the same code with cachalot disabled number of hit to memcache reduces significantly. With Cachalot enabled the number of get_multi calls is in 100s even for opening a list page in Django admin.

Most of the call retrieve keys which have some floating point values saved for them and these floating point values looks like to be epoch timestamps. Following are few example keys and their values retrieved from get_multi calls:

get_many ['f330760c95588f8c795d0df3f699f90ae143e26a', 'b5ac959681ff06eff632d25a5961a4d3cad09b42', '8c6f559b37d1caa6e3d892aa317eca64377a43af', '887f8247d0ff979264c73b069f790024b2aa98e1', '5c01fcd741777a41b0557cb7a62501de3a0b31d6'] {'5c01fcd741777a41b0557cb7a62501de3a0b31d6': (1476366950.845727, []), '8c6f559b37d1caa6e3d892aa317eca64377a43af': 1476366950.486771, 'f330760c95588f8c795d0df3f699f90ae143e26a': 1476340903.45008, '887f8247d0ff979264c73b069f790024b2aa98e1': 1476340903.45008, 'b5ac959681ff06eff632d25a5961a4d3cad09b42': 1476340903.45008}

And following is a set_multi call which sets one such key:

set_many {'8c6f559b37d1caa6e3d892aa317eca64377a43af': 1476366950.903075}

Could someone please tell me why Cachalot is caching these floating point numbers and why there are so many cache hits of these keys?

Exception, when app without models is migrated

I get following exception, when try to run ./manage.py migrate:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 2358, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1778, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc) 
  File "manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 165, in handle
    emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/core/management/sql.py", line 268, in emit_post_migrate_signal
    using=db)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 198, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/cachalot/monkey_patch.py", line 161, in _invalidate_on_migration
    invalidate_models(sender.get_models(), db_alias=kwargs['using'])
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/cachalot/api.py", line 66, in invalidate_models
    cache_alias, db_alias)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/cachalot/api.py", line 47, in invalidate_tables
    _invalidate_table_cache_keys(cache, table_cache_keys)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/cachalot/utils.py", line 133, in _invalidate_table_cache_keys
    cache.set_many(d, None)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis_cache/backends/single.py", line 67, in set_many
    return self._set_many(self.master_client, new_data)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis_cache/backends/base.py", line 315, in _set_many
    return client.mset(data)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis/client.py", line 940, in mset
    return self.execute_command('MSET', *items)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis/client.py", line 565, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis/client.py", line 577, in parse_response
    response = connection.read_response()
  File "/Users/devoto13/.virtualenvs/backend/lib/python3.4/site-packages/redis/connection.py", line 574, in read_response
    raise response
redis.exceptions.ResponseError: wrong number of arguments for 'mset' command

The problems seems to be in the _invalidate_table_cache_keys(), when it receives empty sequence as a second argument. And issues exists for all apps without models.

CachalotPanel no longer works. Error importing _invalidate_tables in transaction.py

adding cachalot.panels.CachalotPanel to DEBUG_TOOLBAR_PANELS causes: django.core.exceptions.ImproperlyConfigured: Error importing debug panel cachalot.panels: "cannot import name '_invalidate_tables'".

Full traceback:

Traceback (most recent call last):
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 113, in inner_run
    autoreload.raise_last_exception()
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 249, in raise_last_exception
    six.reraise(*_exception)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/apps.py", line 12, in <module>
    from debug_toolbar.middleware import DebugToolbarMiddleware
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/middleware.py", line 17, in <module>
    from debug_toolbar.toolbar import DebugToolbar
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 151, in <module>
    urlpatterns = DebugToolbar.get_urls()
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 145, in get_urls
    for panel_class in cls.get_panel_classes():
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 122, in get_panel_classes
    (panel_module, e))
django.core.exceptions.ImproperlyConfigured: Error importing debug panel cachalot.panels: "cannot import name '_invalidate_tables'"
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x10d2cc158>
Traceback (most recent call last):
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 118, in get_panel_classes
    mod = import_module(panel_module)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/cachalot/panels.py", line 14, in <module>
    from .utils import _get_table_cache_key
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/cachalot/utils.py", line 18, in <module>
    from .transaction import AtomicCache
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/cachalot/transaction.py", line 40, in <module>
    from .utils import _invalidate_tables
ImportError: cannot import name '_invalidate_tables'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 113, in inner_run
    autoreload.raise_last_exception()
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 249, in raise_last_exception
    six.reraise(*_exception)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/apps.py", line 12, in <module>
    from debug_toolbar.middleware import DebugToolbarMiddleware
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/middleware.py", line 17, in <module>
    from debug_toolbar.toolbar import DebugToolbar
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 151, in <module>
    urlpatterns = DebugToolbar.get_urls()
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 145, in get_urls
    for panel_class in cls.get_panel_classes():
  File "/Users/myuser/.virtualenvs/myproj/lib/python3.5/site-packages/debug_toolbar/toolbar.py", line 122, in get_panel_classes
    (panel_module, e))
django.core.exceptions.ImproperlyConfigured: Error importing debug panel cachalot.panels: "cannot import name '_invalidate_tables'"

Cache Invalidation Problem in DRF

I am using cachalot with Django Rest Framework and redis. One viewset has a complicated queryset, with slecte_related and prefetch_related. post in this viewset did not invalidate the existing cached queries. I am using redis master/slave setup. Is this the problem?

Support for django_pylibmc (Heroku)?

Hi there. I was looking to use django-cachalot with a Django site I've deployed on Heroku. I'm using the third-party django_pylibmc package to make use of Heroku's Memcached addon, because Django's default pylibmc (i.e. django.core.cache.backends.memcached.PyLibMCCache) does not support the binary protocol (see this StackOverflow post and this DjangoProject ticket). With the django_pylibmc package, I am able to get my Heroku site working with Memcachier.

Unfortunately, django_pylibmc does not work with django-cachalot. When I run heroku run python manage.py migrate, I get the following error message:

(cachalot.E001) `django_pylibmc.memcached.PyLibMCCache` is not compatible with django-cachalot.

Are there any plans for supporting django_pylibmc? I'd really love it if I could use django-cachalot on my Heroku site, since it seems to be a very robust and useful package.

500 response on cache miss

running django-cachalot with Django 1.8, memcached and ElastiCache

Thoughts on post cache invalidation signals?

Would love to be able to do some extra logic on invalidation of cache.
I was personally thinking it could be useful to invalidate api/response caches on query cache invalidations but would love to hear other use cases and see if this is a good thing to do in the first place.

Don't cache the admin UI queries?

We are noticing that our cache is 3 times the size of our database, and as it is on Heroku it is getting very expensive.

Could this be due to caching admin UI queries? We only want to cache the user facing parts of our site (news articles/typical CMS stuff), and something is amiss...

update PIP for 1.4.1

Hello,
there is not 1.4.1 on PIP. Can you update that?
Thank you in advance,
Hanz

Deleting Cache Instead of Setting it to None

Hi All,

I'm using Django 1.8 / Memcached / Cachalot. After installing Cachalot, I noticed that the memory usage of memcached was ever-increasing.

After looking into the source, I realized that Cachalot invalidates keys by setting the value to None instead of calling cache.delete(). However, because Memcached is content-agnostic, this doesn't actually free up the allocated memory. Instead, the object sits in memcached until an eviction takes place. This makes it really hard to determine how much space Django-Cachalot requires in production to operate at maximum efficiency.

Is there a reason cache.delete() is not being used? Could we at least use it for the Memcached Backend? Happy to do a PR if you agree.

Best.

support for Django 1.10?

as titled

Adding 'cachalot' to INSTALLED_APPS makes test suite four times slower

We are evaluating django-cachalot. We are using:

django-cachalot 1.0.0
Django 1.6.8
Postgres 9.3 and 9.4

Other possibly relevant package we use is django-transaction-hooks 0.2.

We noticed that when we have 'cachalot' present in INSTALLED_APPS our test suite run time goes to 600+ seconds. Commenting it out makes that go back to 160+ seconds.

(Found this trying to debug some lockups we are seeing in our CI environment when:

Running the full South migration history
Running our test suite

which made us to go back and run the test suite in our local development environment.)

Caching backend is memcached. Memory assigned to it with -m doesn't seem to have any influence in observed behavior.

I'll try to isolate the test cases which actually are the ones that get slower, in the meanwhile I wanted to open this issue because the slowdown happens even when the CACHALOT_ENABLED setting is False.

Add a template tag to get the last invalidation timestamp for given model(s)

Such a template tag could be extremely useful for partial template caching. Something like:

{% load cachalot cache %}

{% get_last_invalidation auth.User yourapp.YourModel as last_invalidation %}
{% cache 86400 cache_key last_invalidation %}
    […]
{% endcache %}

Amongst the invalidation timestamps of all the specified model names, the closest from now is returned. If no timestamp is found, we return a naive now() instead of None.

That way, this part of the template is updated when data changes the table auth_user or yourapp_yourmodel.

The Python equivalent could be added to the API.

Easy control over cache keys

Since this is still in beta, I would like to point out 1 thing most caching libraries miss out on. Easily lets devs influence the cache key generation. This is useful for a lot of situations but mostly useful for SaaS apps that use multi-tenant (or multi-schema) architecture. Same table+id combination does not always mean the same row in such scenarios as there can be multiple schemas and in some cases even databases.

DiskCache compatibility

Are there any plans to support DiskCache? I haven't tried it yet. The docs don't seem to mention if 3rd party caches are supported or what it would take implement.

Also, the docs mention going to the Google Groups page to ask questions, but I applied for membership over a week ago and it doesn't look like I have access. Are you open to switching to Gitter or Slack?

Potential bug when using django.contrib.gis

Apologies for the vagueness here, I've looked at this a bit and plan on coming back to it, but I thought I'd write this down in case you have any ideas.

These lines in my Django site return inconsistent and incorrect results. They count the number of model instances within a geometry represented in another instance. I'm relatively sure the issue has to do with django-cachalot since invalidating the model cache before each query fixes the problem.

Each time the sql is the same, and the parameters will be something like this:

('*', 41, True, True, 'private', True, 3, <django.contrib.gis.db.backends.postgis.adapter.PostGISAdapter object at 0x7fcd2a807790>, True)

My hunch right now is that the PostGISAdapter gets SHAed to the same value sometimes as it is reused even for different geometries, so the cache key is the same for multiple queries that use different geometries.

Please let me know if there's more info I can provide, and what you might suggest I look at next.

How to monitor requests?

Is there any way to monitor how much requests django-cachalot reads from cache and how much passes to DB backend? I've just setup django-cachalot and can't really feel the difference, so I want to be sure it really works as supposed.

Using it with DatabaseCache

Is it possible to use it with DatabaseCache?

using the following config:

CACHES['cachalot'] = {
    'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
    'LOCATION': 'cachalot_cache_table',
}

CACHALOT_CACHE = 'cachalot'

CACHALOT_UNCACHABLE_TABLES = frozenset(('django_migrations', 'cachalot_cache_table'))

but I'm getting a maximum recursion depth.

Traceback:
File "***/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  223.                 response = middleware_method(request, response)
File "***/lib/python2.7/site-packages/django/contrib/sessions/middleware.py" in process_response
  50.                         request.session.save()
File "***/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in save
  56.             return self.create()
File "***/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in create
  41.                 self.save(must_create=True)
File "***/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in save
  65.                 obj.save(force_insert=must_create, using=using)
File "***/lib/python2.7/site-packages/cachalot/monkey_patch.py" in inner
  136.                 self.using, exc_type is None and not needs_rollback)
File "***/lib/python2.7/site-packages/cachalot/cache.py" in exit_atomic
  51.                 atomic_cache.commit()
File "***/lib/python2.7/site-packages/cachalot/transaction.py" in commit
  27.         self.parent_cache.set_many(self, None)
File "***/lib/python2.7/site-packages/django/core/cache/backends/base.py" in set_many
  194.             self.set(key, value, timeout=timeout, version=version)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in set
  89.         self._base_set('set', key, value, timeout)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in _base_set
  143.                                        [key, b64encoded, exp])
File "***/lib/python2.7/site-packages/cachalot/monkey_patch.py" in inner
  136.                 self.using, exc_type is None and not needs_rollback)
File "***/lib/python2.7/site-packages/cachalot/cache.py" in exit_atomic
  51.                 atomic_cache.commit()
File "***/lib/python2.7/site-packages/cachalot/transaction.py" in commit
  27.         self.parent_cache.set_many(self, None)
File "***/lib/python2.7/site-packages/django/core/cache/backends/base.py" in set_many
  194.             self.set(key, value, timeout=timeout, version=version)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in set
  89.         self._base_set('set', key, value, timeout)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in _base_set
  143.                                        [key, b64encoded, exp])
...
File "***/lib/python2.7/site-packages/cachalot/monkey_patch.py" in inner
  136.                 self.using, exc_type is None and not needs_rollback)
File "***/lib/python2.7/site-packages/cachalot/cache.py" in exit_atomic
  51.                 atomic_cache.commit()
File "***/lib/python2.7/site-packages/cachalot/transaction.py" in commit
  27.         self.parent_cache.set_many(self, None)
File "***/lib/python2.7/site-packages/django/core/cache/backends/base.py" in set_many
  194.             self.set(key, value, timeout=timeout, version=version)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in set
  89.         self._base_set('set', key, value, timeout)
File "***/lib/python2.7/site-packages/django/core/cache/backends/db.py" in _base_set
  139.                                        [b64encoded, exp, key])
File "***/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
  79.             return super(CursorDebugWrapper, self).execute(sql, params)
File "***/lib/python2.7/site-packages/cachalot/monkey_patch.py" in inner
  112.                     invalidate(*tables, db_alias=cursor.db.alias)
File "***/lib/python2.7/site-packages/cachalot/api.py" in invalidate
  65.             _get_tables(tables_or_models), cache_alias, db_alias):
File "***/lib/python2.7/site-packages/cachalot/api.py" in _cache_db_tables_iterator
  23.             tables = connections[db_alias].introspection.table_names()
File "***/lib/python2.7/site-packages/django/db/backends/base/introspection.py" in table_names
  56.             with self.connection.cursor() as cursor:
File "***/lib/python2.7/site-packages/django/db/backends/base/base.py" in cursor
  161.         if self.queries_logged:
File "***/lib/python2.7/site-packages/django/db/backends/base/base.py" in queries_logged
  76.         return self.force_debug_cursor or settings.DEBUG

Exception Type: RuntimeError at /
Exception Value: maximum recursion depth exceeded

Thanks!

Cut a new version for postgres compatibility

I don't suppose you could cut a new version with this fix:
b9f84f8

We're not able to use django 1.9.4 which prefers postgres without this.

Awesome library! The drop-in nature of it, has literally saved many projects!

invalidate_cachalot management command : BaseCommand.option_list is deprecated in Django 1.8 and removed in Django 1.10

docker-compose -f dev.yml run django python manage.py invalidate_cachalot
Traceback (most recent call last):
File "manage.py", line 18, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 208, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/local/lib/python3.5/site-packages/django/core/management/init.py", line 40, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 665, in exec_module
File "", line 222, in _call_with_frames_removed
File "/usr/local/lib/python3.5/site-packages/cachalot/management/commands/invalidate_cachalot.py", line 8, in
class Command(BaseCommand):
File "/usr/local/lib/python3.5/site-packages/cachalot/management/commands/invalidate_cachalot.py", line 11, in Command
option_list = BaseCommand.option_list + (
AttributeError: type object 'BaseCommand' has no attribute 'option_list'

Django-CMS,sqlite, cachalot, memcached , apache, - site lock

We experienced weird hangups of our test site with configuration mentioned in title.
There was 2 users making edits in Django-CMS and 4 times we had to restart application because it was locked.
Site started to serve content, but never returned anything, until restart of wsgi process. Apache was happily serving other django sites. With only one person editing there were no problems.
This locking occurred only after configuring cachalot.

Sorry I cannot be more specific on this issue.

Monkey_patch.py ascii encoding of sql in python2.7

At monkey_patch.py:110 if sql is a str type, but the actual query contains unicode characters, an error is generated:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26821804: ordinal not in range(128)

Fixed if this code is inserted before line 110:

if not isinstance(sql, unicode):
    sql = unicode(sql, 'utf-8')

monkey_patch did not read current settings

Hi, can you help me with the following:

lets say I run migrate
django_cachalot monkey_patch will run, to invalidate cache
however i received redis.exceptions.ConnectionError: Error 111 connecting to None:6379. Connection refused --> it happened because django-cachalot still looking for redis at :6379, while my application used redis through different endpoint, lets say SOME_OTHER_REDIS_URL (stated in my app settings.py)

Have been running over the place, but can't find out why the application setting did not apply (it used default settings). This behavior does not exist in normal caching operation, only when I run from management console, eg. manage.py migrate or manage.py invalidate_cachalot

Django 1.9 error

I realise django 1.8 is only just supported, but wanted to give you a heads up on this error one gets when running cachalot 1.1 with django 1.9:

  Traceback (most recent call last):
    File "./manage.py", line 12, in <module>
      execute_from_command_line(sys.argv)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/__init__.py", line 350, in execute_from_command_line
      utility.execute()
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/__init__.py", line 342, in execute
      self.fetch_command(subcommand).run_from_argv(self.argv)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/base.py", line 348, in run_from_argv
      self.execute(*args, **cmd_options)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/base.py", line 399, in execute
      output = self.handle(*args, **options)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 83, in handle
      verbosity=self.verbosity,
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/__init__.py", line 119, in call_command
      return command.execute(*args, **defaults)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/base.py", line 399, in execute
      output = self.handle(*args, **options)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/commands/showmigrations.py", line 36, in handle
      return self.show_list(connection, options['app_labels'])
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/core/management/commands/showmigrations.py", line 44, in show_list
      loader = MigrationLoader(connection, ignore_no_migrations=True)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/migrations/loader.py", line 49, in __init__
      self.build_graph()
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/migrations/loader.py", line 176, in build_graph
      self.applied_migrations = recorder.applied_migrations()
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/migrations/recorder.py", line 66, in applied_migrations
      return set(tuple(x) for x in self.migration_qs.values_list("app", "name"))
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/models/query.py", line 258, in __iter__
      self._fetch_all()
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/models/query.py", line 1074, in _fetch_all
      self._result_cache = list(self.iterator())
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/models/query.py", line 128, in __iter__
      for row in compiler.results_iter():
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 806, in results_iter
      results = self.execute_sql(MULTI)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/cachalot/monkey_patch.py", line 29, in inner
      out = original(compiler, *args, **kwargs)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/cachalot/monkey_patch.py", line 77, in inner
      table_cache_keys = _get_table_cache_keys(compiler)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/cachalot/utils.py", line 147, in _get_table_cache_keys
      tables = _get_tables(compiler.query, db_alias)
    File "/Users/foo/envs/sbox3000/lib/python3.4/site-packages/cachalot/utils.py", line 128, in _get_tables
      + query.having.children)
  AttributeError: 'Query' object has no attribute 'having'

pip install fails

Downloading/unpacking django-cachalot
  Downloading django-cachalot-0.4.0.tar.gz
  Running setup.py (path:/home/hp/.virtualenvs/project/build/django-cachalot/setup.py) egg_info for package django-cachalot
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/hp/.virtualenvs/project/build/django-cachalot/setup.py", line 10, in <module>
        with open(os.path.join(CURRENT_PATH, 'requirements.txt')) as f:
    IOError: [Errno 2] No such file or directory: '/home/hp/.virtualenvs/project/build/django-cachalot/requirements.txt'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/home/hp/.virtualenvs/project/build/django-cachalot/setup.py", line 10, in <module>

    with open(os.path.join(CURRENT_PATH, 'requirements.txt')) as f:

IOError: [Errno 2] No such file or directory: '/home/hp/.virtualenvs/project/build/django-cachalot/requirements.txt'

SELECT FOR UPDATE should never be cached

The SELECT FOR UPDATE mechanism is negated by cachalot because it caches the results of queries that include SELECT FOR UPDATE. This results in unwanted concurrent behavior in systems that rely on the exclusive locking of rows. These bugs are very hard to reproduce but can and do occur.

According to postgres documentation (https://www.postgresql.org/docs/9.3/static/explicit-locking.html):

To acquire an exclusive row-level lock on a row without actually modifying the row, select the row with SELECT FOR UPDATE. Note that once the row-level lock is acquired, the transaction can update the row multiple times without fear of conflicts.

Proper behaviour should be to let all SELECT FOR UPDATE queries through to the database. The testcase in the following link should fail (it should produce a single query, instead of using the cached result):

https://github.com/BertrandBordage/django-cachalot/blob/7e3137eac7bc1b4f683a336579653623e71c991e/cachalot/tests/read.py#L539

A temporary workaround for now is to set the CACHALOT_QUERY_KEYGEN method, and to ignore queries containing select for updates:

def get_query_cache_key(compiler):
    if compiler.query.select_for_update:
        raise UncachableQuery
    return original_query_keygen(compiler)

CACHALOT_ENABLED=False doesn't seem to disable all aspects of the cache

I have a project using celery. For local development I use LocMemCache and also disable cachalot via CACHALOT_ENABLED = False, as the docs say it won't work well in this scenario (plus I don't generally want it for development anyway).

The problem is, even with cachalot disabled I keep running into issues with stale data for models written to from celery tasks and read by the main process. If I add the table to CACHALOT_UNCACHABLE_TABLES the issue disappears.

So it would seem that CACHALOT_ENABLED doesn't control all cache-related functions.

Cache invalidation problems with asynchronous tasks

I am using celery with django for some asynchronous tasks but now I find that when table row is updates by a celery worker process cachalot is not invalidating the cache

pyodbc support

cachalot clearly states on startup that pyodbc is not supported.

We're relying on a third party DB over a VPN connection, for which can only use pyodbc. The data is static and read-only, so cachalot would really be useful in this scenario. I'd be willing to spend some time figuring out what to do to make it supported, at least in our scenario. But in the comments of #15, you state that it will never be supported. Could you explain the reasoning behind that? Is there something fundamentally uncachable about pyodbc? Or are you just not interested in supporting pyodbc?

AttributeError: 'Query' object has no attribute 'having'

Hi, I'm trying to setup cachalot (1.1.0 installed with pip) with a Django site running Django 1.9 + SQLite and I get this error whenever I try to request a page:

Unhandled exception in thread started by <function wrapper at 0x10bbf7668>
Traceback (most recent call last):
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/utils/autoreload.py", line 226, in wrapper
fn(_args, *_kwargs)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 117, in inner_run
self.check_migrations()
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 163, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/migrations/executor.py", line 20, in init
self.loader = MigrationLoader(self.connection)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/migrations/loader.py", line 49, in init
self.build_graph()
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/migrations/loader.py", line 176, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 66, in applied_migrations
return set(tuple(x) for x in self.migration_qs.values_list("app", "name"))
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/models/query.py", line 258, in iter
self._fetch_all()
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
self._result_cache = list(self.iterator())
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/models/query.py", line 128, in iter
for row in compiler.results_iter():
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 806, in results_iter
results = self.execute_sql(MULTI)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 29, in inner
out = original(compiler, _args, *_kwargs)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 77, in inner
table_cache_keys = _get_table_cache_keys(compiler)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/cachalot/utils.py", line 147, in _get_table_cache_keys
tables = _get_tables(compiler.query, db_alias)
File "/Users/tomasandrle/Sites/SilverRocket/env/lib/python2.7/site-packages/cachalot/utils.py", line 128, in _get_tables
+ query.having.children)
AttributeError: 'Query' object has no attribute 'having'

what does this mean?

Ability to set a prefix

This is not an issue - but an enhancement request.

Was wondering if there could a setting which allows to set a cache key prefix? It would help in debugging, etc.. (well to a certain extent at least)

Add a whitelist

There is a blacklist of tables that shouldn't be cached, can we get a whitelist as well?

Profiling Django-Cachalot

I've just implemented Django-Cachalot for my Heroku production app, and everything has been running very smoothly. Thank you for the great work!

I recently took a look at our memcached metrics and noticed that the datastore size has been steadily growing since we launched the package (currently at 20GB, 3.5x the size our actual Postgres DB, with no signs of slowing down).

Given the cost of running a memcached server, we need a way to profile our app under Django-cachalot to determine whether the cost of keeping this data in cache is worth the gains in query speed.

We decided the important profiling metrics for our application were:

Total # of Cache Hits (both in aggregate and by model)
Total # of Cache Misses (both in aggregate and by model)
Average storage size of cached objects (both in aggregate and by model)

Here is the first draft of our monkey patch of Django-Cachalot that we'll be temporarily deploying to profile the current cache's effectiveness: https://gist.github.com/boxbeatsy/87bdbc8f997872c77856c4d8cc8638f4

Here is a reporting script: https://gist.github.com/boxbeatsy/fda5cfe1eb5af03d5289b63f0bd339fd

Does anyone have additional guidance on how to better profile our app? Would this be something worth adding to Django-Cachalot? (I imagine profiling would be an important operational question for anyone deploying this in production at scale).

Invalidation of data stored in a primary/replica configured DB invalidates only for primary instance

I have a Django app with two MariaDB databases configured as primary/replica with the following configuration:

DATABASE_ROUTERS = ['app.db_replica.PrimaryReplicaRouter', ]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'app',
        'USER': 'app',
        'PASSWORD': '*******************',
        'HOST': 'mysql-master',
    },
    'replica1': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'app',
        'USER': 'app',
        'PASSWORD': '*******************',
        'HOST': 'mysql-slave',
    },
}

The router is configured to write on primary and read on replica:

class PrimaryReplicaRouter(object):
    def db_for_read(self, model, **hints):
        return 'replica1'

    def db_for_write(self, model, **hints):
        return 'default'

    def allow_relation(self, obj1, obj2, **hints):
        db_list = ('default', 'replica1')
        if obj1._state.db in db_list and obj2._state.db in db_list:
            return True
        return None

    def allow_migrate(self, db, app_label, model=None, **hints):
        return True

When Django writes on a table, cache gets invalidated only on primary.
You can test this by configuring the router to randomly return the primary or the replica, you will see old and new value alternated refreshing the page.

Calling ./manage.py invalidate_cachalot works, I think because it invalidates all the cache ignoring the database instance.

Returning stale data after object reload

I seem to be running into a few cases where I'm getting stale data returned. One is as follows:

Using Plata (an e-commerce module for Django), there's an OrderPayment model with an FK to the Order model. There comes a time during payment processing when the following occurs:

payment.save()
order = order.reload()

order.reload() reloads the object from the database using the existing id.

There seems to be quite frequent cases where the reloaded order does not acknowledge the newly created payment, and so thinks the order is not yet paid. Disabling caching for these two tables seems to be solve the issue.

Is there a known issue with invalidation in this kind of situation?

django.db.models.loading.get_models will be removed in Django 1.9

django.db.models.loading.get_models will be removed in Django 1.9. In my Django 1.8.6 installation, the following warning is thrown:

/home/xxx/.virtualenvs/xxx/local/lib/python2.7/site-packages/cachalot/panels.py:10: RemovedInDjango19Warning: The utilities in django.db.models.loading are deprecated in favor of the new application loading system.
  from django.db.models.loading import get_models

django.apps.apps.get_models is the equivalent function in Django 1.7+.

>> from django.db.models import loading
>> from django.apps import apps
>> apps.get_models() == loading.get_models()
True

Unable to run the tests without pytz

I don't know if you are aware and it's something done on purpose but if you clone the repository and try to run the tests you get a misleading error that says that cachalot has not module tests.

That's because when importint cachalot.tests there an error that the pytz packet is missing. It would be nice to have it in the file runtests_requirements_py2.txt.

Thanks

default redis maxmemory-policy considered harmful

Please specify somewhere in the documentation that if redis is used as the caching backend, its "maxmemory-policy" settings should be changed to "allkeys-lru" otherwise you risk OOM exceptions bringing the site down.

(the default "volatile-lru" will not remove the keys set by cachalot without an expiry interval)

Unable to disable cache on a per-database basis.

I currently have a project with two database connections; a PostgreSQL database for all the usual Django things and an external MySQL data we're using to pull data from. If objects get deleted from the external MySQL database using that provider's tools (not though Django) django-cachalot (obviously) doesn't know that's happened and continues to return data which no longer exists (and messes up some of the Django logic). What would be great is if we could disable cachalot on the MySQL database, but keep it on the Django one - is this possible?

RemovedInDjango19Warning

When running django-cachalot with Django 1.8 ./manage.py test:

Creating test database for alias 'default'...
/home/socketubs/.virtualenvs/bounty-o-matic/lib/python3.4/site-packages/cachalot/cache.py:34: RemovedInDjango19Warning: 'get_cache' is deprecated in favor of 'caches'.
    return get_django_cache(cache_alias)

Cache not being invalidated under a specific server config

I have two servers running, one is the main application server, the other is a task runner. Each server has it's own memcached instance for storing data related to it's own tasks, but both servers talk to the same database.

If I add data to the database in the task runner, it doesn't appear in the Django admin on the app server until the cache is cleared - I assume because the task runner is clearing it's own memcached data, not that of the application server.

What's the best fix for this? I can't pool the memcache instances together, as the task runner is at 100% CPU, 100% of the time, so I don't want to risk slow gets on the application from it being in the pool.

Is my best option for my task runner server to just use the memcached instance of the application and not maintain it's own?

Support for pre 1.7 not correct

I get the following after applying a migration in 1.6:

db_alias = kwargs['using'] if django_version >= (1, 7) else kwargs['db']
KeyError: u'db'

Docs benchmarks should state which versions were used

Hi,
I've had a look at the benchmarks but this is unclear which versions of SQLite, MySQL and PostgreSQL were used. Postgres seems slow compared to the others, thus I kept wondering "is this 8.4 ?"

Python2.7 compatibility

I am running a dummy test (django TestCase with just pass) with tox. It works on python34 but fails with python27.

I got this error:

File "manage.py", line 10, in <module>
  execute_from_command_line(sys.argv)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
  utility.execute()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
  super(Command, self).run_from_argv(argv)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv
  self.execute(*args, **cmd_options)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/commands/test.py", line 74, in execute
  super(Command, self).execute(*args, **options)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
  output = self.handle(*args, **options)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/commands/test.py", line 90, in handle
  failures = test_runner.run_tests(test_labels)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django_nose/runner.py", line 403, in run_tests
  result = self.run_suite(nose_argv)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django_nose/runner.py", line 335, in run_suite
  addplugins=plugins_to_add)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/nose/core.py", line 121, in __init__
  **extra_args)
File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__
  self.runTests()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/nose/core.py", line 207, in runTests
  result = self.testRunner.run(self.test)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/nose/core.py", line 50, in run
  wrapper = self.config.plugins.prepareTest(test)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
  return self.call(*arg, **kw)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
  result = meth(*arg, **kw)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django_nose/plugin.py", line 82, in prepareTest
  self.old_names = self.runner.setup_databases()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django_nose/runner.py", line 590, in setup_databases
  return super(NoseTestSuiteRunner, self).setup_databases()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/test/runner.py", line 166, in setup_databases
  **kwargs
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/test/runner.py", line 370, in setup_databases
  serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/backends/base/creation.py", line 368, in create_test_db
  test_flush=not keepdb,
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
  return command.execute(*args, **defaults)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
  output = self.handle(*args, **options)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 225, in handle
  emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/core/management/sql.py", line 280, in emit_post_migrate_signal
  using=db)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 201, in send
  response = receiver(signal=self, sender=sender, **named)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/contrib/auth/management/__init__.py", line 82, in create_permissions
  ctype = ContentType.objects.db_manager(using).get_for_model(klass)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/contrib/contenttypes/models.py", line 65, in get_for_model
  ct = self.get(app_label=opts.app_label, model=opts.model_name)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
  num = len(clone)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
  self._fetch_all()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
  self._result_cache = list(self.iterator())
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
  results = compiler.execute_sql()
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 29, in inner
  out = original(compiler, *args, **kwargs)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 85, in inner
  cache_key, table_cache_keys)
File "/home/choclate/documents/projects/web/myproject/.tox/py27/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 53, in _get_result_or_execute_query
  timestamp, result = data.pop(cache_key)
TypeError: 'NoneType' object is not iterable

I use memcached. I think test is only setting up the database, so I don't understand why cachalot code is executed. _get_result_or_execute_query is not executed with python34.

Without setting cachalot in django app, test works with python27.

Any ideas?

"wrong number of arguments for 'mset' command" when using redis

Hi,

When using redis and Dango 1.8, I've got the following exception : "wrong number of arguments for 'mset' command"

The stack trace shows that in cachalot/transaction.py line 34:

_invalidate_table_cache_keys(self.parent_cache, self.to_be_invalidated)

self.to_be_invalidated is []

which makes line 133 of cachalot/utils.py

cache.set_many(d, None)

raise an exception because d is none and Redis does not support "mset" with no arguments.

Any ideas before I try to investigate further ?

tiny documentation error

Only just because I literally just had to deal with this issue on my server, I thought I'd point out: in the part of your documentation about memcached size limit issues, the limit per cache key command actually requires a unit. Setting the max cache size to 10MB would be -I 10m, not -I 10. It's weirdly inconsistent, because the units are not used with the overall max memory (the -m 1000 to set the overall max it 1GB is correct), but there you have it.

This seemed like too minor a thing to make a PR over, but I wanted to give you a heads up. Otherwise, thanks for the great work.

Huge SQL query reaches memcached size limit per key

First, I'm using Django 1.8, Python 3.4.3, Postgres 9.3.x, and memcached 1.4.4 with pylibmc.

I have a form on my site that has a jQuery autocomplete box. This is used to for selecting locations (we have roughly ~13k locations in our database - continents, countries, states, and cities). Here's the view:

def location_query(request):
    # first handle the location autocomplete
    if request.is_ajax():
        term = request.GET['term']

        # I want to explicitly order matching countries at the front of the list
        matching_countries = Location.get_countries().filter(full_name__icontains=term)
        matching_states = Location.get_states().filter(full_name__icontains=term)
        matching_cities = Location.get_cities().filter(full_name__icontains=term)

        matching_locations = list()
        matching_locations.extend(matching_countries)
        matching_locations.extend(matching_states)
        matching_locations.extend(matching_cities)

        locations_json = list()
        for matching_location in matching_locations[:10]:
            location_json = dict()
            location_json['id'] = matching_location.pk
            location_json['label'] = '%s (%s)' % (matching_location.full_name, matching_location.admin_level)
            location_json['value'] = matching_location.pk
            locations_json.append(location_json)

        return JsonResponse(locations_json, safe=False)

And here's the Location model:

class Location(models.Model):
    name = models.CharField(max_length=255)
    full_name = models.CharField(max_length=255, blank=True)  # the name might be "Paris", but full_name would be "Paris, Texas, United States of America"; allowed to be blank only because the script that populates this table will fill it in after all locations are added
    imported_from = models.CharField(max_length=255)
    admin_level = models.CharField(max_length=255, blank=True)
    geometry = models.MultiPolygonField(blank=True, default=None, null=True)
    objects = models.GeoManager()  # override the default manager with a GeoManager instance
    parent = models.ForeignKey('self', blank=True, default=None, null=True)

    def __str__(self):
        return self.full_name

    def get_full_name(self, include_continent=False):
        """
            Get the full name of a location. This includes the entire hierarchy, optionally including the continent.
                e.g., Paris, Texas, United States of America
        """
        full_name = self.name
        current_parent = self.parent
        while current_parent is not None and (include_continent or (not include_continent and current_parent.parent is not None)):
            full_name += ', ' + current_parent.name
            current_parent = current_parent.parent
        return full_name

    def get_country(self):
        if self.admin_level == 'Country':
            return self.name
        return self.parent.get_country()

    @staticmethod
    def get_continents():
        return Location.objects.filter(parent=None).order_by('name')

    @staticmethod
    def get_countries(continent=None):
        if continent:
            # return a single continent's countries, sorted
            return Location.objects.filter(parent=continent).order_by('name')
        else:
            # return all countries, sorted
            return Location.objects.filter(admin_level='Country').order_by('name')

    @staticmethod
    def get_states(country=None):
        if country:
            # return a single country's states, sorted
            return Location.objects.filter(parent=country).order_by('name')
        else:
            # return all states, sorted
            return Location.objects.filter(admin_level='State').order_by('name')

    @staticmethod
    def get_cities(state=None):
        if state:
            # return a single state's cities, sorted
            return Location.objects.filter(parent=state).order_by('name')
        else:
            # return all cities, sorted
            return Location.objects.filter(admin_level='City').order_by('name')

    @staticmethod
    def get_non_continents():
        return Location.objects.exclude(parent=None).order_by('name')

    class Meta:
        ordering = ['full_name']

When I disable cachalot by commenting out the line in INSTALLED_APPS, the autocomplete works. When I enable it, it doesn't work. Other things on my site do indeed work, and the DDT panel shows that cachalot is doing its job. Can it deal with ajax calls like this?

TypeError: 'NoneType' object is not iterable

running django-cachalot with Django 1.8, memcached and ElastiCache

File "/opt/python/run/venv/lib/python2.7/site-packages/cachalot/monkey_patch.py", line 57, in _get_result_or_execute_query
new_table_cache_keys.difference_update(data)
TypeError: 'NoneType' object is not iterable

Django 1.8 support

Just wanted to bring some clarity whether django-cachalot in it's current state supports Django 1.8 or not. I tried running the package on Django 1.8, and while it's not crashing anything, I see that django1.8 branch hasn't been merged to master and is failing the build. So maybe some update for the community about this would be good from you. Thanks.

noripyt / django-cachalot Goto Github PK

django-cachalot's Introduction

Django Cachalot

Quickstart

Usage

Hacking

Benchmark

Third-Party Cache Comparison

Discussion

django-cachalot's People

Contributors

Stargazers

Watchers

Forkers

django-cachalot's Issues

Recommend Projects

Recommend Topics

Recommend Org