heroku-python / django-postgrespool Goto Github PK

View Code? Open in Web Editor NEW

323.0 10.0 66.0 30 KB

[DEPRECATED] Use https://pypi.org/project/django-postgrespool2/ instead

License: MIT License

Python 100.00%

django-postgrespool's Introduction

Django-PostgresPool

This is a simple Postgres Connection Pooling backend for Django 1.4+, powered by the lovely and beautiful SQLAlchemy.

Usage

Using Django-PostgresPool is simple, just set django_postgrespool as your connection engine:

DATABASES = {
    'default': {
        'ENGINE': 'django_postgrespool'

If you're using the dj-database-url module:

import dj_database_url

DATABASES = {'default': dj_database_url.config(engine='django_postgrespool')}

If you're using south:

SOUTH_DATABASE_ADAPTERS = {
    'default': 'south.db.postgresql_psycopg2'
}

Everything should work as expected.

Installation

Installing Django-PostgresPool is simple, with pip:

$ pip install django-postgrespool

Configuration

Optionally, you can provide additional options to pass to SQLAlchemy's pool creation:

DATABASE_POOL_ARGS = {
    'max_overflow': 10,
    'pool_size': 5,
    'recycle': 300
}

Here's a basic explanation of two of these options:

pool_size – The minimum number of connections to maintain in the pool.
max_overflow – The maximum overflow size of the pool. This is not the maximum size of the pool.

The total number of "sleeping" connections the pool will allow is pool_size. The total simultaneous connections the pool will allow is pool_size + max_overflow.

As an example, databases in the Heroku Postgres starter tier have a maximum connection limit of 20. In that case your pool_size and max_overflow, when combined, should not exceed 20.

Check out the official SQLAlchemy Connection Pooling docs to learn more about the optoins that can be defined in DATABASE_POOL_ARGS.

Django 1.3 Support

django-postgrespool currently supports Django 1.4 and greater. See this ticket for 1.3 support.

django-postgrespool's People

Contributors

Stargazers

Watchers

django-postgrespool's Issues

SOUTH_DATABASE_ADAPTERS setting leaves GIS fields out of migrations

If you create a South migration that is dealing with a Geo-enabled Model and then subsequently apply the migration, any GIS fields will be left out - presumably since the standard Postgres driver is being specified.

A solution is to specify an alternative settings that uses the postgis driver and doesn't include the SOUTH_DATABASE_ADAPTERS setting. This isn't a code problem with this library per se, but other devs might be appreciative if the docs could include some notes about this.

related #7

Support for postgis (or arbitrary) driver?

Thank you for providing this library. It's working great so far on a Heroku deployment, providing a noticeable boost for some loads.

We needed to support the postgis driver and it appears to work fine when specifying that module: https://github.com/lincolnloop/django-postgrespool/blob/master/django_postgrespool/base.py#L11-L14

Could the library support an optional setting for specifying the driver module? This way you could theoretically support any Django-compatible Postgres driver.

Open to suggestions, thanks.

Celery jobs throw exceptions

When running Celery jobs, there are frequent (but not entirely consistent) exceptions being thrown from the celery jobs.

The most common exception is thrown from libpq with no message:. Here is an example stack trace:

Task main.tasks.scheduled_messages[3768b5f2-ac3e-4e6c-83ad-5b842861d6e7] raised exception: DatabaseError('error with no message from the libpq',)
Traceback (most recent call last):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/celery/task/trace.py", line 224, in trace_task
R = retval = fun(_args, *_kwargs)
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/celery/task/trace.py", line 403, in protected_call
return self.run(_args, *_kwargs)
File "/Users/sendhub-dev/dev/sendhub/project/app/main/tasks.py", line 62, in scheduled_messages
interactive=False
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/core/management/init.py", line 150, in call_command
return klass.execute(_args, *_defaults)
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(_args, *_options)
File "/Users/sendhub-dev/dev/sendhub/project/app/main/management/commands/send_scheduled_messages.py", line 21, in handle
for message in UserMessage.objects.pending().filter(scheduled_at__lte=datetime.utcnow(), sent__isnull=True):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 118, in _result_iter
self._fill_cache()
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 875, in _fill_cache
self._result_cache.append(self._iter.next())
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 291, in iterator
for row in compiler.results_iter():
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 763, in results_iter
for rows in self.execute_sql(MULTI):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 817, in execute_sql
cursor = self.connection.cursor()
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/backends/init.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django_postgrespool/base.py", line 78, in _cursor
self.connection.set_isolation_level(self.isolation_level)
DatabaseError: error with no message from the libpq

Does this project actually work?

Looking at the code, it seems like db_pool is created at import to properly proxy to the sqlalchemy pool manager as expected.

However, as of the changeset in https://github.com/kennethreitz/django-postgrespool/pull/21 it seems like the db_pool is no longer used to connect to the database and it should fallback to the default django psycopg2 drivers.

The only reference to the db_pool object that should do the pooling magic is in the _dispose method. So it seems like this is a simple wrapper around the default Django backend.

Perhaps I am missing some DB-API knowledge or Django magic though?

Pooling not working with Django 1.7

I followed all of the steps in the README to configure Django to use this library for pooling but when monitoring the connections I couldn't see the pooling behavior.

After some debugging, I couldn't see anywhere that was using db_pool to get connections so I patched base.py and overrode get_new_connection on DatabaseWrapper as follows:

def get_new_connection(self, conn_params):
        return db_pool.connect(**conn_params)

With this patch it seems the pooling is working as expected. I can see the connections being reused and the checkout|checkin|connect log messages that I couldn't before.

Has something changed in Django that could have broken this (I couldn't see anything with a quick look through history)?

support for pypy

users of pypy use psycopg2cffi instead of psycopg2, but maybe that's not all the changes needed.

UnicodeDecodeError while installing on Ubuntu Linux

I have django-postgrespool as a dependency in my Django application. When I try to install it via pip, I get hit with this error:

Downloading from URL https://pypi.python.org/packages/8e/cd/0e9b7c297a84caf6334d1d
bd9da35af50d533f1230b96bc21223d5addc1e/django-postgrespool-0.3.0.tar.gz#md5=39042504
28523e76f057113d07405498 (from https://pypi.python.org/simple/django-postgrespool/)
Running setup.py (path:/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py) 
egg_info for package django-postgrespool
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py", line 26, in <module>
    long_description=open('README.rst').read(),
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1200: ordinal not in     range(128)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "<string>", line 17, in <module>

File "/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py", line 26, in <module>

long_description=open('README.rst').read(),

File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1200: ordinal not in range(128)

I am installing on an Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-76-generic x86_64) machine with python3

Benchmarks

It would be nice to have benchmarks on the kind of performance boost gained by django-postgrespool, even if a very unscientific benchmark, just to get a general feel.

"autocommit cannot be used inside a transaction" error "hides" the actual error?

Hi,

I was having a autocommit cannot be used inside a transaction error.
It was hard to find the reason but when I removed django_postgrespool (`#DATABASES['default']['ENGINE'] = 'django_postgrespool'``), the error was finally more explicit about an integrity error. I was able to fix it and the error was gone.

Why the error is not the actual error when using django_postgrespool? Isn't it possible to raise the message of the real error?

Thanks

fail to install using pip on windows

got this error below, i believe it is because i am using windows and use chinese as my system locale. it would be better to save the readme.rst in utf8, which will solve the problem

Collecting django-postgrespool
Using cached django-postgrespool-0.3.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "C:\Users\zyuan\AppData\Local\Temp\pip-build-hixoxdz4\django-postgrespool\setup.py", line 26, in
long_description=open('README.rst').read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0x93 in position 1202: illegal multibyte sequence

'django_postgrespool' isn't an available database backend

I am getting:

Traceback (most recent call last):
...
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
'dummy', 'mysql', 'oracle', 'postgresql_psycopg2', 'sqlite3'
Error was: cannot import name Diagnostics

What solved my problem was to install psycopg2-2.4.6 instead of psycopg2-2.5.1 (both from http://www.stickpeople.com/projects/python/win-psycopg/).

I tried the advice in issue https://github.com/kennethreitz/django-postgrespool/issues/6 and it didn't help.

I am using Windows Python 2.7.3 AMD64.

The current version doesn't support Django <1.6

There isn't a is_usable() function into the Psycopg2DatabaseWrapper/BaseDatabaseWrapper classes from Django 1.4/1.5 .

Connection leak with StreamingResponse

When we use streaming responses in Django, or the static MEDIA (static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) in url patterns) serving in development mode, the "request_finished" signal is never sent at the end of the request.

Django uses this signal to clean up db connections and return them to the pool, which means these connections leak.

'django_postgrespool' isn't an available database backend

EDIT: this was an issue trying to run on Django < 1.4. Closing.

I've installed the package via pip and can do an import at the command line, but when I try to run the Django server, I get:

django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3'

The site is using two databases and I've probably got some messed-up setting, but I can't figure out why it can't see the option. My settings.py looks like this:

DATABASES = {
    'default': {
        'HOST': 'localhost',
        'ENGINE': 'django_postgrespool',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'pass'
    },
    'readonly': {
        'HOST': 'localhost',
        'ENGINE': 'django_postgrespool',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'pass'
    }
}

DATABASE_POOL_ARGS

https://github.com/andymckay/django-mysql-pool

postgrespool broken with Django 1.6 ('django_postgrespool' isn't an available database backend)

This has the same syndrom than issue #17 but here the problem is clear:

Configuring the DB backend to postgrespool raises the following error:

django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
u'mysql', u'oracle', u'postgresql_psycopg2', u'sqlite3'
Error was: cannot import name CursorWrapper

Indeed, with Django 1.6 (still in beta), the following line will crash
from django.db.backends.postgresql_psycopg2.base import CursorWrapper

Use Django 1.9+ location for postgresql backend

It was renamed from django.db.backends.postgresql_psycopg2 to django.db.backends.postgresql in Django Ticket #25175 and there's an ongoing discussion about deprecating/removing the old name. Better to be future proof!