heroku-python / django-postgrespool Goto Github PK

https://github.com/andymckay/django-mysql-pool

Looking at the code, it seems like db_pool is created at import to properly proxy to the sqlalchemy pool manager as expected.

However, as of the changeset in https://github.com/kennethreitz/django-postgrespool/pull/21 it seems like the db_pool is no longer used to connect to the database and it should fallback to the default django psycopg2 drivers.

The only reference to the db_pool object that should do the pooling magic is in the _dispose method. So it seems like this is a simple wrapper around the default Django backend.

Perhaps I am missing some DB-API knowledge or Django magic though?

It would be nice to have benchmarks on the kind of performance boost gained by django-postgrespool, even if a very unscientific benchmark, just to get a general feel.

users of pypy use psycopg2cffi instead of psycopg2, but maybe that's not all the changes needed.

I am getting:

Traceback (most recent call last):
...
raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
'dummy', 'mysql', 'oracle', 'postgresql_psycopg2', 'sqlite3'
Error was: cannot import name Diagnostics

What solved my problem was to install psycopg2-2.4.6 instead of psycopg2-2.5.1 (both from http://www.stickpeople.com/projects/python/win-psycopg/).

I tried the advice in issue https://github.com/kennethreitz/django-postgrespool/issues/6 and it didn't help.

I am using Windows Python 2.7.3 AMD64.

Hi,

I was having a autocommit cannot be used inside a transaction error.
It was hard to find the reason but when I removed django_postgrespool (`#DATABASES['default']['ENGINE'] = 'django_postgrespool'``), the error was finally more explicit about an integrity error. I was able to fix it and the error was gone.

Why the error is not the actual error when using django_postgrespool? Isn't it possible to raise the message of the real error?

Thanks

got this error below, i believe it is because i am using windows and use chinese as my system locale. it would be better to save the readme.rst in utf8, which will solve the problem

Collecting django-postgrespool
Using cached django-postgrespool-0.3.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "C:\Users\zyuan\AppData\Local\Temp\pip-build-hixoxdz4\django-postgrespool\setup.py", line 26, in
long_description=open('README.rst').read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0x93 in position 1202: illegal multibyte sequence

It was renamed from django.db.backends.postgresql_psycopg2 to django.db.backends.postgresql in Django Ticket #25175 and there's an ongoing discussion about deprecating/removing the old name. Better to be future proof!

EDIT: this was an issue trying to run on Django < 1.4. Closing.

I've installed the package via pip and can do an import at the command line, but when I try to run the Django server, I get:

django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using django.db.backends.XXX, where XXX is one of:
'dummy', 'mysql', 'oracle', 'postgresql', 'postgresql_psycopg2', 'sqlite3'

The site is using two databases and I've probably got some messed-up setting, but I can't figure out why it can't see the option. My settings.py looks like this:

DATABASES = {
    'default': {
        'HOST': 'localhost',
        'ENGINE': 'django_postgrespool',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'pass'
    },
    'readonly': {
        'HOST': 'localhost',
        'ENGINE': 'django_postgrespool',
        'NAME': 'db',
        'USER': 'postgres',
        'PASSWORD': 'pass'
    }
}

If you create a South migration that is dealing with a Geo-enabled Model and then subsequently apply the migration, any GIS fields will be left out - presumably since the standard Postgres driver is being specified.

A solution is to specify an alternative settings that uses the postgis driver and doesn't include the SOUTH_DATABASE_ADAPTERS setting. This isn't a code problem with this library per se, but other devs might be appreciative if the docs could include some notes about this.

related #7

I have django-postgrespool as a dependency in my Django application. When I try to install it via pip, I get hit with this error:

Downloading from URL https://pypi.python.org/packages/8e/cd/0e9b7c297a84caf6334d1d
bd9da35af50d533f1230b96bc21223d5addc1e/django-postgrespool-0.3.0.tar.gz#md5=39042504
28523e76f057113d07405498 (from https://pypi.python.org/simple/django-postgrespool/)
Running setup.py (path:/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py) 
egg_info for package django-postgrespool
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py", line 26, in <module>
    long_description=open('README.rst').read(),
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1200: ordinal not in     range(128)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "<string>", line 17, in <module>

File "/tmp/pip_build_gitlab-runner/django-postgrespool/setup.py", line 26, in <module>

long_description=open('README.rst').read(),

File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1200: ordinal not in range(128)

I am installing on an Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-76-generic x86_64) machine with python3

I followed all of the steps in the README to configure Django to use this library for pooling but when monitoring the connections I couldn't see the pooling behavior.

After some debugging, I couldn't see anywhere that was using db_pool to get connections so I patched base.py and overrode get_new_connection on DatabaseWrapper as follows:

def get_new_connection(self, conn_params):
        return db_pool.connect(**conn_params)

With this patch it seems the pooling is working as expected. I can see the connections being reused and the checkout|checkin|connect log messages that I couldn't before.

Has something changed in Django that could have broken this (I couldn't see anything with a quick look through history)?

This has the same syndrom than issue #17 but here the problem is clear:

Configuring the DB backend to postgrespool raises the following error:

django.core.exceptions.ImproperlyConfigured: 'django_postgrespool' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
u'mysql', u'oracle', u'postgresql_psycopg2', u'sqlite3'
Error was: cannot import name CursorWrapper

Indeed, with Django 1.6 (still in beta), the following line will crash
from django.db.backends.postgresql_psycopg2.base import CursorWrapper

There isn't a is_usable() function into the Psycopg2DatabaseWrapper/BaseDatabaseWrapper classes from Django 1.4/1.5 .

Thank you for providing this library. It's working great so far on a Heroku deployment, providing a noticeable boost for some loads.

We needed to support the postgis driver and it appears to work fine when specifying that module: https://github.com/lincolnloop/django-postgrespool/blob/master/django_postgrespool/base.py#L11-L14

Could the library support an optional setting for specifying the driver module? This way you could theoretically support any Django-compatible Postgres driver.

Open to suggestions, thanks.

When running Celery jobs, there are frequent (but not entirely consistent) exceptions being thrown from the celery jobs.

The most common exception is thrown from libpq with no message:. Here is an example stack trace:

Task main.tasks.scheduled_messages[3768b5f2-ac3e-4e6c-83ad-5b842861d6e7] raised exception: DatabaseError('error with no message from the libpq',)
Traceback (most recent call last):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/celery/task/trace.py", line 224, in trace_task
R = retval = fun(_args, *_kwargs)
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/celery/task/trace.py", line 403, in protected_call
return self.run(_args, *_kwargs)
File "/Users/sendhub-dev/dev/sendhub/project/app/main/tasks.py", line 62, in scheduled_messages
interactive=False
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/core/management/init.py", line 150, in call_command
return klass.execute(_args, *_defaults)
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(_args, *_options)
File "/Users/sendhub-dev/dev/sendhub/project/app/main/management/commands/send_scheduled_messages.py", line 21, in handle
for message in UserMessage.objects.pending().filter(scheduled_at__lte=datetime.utcnow(), sent__isnull=True):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 118, in _result_iter
self._fill_cache()
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 875, in _fill_cache
self._result_cache.append(self._iter.next())
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/query.py", line 291, in iterator
for row in compiler.results_iter():
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 763, in results_iter
for rows in self.execute_sql(MULTI):
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 817, in execute_sql
cursor = self.connection.cursor()
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django/db/backends/init.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/Users/sendhub-dev/dev/sendhub/project/venv/lib/python2.7/site-packages/django_postgrespool/base.py", line 78, in _cursor
self.connection.set_isolation_level(self.isolation_level)
DatabaseError: error with no message from the libpq

When we use streaming responses in Django, or the static MEDIA (static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) in url patterns) serving in development mode, the "request_finished" signal is never sent at the end of the request.

Django uses this signal to clean up db connections and return them to the pool, which means these connections leak.