Git Product home page Git Product logo

testing.postgresql's Introduction

About

testing.postgresql automatically setups a postgresql instance in a temporary directory, and destroys it after testing.

https://travis-ci.org/tk0miya/testing.postgresql.svg?branch=master https://coveralls.io/repos/tk0miya/testing.postgresql/badge.png?branch=master
Documentation
https://github.com/tk0miya/testing.postgresql
Issues
https://github.com/tk0miya/testing.postgresql/issues
Download
https://pypi.python.org/pypi/testing.postgresql

Install

Use pip:

$ pip install testing.postgresql

And testing.postgresql requires PostgreSQL server in your PATH.

Usage

Create PostgreSQL instance using testing.postgresql.Postgresql:

import testing.postgresql
from sqlalchemy import create_engine

# Lanuch new PostgreSQL server
with testing.postgresql.Postgresql() as postgresql:
    # connect to PostgreSQL
    engine = create_engine(postgresql.url())

    # if you use postgresql or other drivers:
    #   import psycopg2
    #   db = psycopg2.connect(**postgresql.dsn())

    #
    # do any tests using PostgreSQL...
    #

# PostgreSQL server is terminated here

testing.postgresql.Postgresql executes initdb and postgres on instantiation. On deleting Postgresql object, it terminates PostgreSQL instance and removes temporary directory.

If you want a database including tables and any fixtures for your apps, use copy_data_from keyword:

# uses a copy of specified data directory of PostgreSQL.
postgresql = testing.postgresql.Postgresql(copy_data_from='/path/to/your/database')

For example, you can setup new PostgreSQL server for each testcases on setUp() method:

import unittest
import testing.postgresql

class MyTestCase(unittest.TestCase):
    def setUp(self):
        self.postgresql = testing.postgresql.Postgresql()

    def tearDown(self):
        self.postgresql.stop()

To make your tests faster

testing.postgresql.Postgresql invokes initdb command on every instantiation. That is very simple. But, in many cases, it is very waste that generating brandnew database for each testcase.

To optimize the behavior, use testing.postgresql.PostgresqlFactory. The factory class is able to cache the generated database beyond the testcases, and it reduces the number of invocation of initdb command:

import unittest
import testing.postgresql

# Generate Postgresql class which shares the generated database
Postgresql = testing.postgresql.PostgresqlFactory(cache_initialized_db=True)


def tearDownModule(self):
    # clear cached database at end of tests
    Postgresql.clear_cache()


class MyTestCase(unittest.TestCase):
    def setUp(self):
        # Use the generated Postgresql class instead of testing.postgresql.Postgresql
        self.postgresql = Postgresql()

    def tearDown(self):
        self.postgresql.stop()

If you want to insert fixtures to the cached database, use initdb_handler option:

# create initial data on create as fixtures into the database
def handler(postgresql):
    conn = psycopg2.connect(**postgresql.dsn())
    cursor = conn.cursor()
    cursor.execute("CREATE TABLE hello(id int, value varchar(256))")
    cursor.execute("INSERT INTO hello values(1, 'hello'), (2, 'ciao')")
    cursor.close()
    conn.commit()
    conn.close()

# Use `handler()` on initialize database
Postgresql = testing.postgresql.PostgresqlFactory(cache_initialized_db=True,
                                                  on_initialized=handler)

Requirements

  • Python 2.7, 3.4, 3.5, 3.6
  • pg8000 1.10

License

Apache License 2.0

History

1.3.0 (2016-02-03)

  • Add testing.postgresql.PostgresqlFactory
  • Depend on testing.common.database package

1.2.1 (2015-08-22)

  • Fix bug:
    • Close #3 Fix AttributeError on end of tests

1.2.0 (2015-05-17)

  • Use pg8000 for connector to create test database
  • Connect to postgres to create test database (instead of template1)

1.1.2 (2015-04-06)

  • Fix bugs:
    • Do not call os.getpid() on destructor (if not needed)
    • Raise detailed RuntimeError if initdb exits non-zero

1.1.1 (2015-01-18)

  • Disable logging_collector feature (For Fedora)
  • Fix bugs:
    • MacPorts default path is /opt/local/lib/postgresql*, no dash

1.1.0 (2014-12-20)

  • Invoke 'postgres' command instead of 'postmaster'

1.0.6 (2014-07-19)

  • Fix #1 Dirty postmaster shut down

1.0.5 (2014-07-19)

  • Fix path for PostgreSQL
  • Use absolute path for which command

1.0.4 (2014-06-19)

  • Fix timeout on terminating postgresql
  • Support PostgreSQL on /usr/local/bin (cf. FreeBSD ports)
  • Fix bugs

1.0.3 (2014-06-11)

  • Fix ImportError if caught SIGINT on py3

1.0.2 (2013-12-06)

  • Change behavior: Postgresql#stop() cleans workdir
  • Fix caught AttributeError on object deletion

1.0.1 (2013-12-05)

  • Add @skipIfNotInstalled decorator (alias of skipIfNotFound)
  • Suport python 2.6 and 3.2

1.0.0 (2013-12-04)

  • Add @skipIfNotFound decorator

0.1.0 (2013-11-26)

  • First release

testing.postgresql's People

Contributors

adelosa avatar eradman avatar gliptak avatar graingert avatar ju2wheels avatar lpsinger avatar rutsky avatar sirex avatar sorki avatar tk0miya avatar tokenmathguy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

testing.postgresql's Issues

Why can't I perform engine.connect() operation ?

I tried using this for testing posgresql database. The engine creating part is working fine and I can execute commands directly using engine.execute().

But when I try using - conn = engine.connect() I get the error - terminating connection due to administrator command

Screenshot from 2019-07-03 18-51-44

RuntimeError: command not found: initdb

Hi , I wanted to write test-cases involving postgres for flask applicattion. When I was trying the testing.postgresql package, it's throwing this command not found: initdb error.
This is the code I'm trying,

import unittest
import testing.postgresql

Generate Postgresql class which shares the generated database

Postgresql = testing.postgresql.PostgresqlFactory(cache_initialized_db=True)

def tearDownModule(self):
# clear cached database at end of tests
Postgresql.clear_cache()

class MyTestCase(unittest.TestCase):
def setUp(self):
# Use the generated Postgresql class instead of testing.postgresql.Postgresql
self.postgresql = Postgresql()

def tearDown(self):
    self.postgresql.stop()

When I ran pytest, I'm getting this error,

src\tests\app\test_db.py:25: in
Postgresql = testing.postgresql.PostgresqlFactory(cache_initialized_db=True)
..\user\appdata\local\programs\python\python39\lib\site-packages\testing\common\database.py:52: in init
self.cache = self.target_class(**settings_noautostart)
..\user\appdata\local\programs\python\python39\lib\site-packages\testing\common\database.py:92: in init
self.initialize()
..\user\appdata\local\programs\python\python39\lib\site-packages\testing\postgresql.py:50: in initialize
self.initdb = find_program('initdb', ['bin'])
..\user\appdata\local\programs\python\python39\lib\site-packages\testing\postgresql.py:144: in find_program
raise RuntimeError("command not found: %s" % name)
E RuntimeError: command not found: initdb

I have set the path for "C:\Program Files\PostgreSQL\13\bin" in my environmental.

Am I missing something? Can someone help me out here.

Cannot run tests with testing.postgresql - application is not being found and failes to shutdown the server automatically

Hello, the below appears when following the guide:

ERROR: tearDownModule (__main__)
----------------------------------------------------------------------
TypeError: tearDownModule() missing 1 required positional argument: 'self'

I'm not quite sure how is this supposed to work, since the mehod is outside of the TestCase class

import unittest
import testing.postgresql

def tearDownModule(self):
    Postgresql.clear_cache()

class Base(unittest.TestCase):

    def setUp(self) -> None:
        self.postgresql = Postgresql()
        print(self.postgresql.url())

    def tearDown(self) -> None:
        self.postgresql.stop()

class TestInitial(Base):
    def some_test_suite(self):
        print('Test this works')

def some_test_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.TestLoader().loadTestsFromTestCase(TestInitial))
    return suite

if __name__ == '__main__':
    with testing.postgresql.Postgresql() as postgresql:
        engine = create_engine(postgresql.url())
        db = psycopg2.connect(**postgresql.dsn())
        runner = unittest.TextTestRunner()
        runner.run(some_test_suite())

Next what happens is two things:

No application found. Either work inside a view function or push an application context. See http://flask-sqlalchemy.pocoo.org/contexts/.

and

ERROR: testing.common.database: failed to shutdown the server automatically. Any server processes and files might have been leaked. Please remove them and call the stop() certainly

Full log:

postgresql://[email protected]:47359/test
No application found. Either work inside a view function or push an application context. See http://flask-sqlalchemy.pocoo.org/contexts/.
.
----------------------------------------------------------------------
Ran 1 test in 0.807s

OK
ERROR: testing.common.database: failed to shutdown the server automatically.
Any server processes and files might have been leaked. Please remove them and call the stop() certainly
ERROR: testing.common.database: failed to shutdown the server automatically.
Any server processes and files might have been leaked. Please remove them and call the stop() certainly

I've been struggling with this for quite a while and I will appreciate any feedback. Thank you

FATAL: could not create shared memory segment: Cannot allocate memory

testing.postgresql Version

testing.postgresql==1.3.0
python 3.10
on MacOs M1

Debug Output

python3.10/site-packages/testing/common/database.py:40: in __init__
    self.cache = self.target_class(**self.settings)
python3.10/site-packages/testing/common/database.py:96: in __init__
    self.setup()
python3.10/site-packages/testing/common/database.py:125: in setup
    self.initialize_database()
python3.10/site-packages/testing/postgresql.py:104: in initialize_database
    raise RuntimeError("initdb failed: %r" % err)
E   RuntimeError: initdb failed: b'2023-02-03 09:50:29.610 CET [20851] FATAL:  could not create shared memory segment: Cannot allocate memory\n2023-02-03 09:50:29.610 CET [20851] DETAIL:  Failed system call was shmget(key=40827474, size=56, 03600).\n2023-02-03 09:50:29.610 CET [20851] HINT:  This error usually means that PostgreSQL\'s request for a shared memory segment exceeded your kernel\'s SHMALL parameter.  You might need to reconfigure the kernel with larger SHMALL.\n\tThe PostgreSQL documentation contains more information about shared memory configuration.\nchild process exited with exit code 1\ninitdb: removing contents of data directory "/var/folders/j6/d40048xdd118w7q8jz_7f2cc0000gn/T/tmpiyy6vqob/data"\n'

Expected Behavior

As a developper I expect no error about memory.

Actual Behavior

Several times a day I have a memory shared error, the lib cannot allocate memory. The only solution who solve my problem is to restart my computer.

  • I try to found a process to kill but I didn't find.
  • I try to delete the data directory but didn't resolve my problem.

Steps to Reproduce

I launch in debug mode and I kill the process then I have the problem.
Randomly sometime when a test in development crash then I have the problem

provide an option to start subprocesses in a new process group

I have a script which uses testing.postgres to create a temporary database to test against.
My script has a SIGINT handler which handles teardown logic, which I want to run before the database is terminated.

The problem is that when one presses CTRL+C in a terminal, the shell sends SIGINT to the entire process group, thereby killing the postgres process before my teardown logic can finish running.

The solution I propose is to expose a boolean parameter to the testing.postgresql.Postgresql() constructor, which if True, will cause start_new_session=True to be passed to the call to subprocess.Popen in common/database.py, which will cause python to run setsid() in the child process before running postgres.

I'd be happy to open a PR if something along these lines sounds good to you.

Issue closing down instance on Windows

I'm working with Python 3.5.2 on WIndows 10.
Code like the following:

if __name__ == '__main__':
    with testing.postgresql.Postgresql() as postgresql:
        print("url={}".format(postgresql.url()))
        print("data directory={}".format(postgresql.get_data_directory()))
        run(postgresql.url())

run does nothing..

Receiving the following error:

Traceback (most recent call last):
  File "C:/Users/adelo/IdeaProjects/pr_data/test_orm.py", line 66, in <module>
    run(postgresql.url())
  File "C:\Users\adelo\venv\paymentrouter\lib\site-packages\testing\common\database.py", line 257, in __exit__
url=postgresql://[email protected]:57380/test
data directory=C:\Users\adelo\AppData\Local\Temp\tmpbnndk7ie\data
    self.stop()
  File "C:\Users\adelo\venv\paymentrouter\lib\site-packages\testing\common\database.py", line 197, in stop
    self.terminate(_signal)
  File "C:\Users\adelo\venv\paymentrouter\lib\site-packages\testing\postgresql.py", line 116, in terminate
    super(Postgresql, self).terminate(signal.SIGINT)
  File "C:\Users\adelo\venv\paymentrouter\lib\site-packages\testing\common\database.py", line 212, in terminate
    self.child_process.send_signal(_signal)
  File "c:\users\adelo\appdata\local\programs\python\python35-32\Lib\subprocess.py", line 1351, in send_signal
    raise ValueError("Unsupported signal: {}".format(sig))
ValueError: Unsupported signal: 2
ERROR: testing.common.database: failed to shutdown the server automatically.
Any server processes and files might have been leaked. Please remove them and call the stop() certainly
Process finished with exit code 1

So its saying that signal.SIGINT is not supported.. is that a Windows thing??

Support of Python 3.7

Hi!
is a release planned in the future which supports Python 3.7?
Thanks,
Steffen

postgresql bin path discovery

Hi, im submitting a PR to add RHEL/CentOS path but opening an issue because I think it would be good to reconsider how this implemented.

Right now its searching a bunch of paths for the postgresql binaries, but as you are already seeing hard coding these paths is not a good idea because it can vary.

On top of the bin dir being in different places, if you take the approach of "lets search for the first bin dir we can find", then we as testers have no control over which postgresql version gets used.

Your current implementation kind of makes the assumption that theres only one postgresql version per system which is not always true. For example, im working in CentOS 7 and if I install postgresql from the standard RHEL repo its at 9.2. However, I need to use 9.5 for testing because it has newer features I need, so I add the postgresql yum repo and install 9.5 (this can be done along side 9.2 without having to uninstall it). Now if I have both installed, how do I tell testing.postgresql which to choose?

I have submitted the patch in the current style but I would recommend it be changed in the future to use os.environ['PATH'] instead so we can just put the path to the postgresql bin dir that we want into the beginning of our PATH and you dont have to search for the directory, you just always execute the version found first in PATH.

I havent read through all the code but if its not only the bin dir we need then maybe have it look for PG_HOME env var or something if you also need the other postgresql folders relative to this.

How to add a POSTGIS extension in a test database ?

I am writing unit test for a function that makes a database connection and retrieves the SRID and Geometry type from a Postgres database ( having POSTGIS extension ) , How can i make an instance with a PostGIS extension?

'NoneType' object has no attribute 'path' in Postgresql.__del__

Great package! It works very well.

I just wanted to let you know: at the end of my tests, even the successful ones, I always get the message:

Exception AttributeError: "'NoneType' object has no attribute 'path'" in <bound method Postgresql.__del__ of <testing.postgresql.Postgresql object at 0x7f656550a4d0>> ignored

Am I doing something wrong?

What data format does copy_data_from expect?

I tried referencing a tablespace directory containing a single database and that failed:

>>> pg = testing.postgresql.Postgresql(copy_data_from='/aledade-test-data')
pg = testing.postgresql.Postgresql(copy_data_from='/aledade-test-data')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/jim/.buildout/eggs/testing.postgresql-1.2.0-py3.4.egg/testing/postgresql.py", line 69, in __init__
    self.setup()
  File "/Users/jim/.buildout/eggs/testing.postgresql-1.2.0-py3.4.egg/testing/postgresql.py", line 132, in setup
    raise RuntimeError("initdb failed: %r" % err)
RuntimeError: initdb failed: b'initdb: directory "/var/folders/vl/jn0sr3ds7h9b1zs9_lts29s00000gp/T/tmp1n_17su_/data" exists but is not empty\nIf you want to create a new database system, either remove or empty\nthe directory "/var/folders/vl/jn0sr3ds7h9b1zs9_lts29s00000gp/T/tmp1n_17su_/data" or run initdb\nwith an argument other than "/var/folders/vl/jn0sr3ds7h9b1zs9_lts29s00000gp/T/tmp1n_17su_/data".\n'

Psycopg2 not found

I can see the documentation mentions pg8000 as one of the requirements for this package to run. But following the basic steps from the documentation, I run into the error that says: ModuleNotFoundError: No module named 'psycopg2'.
As I dug into it, in the snippet:

engine = create_engine(postgresql.url())

the url() method looks like this:

def url(self, **kwargs):
        params = self.dsn(**kwargs)

        url = ('postgresql://%s@%s:%d/%s' %
               (params['user'], params['host'], params['port'], params['database']))

        return url

And since the url doesn't explicitly mention the pg8000 driver, I am getting the error. Is there any workaround for this if I don't want to install psycopg2?

client encoding

How can I specify what the default client encoding should be?

I have tests that fail on a Jenkins server because the client encoding is SQLASCII instead of UTF8...

psycopg2 variant (making pg8000 dependency optional)

Dear @tk0miya,
thanks for creating test.postgresql as Free Software!

Currently I'm considering using this module for testing some python code.
Looking at the requirements, I've noticed that pg8000 is listed.
(And https://github.com/tk0miya/testing.common.database is not, though needed.)

Browsing the code, I see that pg8000 is only used for a few simple operations.
For a python module that already depends on psycopg2 using testing.postgresql will create a new dependency to pg8000. To me it would be an improvement if you make pg8000 optional, so depending on either psycopg2 or pg8000.

Do you agree? Would you accept a pull request for this?

Best Regards

Issue with inserting data

On a regulat pgsql db this works fine but when i tried to to use testing.postgresql it throws this error

Note: the data shown is fake and I am not leaking any personal info as this was created by the faker class

It seems that Accessing class variables its treating it as a table

class AccountStatus(IntEnum):
    """
    Describes the status of a registered person.
    """
    INACTIVE = 0
    ACTIVE = 1
    BOT = 2
    PENDING = 3
    ON_VACATION = 4
    LOCKED = 5
    LOCKED_BY_ADMIN = 6
    DISABLED = 8
    SPAMCHECK_AWAITING = 9
    SPAMCHECK_DENIED = 10
    SPAMCHECK_MANUAL = 11

    def do_executemany(self, cursor, statement, parameters, context=None):
>       cursor.executemany(statement, parameters)
E       ProgrammingError: (psycopg2.ProgrammingError) missing FROM-clause entry for table "accountstatus"
E       LINE 2: ... NULL, NULL, NULL, '-', '-', NULL, NULL, NULL, 1, AccountSta...
E                                                                    ^
E        [SQL: 'INSERT INTO people (id, username, password, fullname, ircnick, avatar, avatar_id, introduction, postal_address, country_code, locale, birthday, birthday_month, telephone, facsimile, affiliation, bio, timezone, gpg_fingerprint, ssh_key, email, recovery_email, bugzilla_email, email_token, unverified_email, security_question, security_answer, login_attempt, password_token, old_password, certificate_serial, status, status_timestamp, privacy, email_alias, blog_rss, latitude, longitude, fas_token, github_token, twitter_token, login_timestamp, creation_timestamp, update_timestamp) VALUES (%(id)s, %(username)s, %(password)s, %(fullname)s, %(ircnick)s, %(avatar)s, %(avatar_id)s, %(introduction)s, %(postal_address)s, %(country_code)s, %(locale)s, %(birthday)s, %(birthday_month)s, %(telephone)s, %(facsimile)s, %(affiliation)s, %(bio)s, %(timezone)s, %(gpg_fingerprint)s, %(ssh_key)s, %(email)s, %(recovery_email)s, %(bugzilla_email)s, %(email_token)s, %(unverified_email)s, %(security_question)s, %(security_answer)s, %(login_attempt)s, %(password_token)s, %(old_password)s, %(certificate_serial)s, %(status)s, CURRENT_TIMESTAMP, %(privacy)s, %(email_alias)s, %(blog_rss)s, %(latitude)s, %(longitude)s, %(fas_token)s, %(github_token)s, %(twitter_token)s, %(login_timestamp)s, %(creation_timestamp)s, CURRENT_TIMESTAMP)'] [parameters: ({'username': u'bob', 'bugzilla_email': None, 'certificate_serial': 1, 'locale': u'en_US', 'fas_token': None, 'telephone': None, 'affiliation': None, 'security_question': u'-', 'country_code': u'US', 'timezone': u'UTC', 'avatar_id': u'[email protected]', 'id': 0, 'privacy': 1, 'introduction': 'hello', 'password_token': None, 'latitude': 43, 'birthday_month': None, 'creation_timestamp': datetime.datetime(2011, 11, 28, 14, 46, 44, tzinfo=<UTC>), 'email': u'[email protected]', 'recovery_email': None, 'status': <AccountStatus.DISABLED: 8>, 'bio': None, 'login_timestamp': None, 'postal_address': u'783 Wiegand Lights Apt. 566\nNorth Marceloview, ND 52464-4628', 'unverified_email': None, 'ssh_key': None, 'birthday': None, 'email_alias': True, 'ircnick': None, 'password': '$6$GA8meLk2obnJCHBz$KWqipYAXkxpCvJ0SEpZcp8pyCrJ1qjAhzMIqNtpnTALmXilKlaxuggi2NRpViHen0isvl0BN3cuqdkvQ5jwkO.', 'twitter_token': None, 'gpg_fingerprint': None, 'login_attempt': None, 'github_token': None, 'email_token': None, 'longitude': 27, 'blog_rss': None, 'security_answer': u'-', 'facsimile': None, 'avatar': u'https://seccdn.libravatar.org/avatar/c961431faea38ed65bfd982cf2e31bd0', 'fullname': u'bob', 'old_password': None}, {'username': u'jerry', 'bugzilla_email': None, 'certificate_serial': 1, 'locale': u'en_US', 'fas_token': None, 'telephone': None, 'affiliation': None, 'security_question': u'-', 'country_code': u'FR', 'timezone': u'UTC', 'avatar_id': u'[email protected]', 'id': 1, 'privacy': 0, 'introduction': 'intro', 'password_token': None, 'latitude': None, 'birthday_month': None, 'creation_timestamp': datetime.datetime(2013, 8, 28, 14, 46, 44, tzinfo=<UTC>), 'email': u'[email protected]', 'recovery_email': None, 'status': <AccountStatus.ACTIVE: 1>, 'bio': 'biography of a simple individual', 'login_timestamp': None, 'postal_address': None, 'unverified_email': None, 'ssh_key': None, 'birthday': None, 'email_alias': True, 'ircnick': None, 'password': '$6$GA8meLk2obnJCHBz$KWqipYAXkxpCvJ0SEpZcp8pyCrJ1qjAhzMIqNtpnTALmXilKlaxuggi2NRpViHen0isvl0BN3cuqdkvQ5jwkO.', 'twitter_token': None, 'gpg_fingerprint': None, 'login_attempt': None, 'github_token': None, 'email_token': None, 'longitude': None, 'blog_rss': None, 'security_answer': u'-', 'facsimile': None, 'avatar': u'https://seccdn.libravatar.org/avatar/284459354f2531f6923b7e4e57c06792', 'fullname': u'jerry', 'old_password': None}, {'username': u'mike', 'bugzilla_email': None, 'certificate_serial': 1, 'locale': u'en_US', 'fas_token': None, 'telephone': None, 'affiliation': None, 'security_question': u'-', 'country_code': u'CA', 'timezone': u'UTC', 'avatar_id': u'[email protected]', 'id': 2, 'privacy': 1, 'introduction': None, 'password_token': None, 'latitude': Decimal('85.5335165'), 'birthday_month': None, 'creation_timestamp': datetime.datetime(2016, 2, 28, 14, 46, 44, tzinfo=<UTC>), 'email': u'[email protected]', 'recovery_email': None, 'status': <AccountStatus.PENDING: 3>, 'bio': 'i <3 oss', 'login_timestamp': None, 'postal_address': u'487 Brown Greens Apt. 191\nSmithamton, AL 33044-3467', 'unverified_email': None, 'ssh_key': None, 'birthday': None, 'email_alias': True, 'ircnick': None, 'password': '$6$GA8meLk2obnJCHBz$KWqipYAXkxpCvJ0SEpZcp8pyCrJ1qjAhzMIqNtpnTALmXilKlaxuggi2NRpViHen0isvl0BN3cuqdkvQ5jwkO.', 'twitter_token': None, 'gpg_fingerprint': None, 'login_attempt': None, 'github_token': None, 'email_token': None, 'longitude': Decimal('74.295654'), 'blog_rss': None, 'security_answer': u'-', 'facsimile': None, 'avatar': None, 'fullname': u'mike', 'old_password': None})]

Update release on PyPI?

@tk0miya would it be possible to make a new release on PyPI, based on the current master branch? There are a few fixes in there (particularly for Windows issues - see #16) which would be really useful to have without having to install via git.

Thanks!

How to change database name?

Is it possible to change the database name, and other configuration apart from the port?

I am trying to do this:

Postgresql = testing.postgresql.PostgresqlFactory(
    cache_initialized_db=True, 
    on_initialized=handler, 
    port=settings.DB_SETTINGS['port'],
    database=settings.DB_SETTINGS['name'],
    user=settings.DB_SETTINGS['user'],
    host=settings.DB_SETTINGS['host']
)

When in the handler method, when running

print(postgresql.dsn())

I receive the default configuration, not the parameters I passed in.

PostgreSQL Semaphore exhaustion

We use testing.postgresql in our automated tests that fire-up a local instance of PostgreSQL using initdb and then run our schema creation and data seeding tasks.

For the past few months we've been getting test failures due with this error:

initdb failed: 'FATAL: could not create semaphores: No space left on device
DETAIL: Failed system call was semget(1, 17, 03600).
HINT: This error does not mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its max_connections parameter.
The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.
child process exited with exit code 1
initdb: removing contents of data directory "/tmp/tmpctGr9I/data"

These tests run on an Amazon EC2 instance running Amazon Linux 2015.09 x64. Here's the output from ipcs -ls showing semaphore configuration:

[ec2-user@ip-127-0-0-1 etc]$ ipcs -ls
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
[ec2-user@ip-127-0-0-1 etc]$

We see the same issue on both Mac OS X with PostgreSQL 9.4 as well as on our Amazon EC2 instances running PostgreSQL 9.3. PostgreSQL is installed via yum on the EC2 instances via our build script, and brew on Mac.

We're unsure if it's a bug in PostgreSQL or testing.postgresql, but we thought we'd start here.

We're currently using testing.postgresql 1.1.2 (we're in the process of upgrading to the latest version - it is currently incompatible with our builds).

trailing postgres cluster

Hi, I'm using the following snippet to run my tests.

import unittest
import testing.postgresql


class MyTestCase(unittest.TestCase):

    @classmethod
    def setUpClass(self):
        self.postgresql = testing.postgresql.Postgresql()

    @classmethod
    def tearDownClass(self):
        self.postgresql.stop()

The problem I'm currently having is that the testing postgres cluster doesn't shutdown if my test run into an Error. Is there a way to avoid this?

testing.postgresql not working

Traceback (most recent call last):
File "C:\Users\Oloyede Emmanuel\Documents\Hook\tests.py", line 42, in setUp
with testing.postgresql.Postgresql() as postgresql:
File "C:\Users\Oloyede Emmanuel\Documents\Hook\env\lib\site-packages\testing\common\database.py", line 92, in init
self.initialize()
File "C:\Users\Oloyede Emmanuel\Documents\Hook\env\lib\site-packages\testing\postgresql.py", line 50, in initialize
self.initdb = find_program('initdb', ['bin'])
File "C:\Users\Oloyede Emmanuel\Documents\Hook\env\lib\site-packages\testing\postgresql.py", line 144, in find_program
raise RuntimeError("command not found: %s" % name)
RuntimeError: command not found: initdb

ImportError: Failed to import test module (in docker).

Running tests works in Windows Subsystem for Linux but not in a Docker container. Is my general test layout correct? Looks like it's not able to find initdb. In other words, is it having conflict issues with the database created in Docker-compose.yml or does it have anything to do with that?

My test file testMyProject.py:

import unittest
import psycopg2
import testing.postgresql
from sqlalchemy import create_engine

class TestPostgresqlInteraction(unittest.TestCase):

    Postgresql = testing.postgresql.PostgresqlFactory(cache_initialized_db=True)

    with testing.postgresql.Postgresql() as postgresql:
        engine = create_engine(postgresql.url())
        dbConn = psycopg2.connect(**postgresql.dsn())
        cursor = dbConn.cursor()
        def setUp(self):
            self.postgresql = self.Postgresql()

        def tearDown(self):
            self.postgresql.stop()

        def testMethod(self)
            conn = psycopg2.connect(**self.postgresql.dsn())
            cursor = conn.cursor()
            # execute cursor and do tests... 
        
        def tearDownModule(self):
            self.Postgresql.clear_cache()

docker-compose.yml:

version: "3"

services:
    myproject:
        build:
            context: .
            dockerfile: ./myproject/Dockerfile
        depends_on:
            - database
        volumes:
            - ./myproject:/code
        stdin_open: true
        tty: true
    database:
        build:
            context: .
            dockerfile: ./database/Dockerfile
        environment:
            POSTGRES_USER_FILE: /secrets/dbUser.txt
            POSTGRES_PASSWORD_FILE: /secrets/dbPassword.txt
        ports:
            - "8765:5432"
        volumes:
            - otherdata:/var/lib/postgresql/data

I run docker-compose exec myproject python3 -m unittest
Traceback:

ERROR: tests.testMyProject (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.testMyProject 
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/code/tests/testMyproject.py", line 40, in <module>
    class TestPostgresqlInteraction(unittest.TestCase):
  File "/code/tests/testMyproject.py", line 46, in TestPostgresqlInteraction
    Postgresql = testing.postgresql.Postgresql(copy_data_from='/database/Dockerfile')
  File "/usr/local/lib/python3.7/site-packages/testing/common/database.py", line 92, in __init__
    self.initialize()
  File "/usr/local/lib/python3.7/site-packages/testing/postgresql.py", line 50, in initialize
    self.initdb = find_program('initdb', ['bin'])
  File "/usr/local/lib/python3.7/site-packages/testing/postgresql.py", line 144, in find_program
    raise RuntimeError("command not found: %s" % name)
RuntimeError: command not found: initdb

How can I read from a pg_dump

Is there a way to read a pg_dump that is in pure sql? Otherwise, populating a test db ends up being very monotonous.

ResourceWarning

ResourceWarning error occurs with basic usage (e.g., from documentation).

ResourceWarning: unclosed <socket.socket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 58967), raddr=('127.0.0.1', 58965)> return self._sock.send(b) ResourceWarning: Enable tracemalloc to get the object allocation traceback

I'm using python 3.7, this may be a compatibility based warning.

Support of pg3.0 and above

It seems like this library hasn't been updated for years. I'm wondering do we have any plans for supporting psycopg3? Thanks

Cache result of initdb

On Ubuntu 14.04 with PostgreSQL 9.3 database initialization with initdb is the slowest part of testing.postgresql: it takes around 2.5 seconds to create data directory with default contents.

While it's possible to create PG data directory outside of testing.postgresql, cache it, copy for each test and pass to testing.postgresql.Postgresql with copy_data_from argument, it's a lot of work and requires to reimplementing of some of the testing.postgresql functionality (e.g. search of initdb utility).

I propose to implement caching of initdb result inside PostgreSQL and use it by default.

This can be done pretty straightforward and I can prepare PR for this issue if you think my approach is satisfactory:

  1. Add dependency for library that provides API to work with user cache directory (e.g. appdirs). Cache directory will be used for caching initdb result. Cache should be verioned by initdb version (initdb -V) and by testing.postgresql version.
  2. Add option to testing.postgresql.Postgresql to disable cache. e.g. cache=False.
    In testing.postgresql.Postgresql if cache is enabled: check if cache for the current version of initdb+testing.postgresql exists, if no, create it and fill it with initdb; copy cached directory content to a temporary PG data directory.

pytest fixture has database refusing connection

I am using python 3.7 and postgresql

@pytest.fixture(autouse=True, scope="session")
def postgres_factory():
    """
    Creates an initial fake database for use in unit tests.
    """
    postgres_factory = postgresql.PostgresqlFactory(cache_initialized_db=True, on_initialized=create_initial_data)
    return postgres_factory


@pytest.fixture(autouse=True, scope="class")
def postgres_instance(postgres_factory):
    fake_db = postgres_factory()
    return fake_db

I have these as pytest fixtures and when it tries to recreate them the first query throws a connection refused error.

Enable customization of kill timeout

Postgres takes about 11 seconds on my machine to shut down for some reason. Currently there's a constant testing.common.database called DEFAULT_KILL_TIMEOUT which is set to 10. I can't override this so I get a lot of superfluous shutdown exceptions.

Would it be possible to make this value configurable?

ModuleNotFoundError: No module named 'testing'

I have the module installed correctly
pip3 show testing.postgresql Name: testing.postgresql Version: 1.3.0 Summary: automatically setups a postgresql instance in a temporary directory, and destroys it after testing Home-page: https://github.com/tk0miya/testing.postgresql Author: Takeshi Komiya Author-email: i.tkomiya at gmail.com License: Apache License 2.0 Location: /Users/prashant/Desktop/work/prashant/venv/lib/python3.8/site-packages Requires: pg8000, testing.common.database Required-by:

still getting the error ModuleNotFoundError: No module named 'testing' when i am running the test

Please sign release tarballs

Hi,

I am packaging testing.* for Debian.

It would be great if you could pgp sign your source tarballs uploaded to PyPI as of the next release โ˜บ!

Cheers,
Nik

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.