Git Product home page Git Product logo

archived-django-concurrent-test-helper's Introduction

django-concurrent-test-helper

Build Status Latest PyPI version

Helpers for executing Django app code concurrently within Django tests.

Tested against the same versions of Python that Django supports:

x Py2.7 Py3.4 Py3.5 Py3.6
Django 1.4
     
Django 1.5
     
Django 1.6
     
Django 1.7
   
Django 1.8
 
Django 1.9
 
Django 1.10
 
Django 1.11

(with the exception of Python 3.2 and 3.3... these are no longer supported)

Getting started

pip install django-concurrent-test-helper

Goes well with https://github.com/box/flaky (pip install flaky), as you may want to run a test several times while trying to trigger a rare race condition.

You may need to set a config value in your Django settings.py. By default we use MANAGE_PY_PATH = "./manage.py", this should work for most cases. If you need another path set MANAGE_PY_PATH in your Django settings.

You need to add this library to your Django project settings too:

INSTALLED_APPS = (
    # ...
    'django_concurrent_tests',
)

Two helpers are provided, call_concurrently:

from django_concurrent_tests.helpers import call_concurrently

def is_success(result):
    return result is True and not isinstance(result, Exception)

def test_concurrent_code():
    results = call_concurrently(5, racey_function, first_arg=1)
    # results contains the return value from each call
    successes = list(filter(is_success, results))
    assert len(successes) == 1

and make_concurrent_calls:

from django_concurrent_tests.helpers import make_concurrent_calls

def is_success(result):
    return result is True and not isinstance(result, Exception)

def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (second_func, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)
    # results contains the return value from each call
    successes = list(filter(is_success, results))
    assert len(successes) == 1

If you are using Django's TestCase class you need to separate your concurrent tests and use Django's TransactionTestCase as the base class instead (see DB Transactions), i.e.:

from django.test import TransactionTestCase
from django_concurrent_tests.helpers import make_concurrent_calls

def is_success(result):
    return result is True and not isinstance(result, Exception)

class MyConcurrentTests(TransactionTestCase):
    def test_concurrent_code(self):
        calls = [
            (first_func, {'first_arg': 1}),
            (second_func, {'other_arg': 'wtf'}),
        ] * 3
        results = make_concurrent_calls(*calls)
        # results contains the return value from each call
        successes = list(filter(is_success, results))
        assert len(successes) == 1

Note that if your called function raises an exception, the exception will be wrapped in a WrappedError exception. This provides a way to access the original traceback, or even re-raise the original exception.

import types

from django_concurrent_tests.errors import WrappedError
from django_concurrent_tests.helpers import make_concurrent_calls

def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (raises_error, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)
    # results contains the return value from each call
    errors = list(filter(lambda r: isinstance(r, Exception), results))
    assert len(errors) == 3

    assert isinstance(errors[0], WrappedError)
    assert isinstance(errors[0].error, ValueError)  # the original error
    assert isinstance(errors[0].traceback, types.TracebackType)

# other things you can do with the WrappedError:

# 1. print the traceback
errors[0].print_tb()

# 2. drop into a debugger (ipdb if installed, else pdb)
errors[0].debug()
ipdb>
# ...can explore the stack of original exception!

# 3. re-raise the original exception
try:
    errors[0].reraise()
except ValueError as e:
    # `e` will be the original error with original traceback

Another thing to remember is if you are using the override_settings decorator in your test. You need to also decorate your called functions (since the subprocesses won't see the overridden settings from your main test process):

from django_concurrent_tests.helpers import make_concurrent_calls

@override_settings(SPECIAL_SETTING=False)
def test_concurrent_code():
    calls = [
        (first_func, {'first_arg': 1}),
        (raises_error, {'other_arg': 'wtf'}),
    ] * 3
    results = make_concurrent_calls(*calls)

@override_settings(SPECIAL_SETTING=False)
def first_func(first_arg):
    return first_arg * 2

def raises_error(other_arg):
    # can also be used as a context manager
    with override_settings(SPECIAL_SETTING=False):
        raise SomeError(other_arg)

On the other hand, customised environment vars will be inherited by the subprocess and an override_environment context manager is provided for use in your tests:

from django_concurrent_tests.helpers import call_concurrently
from django_concurrent_tests.utils import override_environment

def func_to_test(first_arg):
    import os
    return os.getenv('SPECIAL_ENV')

def test_concurrent_code():
    with override_environment(SPECIAL_ENV='so special'):
        results = call_concurrently(1, func_to_test)
    assert results[0] == 'so special'

Lastly, you can pass a string import path to a function rather than the function itself. The format is: 'dotted module.path.to:function' (NOTE colon separates the name to import, after the dotted module path).

This can be nice when you don't want to import the function itself in your test to pass it. But more importantly it is essential in some cases, such as when f is a decorated function whose decorator returns a new object (and functools.wraps was not used). In that situation we will not be able to introspect the import path from the function object's __module__ (which will point to the decorator's module instead), so for those cases calling by string is mandatory. (Celery tasks decorated with @app.task are an example which need to be called by string path)

from django_concurrent_tests.helpers import call_concurrently

@bad_decorator
def myfunc():
    return True

def test_concurrent_code():
    results = call_concurrently('mymodule.module:myfunc', 3)
    # results contains the return value from each call
    results = list(filter(None, results))
    assert len(results) == 3

NOTES

Why subprocesses?

We originally wanted to implement this purely using multiprocessing.Pool to call the function you want to test. If that had worked then this module would hardly be necessary.

Unfortunately we hit a problem with this approach: multiprocessing works by forking the parent process. The forked processes inherit the parent's sockets, so in a Django project this will include things like the socket opened by psycopg2 to your Postgres database. However the inherited sockets are in a broken state. There's a bunch of questions about this on SO and no solutions presented, it seems basically you can't fork a Django process and do anything with the db afterwards.

(Note in 1.8+ versions of Django the multiprocessing approach is possible, we hope to release an updated version of this library based on that at some point...)

So in order to make this work we have to use subprocess.Popen to run with un-forked 'virgin' processes. To be able to test an arbitrary function in this way we do an ugly/clever hack and provide a manage.py concurrent_call_wrapper command (which is why you have to add this module to your INSTALLED_APPS) which handles the serialization of kwargs and return values.

This does mean that your kwargs and return value must be pickleable.

Another potential gotcha is if you are using SQLite db when running your tests. By default Django will use :memory: for the test-db in this case. But that means the concurrent processes would each have their own in-memory db and wouldn't be able to see data created by the parent test run.

For these tests to work you need to be sure to set TEST_NAME for the SQLite db to a real filename in your DATABASES settings (in Django 1.9 this is a dict, i.e. {'TEST': {'NAME': 'test.db'}}).

DB Transactions

Finally you need to be careful with Django's implicit db transactions, otherwise data you create in the parent test case has not yet been committed and is therefore not visible to the subprocesses.

Ensure that you use Django's TransactionTestCase or a derivative (to prevent all the code in your test from being inside an uncommitted transaction).

archived-django-concurrent-test-helper's People

Contributors

anentropic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

archived-django-concurrent-test-helper's Issues

child_exception raised

When executing my test code a exception is thrown. I have searched the web for some information that could help but without success.

The models are a bit complex so I have used a factory (code ommited), but I don't think it is relevant to the problem. Anyway, if required I can create a mininal reproducible code.

The tests are run inside a Docker container.

Docker version 17.03.1-ce, build c6d412e
Python 2.7.13

Django==1.9.2
django-concurrent-test-helper==0.2.9
psycopg2==2.5.1

The command I used. Not clear if I have to do anything different here.

python manage.py test --settings=protocolo.test_settings protocolo.tests.TestesConcorrenciaProcesso
Exception in thread Thread-44:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/django_concurrent_tests/utils.py", line 44, in target
    stderr=subprocess.PIPE,
  File "/usr/local/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Test code:

def obteve_successo(result):
    return result is True and not isinstance(result, Exception)

def criar_processo(setor=None):
    return ProcessoFactory(setor_origem=setor)

class TestesConcorrenciaProcesso(TransactionTestCase):

    def test_criar_processos_concorrentemente(self):
        setor = SetorFactory()
        processos = call_concurrently(20, criar_processo, setor=setor)

        sucesso = all(map(obteve_successo, processos))
        self.assertTrue(sucesso)

I'd appreciate any help.

Support for Django 1.11

Hi there,
This project looks extremely useful. Attempting to reproduce a whole family of concurrency bugs that I've identified in some django apps I have led me here.
Trying it on Django 1.11 fails with:

Exception in thread Thread-17:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/miguel/cm/GoTrips/env/lib/python2.7/site-packages/django_concurrent_tests/utils.py", line 45, in target
    env=os.environ.copy(),
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/miguel/cm/GoTrips/env/lib/python2.7/site-packages/django_concurrent_tests/utils.py", line 45, in target
    env=os.environ.copy(),
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

trying to use this inside test code similar to:

from django.test import TransactionTestCase
from django_concurrent_tests.helpers import call_concurrently

from .mystuff import funcion_with_racy_access_to_database


class TestFoo(TransactionTestCase)
  def test_race(self):
        results = call_concurrently(
            2,
            function_with_racy_access_to_database,
            arg1='foo',
            arg2='bar',
        )
        expected = results[0]
        for e in results[1:]:
            self.assertEqual(expected, e)

where function_with_racy_access_to_database returns an int. contents of results is: [None, None]

Add MANAGE_PY_PATH to the documentation

Document MANAGE_PY_PATH setting in some detail. I.E. for me it does not work without setting MANAGE_PY_PATH="./manage.py" (otherwise the error is, that python manage.py is not found). But I am guessed that value by try-error and I don't know what is the proper value.

Document usage for request testing (if it is possible)

I am trying to write test of concurrent requests (to django-rest-framework in my case, documentation for the normal requests would be very helpful), but I am a bit lost. I am trying following code:

@override_settings(
    ES_ENABLED=False,
)
def put_rating_create(user=None, asset=None):
    return Asset.objects.get(version_uuid=asset.version_uuid)
    client = APIClient()
    client.force_authenticate(user=user)
    url = reverse('v1:rating-detail', kwargs={'parent_lookup_asset__version_uuid': asset.version_uuid, 'rating_type': 'working_hours'})
    response = client.put(url, {'score': '3.5'}, format='json')
    assert response.status_code == status.HTTP_201_CREATED
    assert response.json()['score'] == 3.5
    assert Rating.objects.get(asset=asset, user_profile=user).score == 3.5
    return response


@override_settings(
    ES_ENABLED=False,
)
def test_put_rating_create_duplicate():
    """
    Test simple search for submitting new rating. Test multiple submissions at same time.
    """
    try:
        user = UserProfile.objects.create(username='test-user')
        asset_type = mommy.make('AssetType', name="model")
        asset = Asset.objects.create(author=user, name="foo_asset", asset_uuid="655253f5-c032-4bd6-8ba5-9ec70faeb2a2", asset_type=asset_type)
        calls = [
            (put_rating_create, {'user': user, 'asset': asset}),
        ] * 1
        responses = make_concurrent_calls(* calls)
        print(responses)
        print(responses[0].print_tb())
        assert responses[0].status_code == status.HTTP_201_CREATED
    finally:
        Asset.objects.filter(asset_uuid="655253f5-c032-4bd6-8ba5-9ec70faeb2a2").delete()

The problem is, that the Asset I created in test_put_rating_create_duplicate() is not visible in put_rating_create(), so I end up with following error:

WrappedError('"DoesNotExist(\'Asset matching query does not exist.\')"')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.