mback2k / django-celery-model Goto Github PK

Keep track of Celery tasks assigned to Django models

License: MIT License

Python 97.88% Makefile 2.12%

django-celery-model's Introduction

django-celery-model is an extension to Celery which adds support for tracking Celery tasks assigned to Django model instances.

Installation

Install the latest version from pypi.python.org:

pip install django-celery-model

Install the development version by cloning the source from github.com:

pip install git+https://github.com/mback2k/django-celery-model.git

Configuration

Add the package to your INSTALLED_APPS:

INSTALLED_APPS += (
    'djcelery_model',
)

Make sure that you are receiving Celery events via:

CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_SEND_SENT_EVENT = True
CELERY_SEND_EVENTS = True

Example

Add the TaskMixin to your Django model:

from django.db import models
from django.utils.translation import ugettext_lazy as _
from djcelery_model.models import TaskMixin

class MyModel(TaskMixin, models.Model):
    name = models.CharField(_('Name'), max_length=100)

Queue an asynchronous task from your Django model instance:

from .models import MyModel
from .tasks import mytask

mymodel = MyModel.objects.get(name='test instance')
mymodel.apply_async(mytask, ...)

Retrieve list of asynchronous tasks assigned to your Django model instance:

mymodel.tasks.all()
mymodel.tasks.pending()
mymodel.tasks.started()
mymodel.tasks.retrying()
mymodel.tasks.failed()
mymodel.tasks.successful()
mymodel.tasks.running()
mymodel.tasks.ready()

Check for a running or ready asynchronous task for your Django model instance:

mymodel.has_running_task
mymodel.has_ready_task

Handle asynchronous task results for your Django model instance:

mymodel.get_task_results()
mymodel.get_task_result(task_id)
mymodel.clear_task_results()
mymodel.clear_task_result(task_id)

Filter your Django model based upon asynchronous tasks:

MyModel.objects.with_tasks()
MyModel.objects.with_pending_tasks()
MyModel.objects.with_started_tasks()
MyModel.objects.with_retrying_tasks()
MyModel.objects.with_failed_tasks()
MyModel.objects.with_successful_tasks()
MyModel.objects.with_running_tasks()
MyModel.objects.with_ready_tasks()

MyModel.objects.without_tasks()
MyModel.objects.without_pending_tasks()
MyModel.objects.without_started_tasks()
MyModel.objects.without_retrying_tasks()
MyModel.objects.without_failed_tasks()
MyModel.objects.without_successful_tasks()
MyModel.objects.without_running_tasks()
MyModel.objects.without_ready_tasks()

License

Released under MIT License
Copyright (c) 2014-2019 Marc Hoersken [email protected]

django-celery-model's People

Contributors

Stargazers

Watchers

Forkers

a3linux forcityplatform ungikim score42 mbelousov dr08 luis2906 dmed-ai mikewolfd stormlifter flowt-io toothstone thenewguy autopi-io grusinator martin-sicho sabawesabi

django-celery-model's Issues

Track task state in Django database and expose interface

Use signals to copy task state into Django database in order to expose state-based query interface that does not depend on BaseAsyncResult.

The interface should allow querying based on the following criteria:

AsyncResult.state (PENDING, STARTED, RETRY, FAILURE, SUCCESS)
AsyncResult.ready
AsyncResult.failed
AsyncResult.successful

Does this require to use the database as the broker?

If so, can you please specify it in the readme file?

Django 3 compatibility

Need to update for Django 3. Traceback indicates importing some code that was removed from Django

$ /home/vagrant/.tox/py36/bin/python manage.py makemigrations
Traceback (most recent call last):
  File "manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)

...

    from djcelery_model.models import ModelTaskMeta, TaskMixin
  File "/home/vagrant/.tox/py36/lib/python3.6/site-packages/djcelery_model/models.py", line 12, in <module>
    from django.utils.encoding import python_2_unicode_compatible
ImportError: cannot import name 'python_2_unicode_compatible'
ERROR: InvocationError for command /home/vagrant/.tox/py36/bin/python manage.py makemigrations (exited with code 1)

Type object 'ModelTaskMetaState' has no attribute 'REJECTED'

I have a weird issue after trying to reject a task.
I am attempting to switch from redis to rabbitmq where the rejected / failed tasks should go to a specific queue where they can then be requeued manually if necessary.

But i get the following exception after trying to raise Reject(Exception("example"), requeue=False)

[2020-09-11 14:01:09,827: ERROR/ForkPoolWorker-1] Signal handler <function handle_task_postrun at 0x7f86adad2b50> raised: AttributeError("type object 'ModelTaskMetaState' has no attribute 'REJECTED'",)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 288, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python2.7/site-packages/djcelery_model/models.py", line 234, in handle_task_postrun
    queryset.update(state=ModelTaskMetaState.lookup(state))
  File "/usr/local/lib/python2.7/site-packages/djcelery_model/models.py", line 33, in lookup
    return getattr(cls, state)
AttributeError: type object 'ModelTaskMetaState' has no attribute 'REJECTED'

Does anyone know what might be the issue?
I am using the most recent version of celery 4.4.7

Does anyone have any ideas about why this causes an exception?
It does seem like rejected is just not supported?

Best regards

handle_task_postrun is not run on task failure

I think that task_failure signal handler has to be added to models.py.

Using with celery primatives?

Does this package work with celery primatives, i.e. chords, groups, etc.?

P.S.
Btw, great package.

interest in adding created, modified to task models.

I was going to add created and modified fields to the model task meta class.

i.e. something like

class ModelTaskMeta(models.Model):
    ...
    created = models.DateTimeField(auto_now_add=True, editable=False)
    modified = models.DateTimeField(auto_now=True)

Issue with migrations

Hi, I just installed this module

Followed the tutorial from pypi

However, when I try to run my django instance, it throws me this:

worker_1 |
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python3.8/site-packages/celery/utils/dispatch/signal.py", line 288, in send
worker_1 | response = receiver(signal=self, sender=sender, **named)
worker_1 | File "/usr/local/lib/python3.8/site-packages/djcelery_model/models.py", line 234, in handle_task_postrun
worker_1 | queryset.update(state=ModelTaskMetaState.lookup(state))
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 741, in update
worker_1 | rows = query.get_compiler(self.db).execute_sql(CURSOR)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1462, in execute_sql
worker_1 | cursor = super().execute_sql(result_type)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
worker_1 | cursor.execute(sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
worker_1 | return super().execute(sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
worker_1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
worker_1 | return executor(sql, params, many, context)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
worker_1 | return self.cursor.execute(sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in exit
worker_1 | raise dj_exc_value.with_traceback(traceback) from exc_value
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
worker_1 | return self.cursor.execute(sql, params)
worker_1 | django.db.utils.ProgrammingError: relation "djcelery_model_modeltaskmeta" does not exist
worker_1 | LINE 1: UPDATE "djcelery_model_modeltaskmeta" SET "state" = 4 WHERE ...

Then, by checking out to the branch 'dependabot/pip/django-3.0.7', I found out there are some migrations for Django. (Django 3.x, not south)

This error tells me there is an issue with those

I was expecting it to work out of the box since the tutorial never mentions this, I guess I need to run them?

If so, how? Any idea on this?

Thank You

Tasks that work with multiple model instances

I was looking for a way to tag model instances and tasks together. A helpful user on #django pointed me here. Really cool!

It would be very handy if one could tag multiple instances to a particular task.

Then you could filter all tasks for a user, a group, a record, etc... Or when the tasks involves multiple records you can get there from any of the related models.