pombreda / django-tasks Goto Github PK

The time stamps in logs are :
Run a successful task finished successfully on Aug. 8, 2010 at 9:33 a.m
We should rather use ISO UTC time stamps.

It's great that there's a cancel task method in django-tasks. But this is a 
very 'ugly' solution to the "how can I stop my long running task in the middle 
of execution" problem. If you're unlucky your task might stop without properly 
closing/otherwise finalizing some critical resource.

I propose that the `cancel_task` method should by split into two functions:
- stop_task(task)
- kill_task(task)

`kill_task` would function identically to how `cancel_task` currently function, 
i.e., tell the os to kill the process.

`stop_task` would just set a flag on the task model, e.g., `stopped = True`. If 
the method that is running is django-tasks aware then it would be periodically 
checking the `stopped` flag, and upon being set to `True` it would gracefully 
exit. If the method is not django-tasks aware then this method would 
effectively do nothing.

manage.py syncdb fails with following message.

Error: One or more models did not validate:
djangotasks.functiontask: "function_name": CharField cannot have a "max_length" 
greater than 255 when using "unique=True".

I'm using djangotasks r41 and django 1.2.3 on Win32 connected to mysql 5.0.24a 
on Debian. It's a working environment that I'm trying to add task processing to.


So, I'm just curious.  Are you running into a lot of instances where you need 
more than 255 characters to represent the name of a function?

I'm modifying my version in place, so as to keep moving forward for now.

Django 1.3
Current version of django-task


Example Model

class Group(EC2Model):
    env = models.ForeignKey(Env)
    name = models.CharField(max_length=50)
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return "%s for %s" % (self.name, self.env)

    def test(self):
        print "got here"
        return 0


In my view

m = Group()
m.save()

task = djangotasks.task_for_object(m.test)


Exception


Environment:


Request Method: POST
Request URL: http://127.0.0.1:8000/operations/setup/group/?id=1

Django Version: 1.3
Python Version: 2.7.1
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.admin',
 'config',
 'index',
 'operations',
 'djangotasks']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'middleware.auth.LoginRequiredMiddleware',
 'middleware.setup.SetupRequiredMiddleware')


Traceback:
File 
"/Users/mzupan/django/sp/lib/python2.7/site-packages/django/core/handlers/base.p
y" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)
File "/Users/mzupan/django/sp/www/operations/views.py" in config_group
  29.             form.save()
File "/Users/mzupan/django/sp/www/operations/forms.py" in save
  74.         task = djangotasks.task_for_object(m.test)
File "/Users/mzupan/django/sp/www/djangotasks/__init__.py" in task_for_object
  66.     return Task.objects.task_for_object(object_method.im_class, object_method.im_self.pk, object_method.im_func.__name__)
File "/Users/mzupan/django/sp/www/djangotasks/models.py" in task_for_object
  99.             raise Exception("Method '%s' not registered for model '%s'" % (method, model))

Exception Type: Exception at /operations/setup/group/
Exception Value: Method 'test' not registered for model 'operations.group'

Just an issue to know if anyone plan to make this contribution ? One can see 
that the projet is not updated anymore since a while. Since I will integrate it 
to a django project, I would like to know if you are interested to add me to 
the project in order to bring my contributions in this repos otherwise I have 
planned to make a fork on github.

Thanks.
Caner

I've attached three patches which improve mainly the pidfile handling and 
Windows compatibility.

01_pidfile_setting.patch - Adds a settings variable for the pidfile (similarly 
to the one for the logfile). This enables running multiple instances of taskd 
on the same machine.

02_logfile_tmp_path.patch - Adds the same logic for determining the temp path 
for the logfile as the pidfile uses.

03_remove_stale_pidfile.patch - Removes the stale pidfile on startup. Also adds 
better support for checking whether a process exists with a given pid. This 
works on Windows too, although it requires psutil (but only on Windows).

I am using an early june 2010 svn export of django-tasks (probably r12) and any 
traceback in the log output contains ANSI color escape codes if ipython is 
installed.

Uninstalling ipython resolves the issue, but its probably possible for those 
color tracebacks to be disabled by the code running the tasks.

django-tasks has been working well for me so far.  Great job guys.

I started the task server and added the following to my model:

    def task_do_something(self):
        self.results = process_info(self.id)
        self.save()

Then call:
djangotasks.register_task(MyModel.task_do_something, "Add data returned from 
function")

But in my view, I get an exception when calling:
        task = djangotasks.task_for_object(MyModel.task_do_something)

In the log, I see no indication that the "MyModel.task_do_something" was 
loaded/registered.  Am I using this properly?

Nothing is shown in the log, just that exception in the view. It seems like the 
task isn't registering properly.

Hi Guys, its been working well for me given my limited usage at this point.  It 
would be cool if django-tasks sent email notification when something finishes 
with error.

Austin

The attached patch contains a couple of fixes needed to use djangotask in 
Windows. Namely,

 - The PYTHONPATH separator is ';' on windows, thus it must be set differently depending on the os.
 - Similarly, windows version of subprocess.Popen() doesn't support close_fds=True together with stdout/stderr redirections.
 - There are some hard coded Unix specific file names in the source, that must be dealt with. I couldn't really find good standard windows alternatives for them, thus:
   - I made django-taskd.pid file configurable with settings.TASKS_PID_FILE. This can actually be useful on linux, too.
   - The standard paths in become_daemon() were overwritten when used in taskd.py. It didn't really make sense even on Linux, thus I removed them.
 - Finally, I also fixed a buggy exception text in models.py. (It does not affect windows compatibility.)

What steps will reproduce the problem?
1. Install Django > 1.5
2. run pip install django-tasks

What is the expected output? What do you see instead?
I expect the installation to notice that I have Django 1.5.5 installed and 
proceed without downgrading the existing installation. Instead pip removes 
Django 1.5.5 and installs 1.5 instead

What version of the product are you using? On what operating system?
django-tasks-0.3 installation with "pip 1.5.2 from 
/usr/lib/python2.6/site-packages/pip-1.5.2-py2.6.egg (python 2.6)" on Amazon 
Linux

Please provide any additional information below.
Copy of installation output
[ec2-user]$ sudo pip install django-tasks
Downloading/unpacking django-tasks
  Downloading django-tasks-0.3.tar.gz
  Running setup.py (path:/tmp/pip_build_root/django-tasks/setup.py) egg_info for package django-tasks
Downloading/unpacking django==1.5 (from django-tasks)
  Downloading Django-1.5.tar.gz (8.0MB): 8.0MB downloaded
  Running setup.py (path:/tmp/pip_build_root/django/setup.py) egg_info for package django
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Requirement already satisfied (use --upgrade to upgrade): south in 
/usr/lib64/python2.6/site-packages (from django-tasks)
Installing collected packages: django-tasks, django
  Running setup.py install for django-tasks
  Found existing installation: Django 1.5.5
    Uninstalling Django:
      Successfully uninstalled Django
  Running setup.py install for django
    changing mode of build/scripts-2.6/django-admin.py from 644 to 755
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
    changing mode of /usr/bin/django-admin.py to 755
Successfully installed django-tasks django
Cleaning up...

With thread (i.e. 'runserver'), we should show the scheduler log on the 
standard output

Today it's Logged to "/tmp/django-tasks.log". It should be configurable.

When a task fail, the message is "finished with error" which is not clear - we 
had user who still expected that (even though there was an error) the output of 
the task woudl be available. 

Maybe a better wording would be "failed"

./manage.py taskd stop
[Errno 1] Operation not permitted

Is anyone actively using this project?  It seems to work ok after my very 
simple inspection.  It is probably good enough for my immediate purposes and I 
might just proceed and use it.  I would be happy to contribute back some 
additional documentation or any fixes or issues I come across.

Austin

What steps will reproduce the problem?

1. Start multiple tasks longer that take longer than 5 seconds 

What is the expected output? What do you see instead?

Only one tasks should start, and the remainding tasks should wait for it to 
finish. However, every 5 seconds, a new task is started regardless of how many 
tasks are already running.

What version of the product are you using? On what operating system?

r51, using Django 1.6 on Ubuntu 12.04 Server.

Please provide any additional information below.

I've attached a patch that checks for running tasks and only starts a new task 
if there aren't any running tasks.

...that will display the current 'running' and 'scheduled' tasks in the console.

test failure:

======================================================================
FAIL: test_tasks_run_required_task_successful (djangotasks.tests.ViewsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/private/tmp/bittenlmv3C-/build_toolkit-with-spaces_35919/trunk/subdir 
with space/portal/djangotasks/tests.py", line 403, in 
test_tasks_run_required_task_successful
u'Run a task with a required task finished successfully on ', complete_log)
AssertionError: u'Run a successful task started on \nrunning 
something...\nstill running...\n\nRun a successful task finished successfully 
on \nRun a task with a required task started on \nrunning required...\n\nRun a 
task with a required task finished successfully on ' != u'Run a successful task 
started on Aug. 8, 2010 at 9:33 a.m.\nrunning something...\nstill 
running...\n\nRun a successful task finished successfully on Aug. 8, 2010 at 
9:33 a.m.\nRun a task with a required task started on Aug. 8, 2010 at 9:33 
a.m.\nrunning required...\n\nRun a task with a required task finished 
successfully on Aug. 8, 2010 at 9:33 a.m.'


May be the assertion should be more flexible to ignore the time stamps?

I've set up django-tasks as described on the main page. In order to get all 
tests pass I had to do the following modifications:

1. Since not all time zone abbreviations are three letters long, I had to apply 
the attached patch.
2. I had to set the logging level to INFO in settings.py.
3. I had to set the DJANGO_SETTINGS_MODULE and PYTHONPATH environment variables 
appropriately.

Is there a way to modify the testing code so that 2. and 3. would not be 
necessary?

Following issue 9, it will be nice to format the message with the timestamp, 
like django does.
We already logging the time with the log message in the log file when 
djangotask run as a daemon.

So when using "runserver", instead of having:

[07/Dec/2010 09:10:47] "GET /admin/custom_media/img/favicon.ico HTTP/1.1" 200 
485
Task 41 finished with status "successful"
Starting task 42...

we will have:

[07/Dec/2010 09:10:47] "GET /admin/custom_media/img/favicon.ico HTTP/1.1" 200 
485
[07/Dec/2010 09:10:48] Task 41 finished with status "successful"
[07/Dec/2010 09:10:49] Starting task 42...

What steps will reproduce the problem?
1. Install MySQL and Django
2. Run Djangotask
3. Add some task to the scheduler

What is the expected output? What do you see instead?
The Scheduler executed just one task.

See log:
I added a print statement to djangotasks/models.py:282
 # ... Then start any new task
        tasks = self.filter(status="scheduled",
                            archived=False)
        print tasks
        for task in tasks:


INFO:djangotasks:Starting task 18...
INFO:djangotasks:...Task 18 started.
INFO:djangotasks:Task 18 finished with status "successful"
[<Task: 18 - video.9.create_bittorrent>] 
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
INFO:djangotasks:Starting task 18...
INFO:djangotasks:...Task 18 started.
WARNING:djangotasks:Failed to change status from "scheduled" to "running" for 
task 18
[<Task: 18 - video.9.create_bittorrent>] 
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
INFO:djangotasks:Starting task 18...
INFO:djangotasks:...Task 18 started.
WARNING:djangotasks:Failed to change status from "scheduled" to "running" for 
task 18
[<Task: 18 - video.9.create_bittorrent>]
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
WARNING:djangotasks:A task on model=videoportal.video exists in the database, 
but is not defined in the code
[..every 5 sec]

As you can see the scheduler thinks the task has stil the status "scheduled"  
allthough it was succesfully processed before. The scheduler gets the status 
before the transaction in djangotasks/models.py:288.

tasks = self.filter(status="scheduled",
                            archived=False)


What version of the product are you using? On what operating system?
mysqld --version
mysqld  Ver 5.5.28-0ubuntu0.12.04.3 for debian-linux-gnu on x86_64 ((Ubuntu))
python --version
Python 2.7.3
django-tasks from
https://github.com/Piratenfraktion-Berlin/djangotasks

Please provide any additional information below.
The issue is related to Mysql and Django: 
http://stackoverflow.com/questions/3346124/how-do-i-force-django-to-ignore-any-c
aches-and-reload-data

adding the READ COMMITED Isolation to the DATABASES OPTIONS solves the issue

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': '',                      # Or path to database file if using sqlite3.
        'USER': '',                      # Not used with sqlite3.
        'PASSWORD': '',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
        'OPTIONS': { "init_command": "SET storage_engine=INNODB, SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED", }
   }
}

[could not find any contact information, so using the issue system]

I am looking for exactly this project: a django based super minimal task 
manager.

However, this is one thing I am uncomfortable with: you can not pass argument 
to ask.

I have for example daily and weekly tasks, that will obviously take as a date 
or a week number. How to achieve that ?