boxine / django-huey-monitor Goto Github PK
View Code? Open in Web Editor NEWDjango based tool for monitoring huey task queue: https://github.com/coleifer/huey
License: GNU General Public License v3.0
Django based tool for monitoring huey task queue: https://github.com/coleifer/huey
License: GNU General Public License v3.0
Hi, could you proceed with the merge of #44 ?
One month has passed since linting preferences have been adapted according to your preferences
Thanks in advance
Hey,
I am currently testing huey with this monitor.
The QuickStart guide should contain that after installing it needs a python manage.py migrate
.
Without it you will be greeted with an error message in the Django admin interface.
For the single list page of Task model in Admin, it takes 300+ queries for fetching query set.
I have eliminated a few columns and make some query optimizations, then the number of queries drops to only 11. Now it's smooth to load the list page.
I wish subsequently this project can have some optimization.
Hi,
I had a strange error error - (1406, "Data too long for column 'desc' at row 1")
After browsing the net, and then exploring the code of huey-monitor,
I finally manage to realize the reason my task was failling was that the description I was setting for my ProcessInfo
was 67 char while only 64 were authorized by the TaskModel
model.
This error was quite small, and may happen easily ... but it was quite hard to find-out.
I therefore would suggest to make this error much more easy to find-out for users.
One solution could be to implement a custom validator for the TaskModel.desc
field to document specifically the error for the user.
However, having a desc
bigger than 64 char should not for me prevent a task to run.
I would therefore suggest another simpler solution:
to implement in the ProcessInfo
model a crop to 64 char when updating TaskModel
.
A minimal implementation could be:
TaskModel.objects.filter(task_id=task.id).update(
desc=self.desc[:64],
total=self.total,
unit=self.unit,
unit_divisor=self.unit_divisor,
)
The version 0.7.0 gives a ConnectionError when using this library with django_huey
's multi-queues.
Error seems to be here:
@register.simple_tag
def huey_counts_info():
context = dict(
huey_pending_count=HUEY.pending_count(),
huey_scheduled_count=HUEY.scheduled_count(),
huey_result_count=HUEY.result_count(),
)
It tries to connect to a non-existing queue
Hi,
I've been implementing django-huey with django-huey-monitor.
It starts to do more or less what I want (please look also into the discussion thread #28).
... until I started to launch some testing on a totally unrelated part of my application.
And there, surprise, I have an avalanche of errors linked to django-huey-monitor during the setup of the Testing environement (cf Traceback below).
There is not much about huey_monitor on the net or in term of documentation, so I'll greatly appreciate I you could help me out.
Error occurred sending signal "executing"
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: huey_monitor_taskmodel
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 274, in _emit
self._signal.send(signal, task, *args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey\signals.py", line 41, in send
receiver(signal, task, *args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey_monitor\tasks.py", line 49, in store_signals
defaults={'name': task.name}
File "C:\Users\...\venv\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 581, in get_or_create
return self.get(**kwargs), False
File "C:\Users\...\venv\lib\site-packages\nplusone\core\signals.py", line 23, in wrapped
ret = func(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 431, in get
num = len(clone)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 262, in __len__
self._fetch_all()
File "C:\Users\...\venv\lib\site-packages\nplusone\ext\django\patch.py", line 295, in fetch_all
original_fetch_all(self)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1169, in execute_sql
cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: huey_monitor_taskmodel
Unhandled exception in task 20046722-1606-4d32-80a2-3f88ed08d4fe.
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: huey_monitor_taskmodel
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 367, in _execute
task_value = task.execute()
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 740, in execute
return func(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey\contrib\djhuey\__init__.py", line 133, in inner
return fn(*args, **kwargs)
File "C:\Users\...\goodies\decorators\debug.py", line 74, in wrapper
result = func(*args, **kwargs)
File "C:\Users\...\puzzles\tasks\puzzles.py", line 49, in recalculate_pzl_medians
unit='puzzles',
File "C:\Users\...\venv\lib\site-packages\huey_monitor\tqdm.py", line 47, in __init__
unit_divisor=self.unit_divisor,
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 783, in update
rows = query.get_compiler(self.db).execute_sql(CURSOR)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1535, in execute_sql
cursor = super().execute_sql(result_type)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1169, in execute_sql
cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: huey_monitor_taskmodel
Error occurred sending signal "error"
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: huey_monitor_taskmodel
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 367, in _execute
task_value = task.execute()
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 740, in execute
return func(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey\contrib\djhuey\__init__.py", line 133, in inner
return fn(*args, **kwargs)
File "C:\Users\...\goodies\decorators\debug.py", line 74, in wrapper
result = func(*args, **kwargs)
File "C:\Users\...\puzzles\tasks\puzzles.py", line 49, in recalculate_pzl_medians
unit='puzzles',
File "C:\Users\...\venv\lib\site-packages\huey_monitor\tqdm.py", line 47, in __init__
unit_divisor=self.unit_divisor,
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 783, in update
rows = query.get_compiler(self.db).execute_sql(CURSOR)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1535, in execute_sql
cursor = super().execute_sql(result_type)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1169, in execute_sql
cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 423, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: huey_monitor_taskmodel
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\huey\api.py", line 274, in _emit
self._signal.send(signal, task, *args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey\signals.py", line 41, in send
receiver(signal, task, *args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\huey_monitor\tasks.py", line 49, in store_signals
defaults={'name': task.name}
File "C:\Users\...\venv\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 581, in get_or_create
return self.get(**kwargs), False
File "C:\Users\...\venv\lib\site-packages\nplusone\core\signals.py", line 23, in wrapped
ret = func(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 431, in get
num = len(clone)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 262, in __len__
self._fetch_all()
File "C:\Users\...\venv\lib\site-packages\nplusone\ext\django\patch.py", line 295, in fetch_all
original_fetch_all(self)
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "C:\Users\...\venv\lib\site-packages\django\db\models\query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "C:\Users\...\venv\lib\site-packages\django\db\models\sql\compiler.py", line 1169, in execute_sql
cursor.execute(sql, params)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 78, in _execute
self.db.validate_no_broken_transaction()
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\base.py", line 448, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
Traceback (most recent call last):
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\executor.py", line 229, in apply_migration
self.record_migration(migration)
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\executor.py", line 244, in record_migration
self.recorder.record_applied(migration.app_label, migration.name)
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\recorder.py", line 86, in record_applied
self.ensure_schema()
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\recorder.py", line 63, in ensure_schema
if self.has_table():
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\recorder.py", line 56, in has_table
tables = self.connection.introspection.table_names(cursor)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\introspection.py", line 52, in table_names
return get_names(cursor)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\introspection.py", line 47, in get_names
return sorted(ti.name for ti in self.get_table_list(cursor)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\introspection.py", line 77, in get_table_list
ORDER BY name""")
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 78, in _execute
self.db.validate_no_broken_transaction()
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\base.py", line 448, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 23, in <module>
main()
File "manage.py", line 19, in main
execute_from_command_line(sys.argv)
File "C:\Users\...\venv\lib\site-packages\django\core\management\__init__.py", line 419, in execute_from_command_line
utility.execute()
File "C:\Users\...\venv\lib\site-packages\django\core\management\__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\...\venv\lib\site-packages\django\core\management\commands\test.py", line 23, in run_from_argv
super().run_from_argv(argv)
File "C:\Users\...\venv\lib\site-packages\django\core\management\base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\...\venv\lib\site-packages\django\core\management\base.py", line 398, in execute
output = self.handle(*args, **options)
File "C:\Users\...\venv\lib\site-packages\django\core\management\commands\test.py", line 55, in handle
failures = test_runner.run_tests(test_labels)
File "C:\Users\...\venv\lib\site-packages\django\test\runner.py", line 725, in run_tests
old_config = self.setup_databases(aliases=databases)
File "C:\Users\...\venv\lib\site-packages\django\test\runner.py", line 645, in setup_databases
debug_sql=self.debug_sql, parallel=self.parallel, **kwargs
File "C:\Users\...\venv\lib\site-packages\django\test\utils.py", line 183, in setup_databases
serialize=connection.settings_dict['TEST'].get('SERIALIZE', True),
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\creation.py", line 79, in create_test_db
run_syncdb=True,
File "C:\Users\...\venv\lib\site-packages\django\core\management\__init__.py", line 181, in call_command
return command.execute(*args, **defaults)
File "C:\Users\...\venv\lib\site-packages\django\core\management\base.py", line 398, in execute
output = self.handle(*args, **options)
File "C:\Users\...\venv\lib\site-packages\django\core\management\base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "C:\Users\...\venv\lib\site-packages\django\core\management\commands\migrate.py", line 246, in handle
fake_initial=fake_initial,
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "C:\Users\...\venv\lib\site-packages\django\db\migrations\executor.py", line 230, in apply_migration
migration_recorded = True
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\schema.py", line 35, in __exit__
self.connection.check_constraints()
File "C:\Users\...\venv\lib\site-packages\django\db\backends\sqlite3\base.py", line 329, in check_constraints
violations = cursor.execute('PRAGMA foreign_key_check').fetchall()
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "C:\Users\...\venv\lib\site-packages\django\db\backends\utils.py", line 78, in _execute
self.db.validate_no_broken_transaction()
File "C:\Users\...\venv\lib\site-packages\django\db\backends\base\base.py", line 448, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
Hello, would it be possible to add a view to this package that would display info about huey ?
Like connection, queue size, schedule size etc. Unless there's a way to see that info and I've missed it....
Hi, when going through the Tasks
admin, I find it pretty difficult to know what the task is about as only the task name is availlable:
Through this view, we cannot know which task failed (and should trigger a corrective action) and which worked (and can eventually be deleted).
To have an idea about what the tasks are about, we need to edit each task until we find the right one:
then, first line is the only place where the task description (desc
) is mentionned.
same with sub-tasks within the changelist view:
I would suggest to :
1/ display desc
as the first parameter in list_view
2/ display desc
as a parameter in change_view (in addition to the current title
)
I'll be proposing a PR to implement this.
PS: What is your preference? that I continue creating Issues and corresponding PR ? or that I create PR directly, without Issue?
Hi @jedie,
I had a bad surprise this morning while checking on the progress of a process which had it's first sub-process running for over than 24h.
There was an error: django.db.utils.OperationalError: (1114, "The table 'huey_monitor_taskprogressmodel' is full")
Then I realized that for each tiny progress report, a record is created for TaskProgressModel
Could you please tell me what is the benefit of creating a new TaskProgressModel
(even two when you want to report progress on both sub-task and maintask) each time you report a progress vs. incrementing directly the progress_count
field of an existing TaskProgressModel
?
That would mean that in general, we would have only one TaskProgressModel
for each TaskModel
.
Thanks in advance for your feedback
In the task model, a lot of info is saved which is very useful. But in a production scenario, it will also be super helpful if the args passed to the task function are also stored. If a task fails, the error message is there but it becomes difficult to map it to the input. If you think this is useful, I would be willing to work on a PR for this.
Thanks for the awesome project! I have a question that might be out of scope, but I'd appreciate any pointers.
I've been using Huey in a django project without huey.contrib.djhuey
, so that I can run a huey_consumer
on a different machine where the django application is not fully set up (the code is obviously accessible). I've organized my code as described here and on the other machine, I start a worker like this: REDIS_HOST=172.0.20.2 huey_consumer app.huey.main.huey
. This works great and makes it so that I don't have to install anything on the worker node other than huey and redis-py.
However, I can't figure out a way to wire up huey-monitor to all of this. Is there a mechanism by which I can provide it with a huey instance?
Thanks!
I use celery with django. want to come over to huey.
in celery, it separates broker/queue from result storage.
so by using django-huey-monitor (dhm) are you making huey use the database purely for result storage? and leave redis to handle the broker/queue?
We should implement a kind of https://pypi.org/project/tqdm/ for long running task like: "Do something with 1000 items"...
Can we so something like the concept of tqdm and use the django cache to collect these kind of progess information?
Thank you again for this lib :)
I would like to see this fix soon be integrated.
#45
Could you please provide a new release or are you waiting for the completion of a new feature first?
In order to allow us do some regular house-cleaning, this extension should have a command to flush old (no longer relevant) task data.
Would be even more helpful if we could specify the number of days to keep,
something like:
python3 manage.py huey_monitor_flush 30
would delete task data older than 30 days.
This would be a great addition, as task data, if left unchecked, will start to grow larger and larger.
When clicking on a task-name this url is linked /admin/huey_monitor/taskmodel/<taskid>/change/
. I then get the following error message.
'humanize_time' is not a registered tag library. Must be one of:
admin_list
admin_modify
admin_urls
cache
debugger_tags
guardian_tags
highlighting
i18n
import_export_tags
indent_text
l10n
log
rest_framework
static
syntax_color
tz
widont
Hi,
My tasks are organised as a set of sub-tasks
completion of each sub-task is adding 1 to the progress of the parent task
however if you look at the first snapshot that shows only the columns visible without scrolling, one could think that all activities are complete and that it would be ok to switch off the server.
But if you scroll right to see the progression data, the situation is very different:
At the moment, the status of the Parent task is not related at all to the task progression (TaskModel.progress_info[0]
vs. TaskModel.total
)
In my use case, that would be necessary to take task progression into account when a total is specified.
How could I ensure that the "signal_complete" is not sent before each sub-task is complete?
and that it is sent once the last sub-task has been completed?
Hi,
In both TaskModel and SignalInfoModel, the project uses CharField with a specified max_length
. While this is okay for most fields, I was thinking that this could be a limitation for exception_line
field of the SignalInfoModel
. Can we move to TextField for that field and remove the max_length
limitation?
Also, separately, if target database is Postgres, it seems to not be an issue and infact text
is more perfmant than char
: https://www.postgresql.org/docs/9.0/datatype-character.html
Regards,
TaskModelAdmin
references a CSS file called huey_monitor.css
. This file doesn't actually exist, which causes problems if you're using Django's ManifestStaticFilesStorage
or something inheriting from it like Whitenoise's CompressedManifestStaticFilesStorage
. Specifically you end up with a ValueError and a 500 when navigating to the admin task list in production, unless you tweak the manifest_strict setting. I'm not sure if this CSS thing is leftover from something or if it's meant to be an optional file that can be added by users of the library to customize how the Huey admin looks. In the former case, could we remove it so that we don't have to disable manifest_strict
? In the latter case, is disabling manifest_strict
the intended workaround for this issue?
In #10 we add a work-a-round for died Huey worker.
But this is no a good solution, because non affected, running tasks will also get a "unknown" signal :(
See also: coleifer/huey#663
Maybe:
HUEY.flush_locks()
Environment:
Request Method: GET
Request URL: http://127.0.0.1:7000/admin/huey_monitor/taskmodel/e2fe16e4-2d57-45b5-95ae-db99b01b4659/change/
Django Version: 3.2.21
Python Version: 3.9.18
Installed Applications:
[...
'bx_django_utils',
'huey_monitor',
..]
Traceback (most recent call last):
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 710, in get_form
return modelform_factory(self.model, **defaults)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\forms\models.py", line 563, in modelform_factory
return type(form)(class_name, (form,), form_class_attrs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\forms\models.py", line 276, in __new__
raise FieldError(message)
During handling of the above exception (Unknown field(s) (task_hierarchy_info, human_progress_string) specified for TaskModel), another exception occurred:
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\core\handlers\exception.py", line 47, in inner
response = get_response(request)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\core\handlers\base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 616, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\sites.py", line 232, in inner
return view(request, *args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 1660, in change_view
return self.changeform_view(request, object_id, form_url, extra_context)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\utils\decorators.py", line 43, in _wrapper
return bound_method(*args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\utils\decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 1540, in changeform_view
return self._changeform_view(request, object_id, form_url, extra_context)
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 1574, in _changeform_view
ModelForm = self.get_form(
File "D:\Users\x\mambaforge\envs\production\lib\site-packages\django\contrib\admin\options.py", line 712, in get_form
raise FieldError(
Exception Type: FieldError at /admin/huey_monitor/taskmodel/e2fe16e4-2d57-45b5-95ae-db99b01b4659/change/
Exception Value: Unknown field(s) (task_hierarchy_info, human_progress_string) specified for TaskModel. Check fields/fieldsets/exclude attributes of class CustomTaskModelAdmin.
from django.contrib import admin
from huey_monitor.models import TaskModel, SignalInfoModel
from huey_monitor.admin import TaskModelAdmin, SignalInfoModelAdmin
admin.site.unregister(TaskModel)
admin.site.unregister(SignalInfoModel)
class CustomTaskModelAdmin(TaskModelAdmin):
def has_view_permission(self, request, obj=None):
return request.user.is_superuser
def has_delete_permission(self, request, obj=None):
return request.user.is_superuser
def has_change_permission(self, request, obj=None):
return request.user.is_superuser
def has_add_permission(self, request, obj=None):
return request.user.is_superuser
def has_module_permission(self, request):
return request.user.is_superuser
admin.site.register(TaskModel, CustomTaskModelAdmin)
class CustomSignalInfoModelAdmin(SignalInfoModelAdmin):
def has_view_permission(self, request, obj=None):
return request.user.is_superuser
def has_delete_permission(self, request, obj=None):
return request.user.is_superuser
def has_change_permission(self, request, obj=None):
return request.user.is_superuser
def has_add_permission(self, request, obj=None):
return request.user.is_superuser
def has_module_permission(self, request):
return request.user.is_superuser
admin.site.register(SignalInfoModel, CustomSignalInfoModelAdmin)
My PR has been closed without being merged.
My code is plenty of patches since begining of september because I'm waiting for you to validate the code.
You make me wait 6 months to change the formating of the code while it would take you less than 5 minutes to do it.
I have to wait for 6 month ...
and then you decide to kill everything because you have another idea ... that is not even yet implemented !
Please first merge my PR, so that I can have some code that works correctly and get rid of the patches.
Then when you have the time, there is no problem to implement your new idea.
But at least in the meantime, I can have a code that work, without monkey patch.
and also, it will be easy to test if your new idea is still functional for tasks that takes few days or few weeks to execute
Hi @jedie,
I realized that when tasks are running with subtasks, the Last update
value is not updated when one of the sub-task is updated (as shown by the screenshot below).
Therefore, it is hard without opening the subtask to know if something has happened or not, or to get any sense of progress.
It would be great if the Last update
value could also be updated each time one of the sub-task is updated, and not only when the main-task is updated (could be days in my case).
Thanks in advance
Hello,
We have a production setup with lots of workers running in parallel and a deadlock error occur from time to time, I think when two workers are trying the update the same task at the same moment. Here is the traceback of the error:
DeadlockDetected: deadlock detected
DETAIL: Process 3256555 waits for ShareLock on transaction 3220907091; blocked by process 3256561.
Process 3256561 waits for ShareLock on transaction 3220907090; blocked by process 3256555.
HINT: See server log for query details.
CONTEXT: while updating tuple (1239,37) in relation "huey_monitor_taskmodel"
File "django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
OperationalError: deadlock detected
DETAIL: Process 3256555 waits for ShareLock on transaction 3220907091; blocked by process 3256561.
Process 3256561 waits for ShareLock on transaction 3220907090; blocked by process 3256555.
HINT: See server log for query details.
CONTEXT: while updating tuple (1239,37) in relation "huey_monitor_taskmodel"
File "huey/consumer.py", line 94, in initialize
startup_hook()
File "huey_monitor/tasks.py", line 122, in startup_handler
update_task_instance(
File "huey_monitor/tasks.py", line 45, in update_task_instance
instance.save(update_fields=update_fields)
File "bx_django_utils/models/timetracking.py", line 59, in save
super().save(**kwargs)
File "django/db/models/base.py", line 814, in save
self.save_base(
File "django/db/models/base.py", line 877, in save_base
updated = self._save_table(
File "django/db/models/base.py", line 990, in _save_table
updated = self._do_update(
File "django/db/models/base.py", line 1054, in _do_update
return filtered._update(values) > 0
File "django/db/models/query.py", line 1231, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "django/db/models/sql/compiler.py", line 1982, in execute_sql
cursor = super().execute_sql(result_type)
File "django/db/models/sql/compiler.py", line 1560, in execute_sql
cursor.execute(sql, params)
File "django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
Any idea on how to prevent this?
We are using postgreSQL if it matters.
Refering to issue #91,
I have implemented locally a few helpers which enable better monitoring of progression, especially useful in the case of recursive sub-tasks.
To document their usage, I have added functional example in https://github.com/boxine/django-huey-monitor/blob/master/huey_monitor_tests/test_app/tasks.py
Please have a look at the corresponding PR
Hi @boxine team!
Thanks for making this awesome Huey monitor open source! It's clearly one of the best options for feature-complete job/worker handling in the Django ecosystem right now, and I'm sure it will grow quickly!
I have a large existing Django project https://github.com/ArchiveBox/ArchiveBox and I'm looking for a new async job system to move our project onto. Huey + your monitor is our highest candidate right now as it supports SQLite, nested tasks with hierarchy and global mutexes, live progress info, and provides a nice UI in the django admin.
If we do this refactor, we'll likely become big contributors to this project (and I'd be happy to send a subset of our donations your way).
I'd love to chat with any of the devs who've worked on this project to get more context on your goals with it, hear about any lessons you've learned with Huey while building this, and see how we can help out if it's a good fit.
If you're down with a quick 20min video call, that would be ideal https://calendly.com/nicksweeting/choose-a-time, otherwise happy to just continue the discussion here too.
Hi Jedie,
I'm coming back to you about one of the first topics we discussed together a few years back.
For you, it is important to have cumulate_progress = True
because of the nature of the tasks you're using.
So you decided to set cumulate_progress = True
by default when creating a TaskModel
object.
for me, it is the contrary, it is important to have cumulate_progress = False
because of the nature of the tasks I'm using.
In most application, it will be either one way, or the other, but very rarely both ways in the same application.
What would you say about introducing a constant in the app.settings where we could define DEFAULT_TASK_CUMULATE_PROGRESS
to be either True
or False
in order to keep retrocompatibility, we could have cumulate_progress = settings.DEFAULT_TASK_CUMULATE_PROGRESS
or True
if there is no DEFAULT_TASK_CUMULATE_PROGRESS
in settings.py
we would also use this as the default value for ProcessInfo.cumulate2parents
Would that be something ok for you ?
Currently all web pages are static. Would be great if running task will be have live updating pages...
Hi,
I'm testing this module with --worker-type process
and 5 workers.
Huey consumer started with 5 process, PID 338935 at 2021-04-01 09:21:11.148253
Scheduler runs every 1 second(s).
Periodic tasks are enabled.
The following commands are available:
+ openrouteservice.tasks.isochrone_from_layer_task
Checking periodic tasks
Sleeping for 0.9916734490107046
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
calling startup hook "startup_handler"
startup hook "startup_handler" failed
Traceback (most recent call last):
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.OperationalError: SSL error: decryption failed or bad record mac
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ale/.local/lib/python3.8/site-packages/huey/consumer.py", line 94, in initialize
startup_hook()
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/huey_monitor/tasks.py", line 78, in startup_handler
for task_model_instance in qs:
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
cursor.execute(sql, params)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
return super().execute(sql, params)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL error: decryption failed or bad record mac
startup hook "startup_handler" failed
Traceback (most recent call last):
File "/home/ale/.virtualenvs/g3wsuite/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Note that the error does not appear when I use a single worker.
Huey has the api to define your own task_id
when scheduling a task. If we set a task_id=123
the code in huey_monitor/tasks.py: 53
it is forced the task_id
to be a uuid.
If we pass the task_id = 123 in the schedule task function, Huey_monitor signal will fail with error
ValueError
badly formed hexadecimal UUID string
Hello, happy new year.
When setting a task description like in the minimal example:
@task(context=True)
def foobar_task(task, list_to_process):
process_info = ProcessInfo(
task,
desc='A description of this Job',
total=len(list_to_process)
)
The taskname column in the admin panel no longer shows the task name. It displays the complete description.
A long description results in a very ugly admin view.
Here is an example of my functionclean_huey_db()
with desc='Huey DB Cleanup',
:
Maybe it would be usefull to seperate this:
verbose_name
. If it is set, the task_name is overwritten with the verbose one.Or am I wrong about the desc parameter?
Hi,
In SignalInfoModel, exception_line
field is not nullable, blanks are also not allowed. When storing signals in store_signals
if exc = None
, it does not look like exception_line
will be populated. Consequentially, I see:
django.core.exceptions.ValidationError: {'exception_line': ['This field cannot be blank.']}
I also noticed that in the TaskModel, field desc
(in Huey-Monitor v0.4.6) can't be blank. But the default value for the field is ''
and no value is passed when creating TaskModel in store_signals
. Consequentially, I see:
django.core.exceptions.ValidationError: {'desc': ['This field cannot be blank.']}
Package Versions:
Python: 3.9
Django: 3.2.2
Huey: 2.4.3
Huey-Monitor: 0.4.6
could you please give feedback and / or integrate PR #48 "throughput formating for rate<1" ?
It is availlable for one month already and there was not even a feedback even if it is quite straightforward.
Thanks in advance
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.