josiahcarlson / rpqueue Goto Github PK

Shouldn't this be setting the score to be new_t rather than ARGV[1]? It seems like the current behavior would just immediately expire every task instead.

Thanks

When a task finishes, it may return something.

Offer the ability to store the result into a tombstone, which can then be referenced by the caller.

We removed certain cron_tasks and restarted our rpqueue server. The tasks remained scheduled and when their time came, a KeyError arose in _get_work() during the following if branch:

if item_id == work[1]:
    # periodic or cron task, reschedule it
    sch = work[-1] if item_id in NEVER_SKIP else time.time()
    delay = REGISTRY[item_id].next()  # KeyError here

All of our worker threads died as a result of this KeyError.

In principle this can be avoided, but we had no way to predict it nor reason to expect it. It seems to me a warning rather than a full exception is a more appropriate response. Other parts of rpqueue server code do check for the function name before acting on it, so it makes sense to do so here as well.

PR coming shortly.

Is it possible that the rpqueue.run method or the whole project is not compatible with Python3? I wanted to start the execution daemon and got the following error message:

File "/virtenv/lib/python3.6/site-packages/rpqueue/__init__.py", line 881, in execute_tasks
    for p in xrange(processes):
NameError: name 'xrange' is not defined

After switching to python2 the executor started as expected.
If so you should probably mention it somewhere in the documentation. Otherwise I would love to know why it does not work for me.

Thanks!

Add the ability to automatically retry a task if it has taken longer than a specified number of seconds to execute within the task processor.

We spent an awful long time debugging insane defaults and a printout akin to " scheduled to execute every 9999999 seconds" would have helped.

No, seriously, I should be ashamed for the awful logging configuration that rpqueue uses by default. I need to fix that.

We have a priority task queue implementation in Nodejs https://github.com/LearnBoost/kue which uses a pattern like event notification sketch in http://redis.io/commands/blpop but thats not atomic and the helper list may be inconsistent with the actual tasks sorted set...
To overcome this I was thinking of using BLPOPRPUSH (to add reliability) and keep the list sorted as the main task container (atomicity) to provide priorities.
So I was thinking of using a background helper sorted set, using prioidical calls to SORT...STORE to keep the main list sorted. (but SORT...STORE doesn't trigger blocking clients)
until I faced your implementation which is very near to my mind. I think I can something like _handle_delayed_lua but I doubt I should use a small LIMIT, since large one can hurt new task priorities not being respected. (mine is not time-based, instead weighted tasks)

This change introduces a new bug:
6d9caa7#diff-2cee58761fd6da04a3218ebb31016238
To reproduce use @task decorator without parameters:

@rpqueue.task
def task1(a):
    saw[0] = a

You will get error:

 File "/opt/chownow/venv-hermosa/local/lib/python2.7/site-packages/rpqueue/__init__.py", line 1364, in execute_tasks
   __import__(module) # for any connection modification side-effects
 File "/opt/chownow/Hermosa/_tasks.py", line 7, in <module>
   import tasks
 File "/opt/chownow/Hermosa/tasks/__init__.py", line 7, in <module>
   from . import billing
 File "/opt/chownow/Hermosa/tasks/billing.py", line 10, in <module>
   @close_session
 File "/opt/chownow/venv-hermosa/local/lib/python2.7/site-packages/rpqueue/__init__.py", line 1233, in task
   return decorate(args[0])
TypeError: 'function' object has no attribute '__getitem__'

*args is a list of parametrs and args is just one parameter
This code will throw exception: return decorate(args[0])

Add the ability to put a task in a different queue. Can be used for re-prioritization, etc.

Ref : https://www.python.org/dev/peps/pep-0632/#migration-advice

setup from setuptools can be used.

Where is the documentation around the integer passed to save_results, maybe
https://github.com/josiahcarlson/rpqueue/blame/master/README.rst#L161