marrow / task Goto Github PK

Allow scheduling of tasks for execution using the Futures API, to allow for easy drop-in migration.

• The executor API should be class methods of the Task model.
• Should, in the end, return Task model instances wrapped in a Futures-compatible translation object that are compatible with the built-in concurrent.futures utilities such as as_completed. See #4.

pip install marrow.task doesn't work :-]

Wrap Task instances in a Future-compatible API. These are what will be returned by the Executor-compatible object when scheduling tasks.

Utilize Futures task pooling (configurable thread or process pool) to schedule tasks for execution as they arrive. Use MongoDB atomic locking to ensure only one instance of the runner actually executes the task.

• Initial version using a basic console_scripts entry point that when started configures logging, database connections, and Futures executor via YAML, then blocks forever waiting on new tasks.
• New tasks should be submitted to the Futures executor (i.e. process pool or thread pool).
• Task locking in this version following the "first to lock the task wins" approach, with graceful handling for failing that race.

Construct a light-weight and elegant API for requesting local, deferred, or scheduled execution of callable objects.

• See examples for desired interactions.
• See these presentation slides for an overview of the process.
• See this naive implementation for execution flow.
• Utilize marrow.package for callable path canonicalization and resolution.
• These should be built-in methods of the Task model.
• Should be built with Futures-compatible wrappers in mind.
• Provide a wrapt-based decorator suitable for optional deferring by default of functions and class methods; default is to not defer. (See examples.)
• Task methods that check for completion/return values/exceptions should take timeout values; until MongoDB SERVER-15815 is resolved these timeouts will not be particularly accurate. This is acceptable. (No timeout = wait forever.)
• Utilize and extend the capped collection queryset already provided.

Allow three forms of execution:

• Immediate. I.e. calling a function or method with the default of "defer=False" from the decorator, or calling the "call" method of a "defer=True" decorated callable.
• Deferred. I.e. calling a function or method with a "defer=True" decorated callable, or calling the "defer" method of a "defer=False" decorated callable.
• Scheduled. I.e. calling the "at" method of decorated callable.

If the callable is a standard one (i.e. not a generator) when the function returns the value would be $set on the Task and a message emitted. If the function is a generator, $push each value to the result in the Task and send a message per yielded chunk, submitting all "completion callbacks" once for each chunk. If the callable is a generator, attempting to get the result of execution client-side would return a wrapper object (passing the timeout through) which is iterable.