nbren12 / dask.targeted Goto Github PK

This package currently only supports functions with single outputs. Of course,
it can handle multiple outputs using a syntax like

b = delayed(get)(a, 1)
targeted(tgt, b)

It might be easier if targeted could be modified to accept a list of targets so that

b1, b2, b3 = targeted([Target(1), Target(2), Target(3)], a)

and b1, b2, b3 are all delayed objects.

The main issue here is that targeted evaluates during graph construction and could access what kind of dask object it is passed. However, if it is passed a Delayed object, it has no idea of knowing what the output type is.

On the other hand, writer is executed on task execution, so it should now what the output type is. Unfortunately, reader will not know what the output type is since it will only have access to the target object. The best solution then is probably to create a Reader/Writer object which knows how to write a given type of data to a variety of luigi targets. Then, I can dispatch targeted based on the following rules

1. targeted(dask_collection, target) is routed to a default argument for the given dask collection.
2. targeted(delayed, target, reader_writer). Delayed objects need to explicitly specify a reader_writer.