alcrene / smttask Goto Github PK

Pydantic v2 has may improvements over v1.
In particular it should make #1 obsolete, since there are no json_encoders in v2.

Most of the work for this actually needs to be done in scityping #2.

Although pickling simple minimal tasks sometimes works, this a) is untested and b) fails as soon as inputs or outputs involve dynamically created types (which are quite common in smttask). Thus we need

• Support for pickling tasks with dynamic types. These may be introduced via
  • task generators (task_generators.py)
  • multi-outputs (see base.TaskOutput.__init_subclass__)
• Comprehensive tests
  • Include all types of tasks (esp. all generated types) and both single- and multi-output tasks.
  • Test multiprocessing where:
    • Tasks are passed as arguments
    • Tasks are returned as results
    • Tasks are executed in the subprocess, and their result is returned.

The main use case I see for supporting pickle is to allow parallelizing task execution with the multiprocessing module.

Relevant SO question: Pickling dynamically generated classes?

Although many users would install them anyway, they seem superfluous for this package.

A user of Pydantic would reasonably expect that if they use a Pydantic type as Task input, the json encoders for that type would be used to encode its attributes. At present this is not the case: since Task inputs are combined into one big Pydantic BaseModel, the global encoder dict defined in smttask.typing.json_encoders is always used. Beyond being unexpected, this has at least two drawbacks:

• One needs to pollute the global json_encoders dict with all possible custom types.
• It is not possible to define different JSON encoder mappings for different task inputs.

Relevant test is added here.