Currently two tools (comet-ml and neptune.ai) are integrated as third party experiment savers/loggers.

Other tools we would like to integrate with:

• mlflow
• Weights & biases

You can see current work in callbacks module

TO-DO BEFORE 0.1.0 RELEASE:

• Integration tests before 0.1.0 release
• Implementation of the above two integrations

Other suggestions for similar integrations are more than welcome. Also if you have an idea for other non-experiment logging callbacks worth checking out (and possibly integrating), list those below as well, thanks.

What and why

pytorch-lightning has separate module responsible for accelerating PyTorch's training (multi-GPU, distributed across multiple nodes etc.).

Obviously users would benefit from similar functionality.

API

Currently torchtrain.steps, torchtrain.iterations, torchtrain.epochs cannot have anything piped into it, e.g. this is not allowed:

class TrainStep(tt.steps.Train):
    def forward(self, module, sample):
        ...
        return loss, predictions, labels

# This would make no sense
tt.metrics.regression.SquaredError() > TrainStep(criterion, device)

On the other hand accelerators would fit quite nicely here (I think):

class TrainStep(tt.steps.Train):
    def forward(self, module, sample):
        ...
        return loss, predictions, labels

# This way
tt.accelerators.PyTorch(...) > TrainStep(criterion, device)
# or this way
TrainStep(criterion, device) < tt.accelerators.PyTorch(...)

Current stage

• WIP horovod integration is more or less in-place, see accelerators module

TO-DO BEFORE 0.1.0:

• test existing integrations
• Add torch.nn.parallel.DistributedDataParallel (help wanted)
• Add ray integration
• Outline which one should be used when

Discussion

• What other accelerators would be reasonable?
• How other accelerators fit in this library? Are custom steps, iterations needed? How (un)flexible those are?
• What should the API for them be? Currently tt.accelerators.Horovod just has to be created, no need to pipe it into anything (though it's left like that for clarity).