Deploy worker on Kubernetes about data-transfer-project HOT 2 CLOSED

Initial setup for worker pool should be pretty similar to API pool (except no backend service, as no traffic needs to be directed to workers)

Workers are configured to shut themselves down after processing a job, and our Kubernetes pod restart policy is configured to Always restart, so we should be good for refilling the initial pool of workers as they process jobs

For scaling, I think we should try Kubernetes autoscaler, which has a beta feature to scale based on a custom metric. We could make that use whether the worker is processing a job (i.e. it's at capacity) or not. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

from data-transfer-project.

This is working now with a fixed pool size. More advanced autoscaling work can be tracked in a separate issue.

from data-transfer-project.