diegodelemos / cap-reuse Goto Github PK

• Create Flask app
• Containerize it
• Define API for creating jobs
• Connect with k8s apiserver
• Create jobs through the microservice
  - [ ] Notify workflow engines when steps success or fail
  - [] Store stream data in cache
• Poll microservice to notice status change

Now raw requests are being sent to the k8s apiserver but ideally it would be better to choose a library that covers all the needed use cases. In case there is no such library, get the most mature/best documented/most active and make a PR adding the needed functionality.

Candidates table

This could lead to a potentially infinite number of restarts of a container, for instance, their execution may be stopped because a dependency can not be met anymore (broken link).

A workflow is composed by a set of steps which are mapped to k8s Jobs. In terms of workflows, the expected behaviour is that if a step fails a log is retrieved and the user gets a message saying that there was a problem, specifically in a certain step.

Include Redis nodes inside actual cluster in order to use it from step-broker and k8s-job-patch

If the number of Jobs created exceeds the defined ResourceQuota the waiting jobs are scheduled after a while even though there is available quota, it may be seconds, it may be minutes, but it affects the whole cluster. When Kubernetes "refresh its internal state" all the waiting jobs get executed.

Because of #3, #4 and the lack of a status notifying system for Jobs in Kubernetes(which conduced to #6) the decision of using Kubernetes has to be reconsidered.

Alternatives:

Apache Mesos with Chronos framework (by Airbnb).

• Dockerized jobs can be scheduled. However, each job itself should notify Chronos of its final state which does not fit with our use case where Docker images are treated as units of work which can not be modified.

Apache Mesos with Apache Spark

• The docker image as we understand it (unit of work received from some private registry and immutable) is not compatible since it should contain Apache Spark installed. ("... The Docker image used must have an appropriate version of Spark already part of the image ...." source)

Docker swarm

• moby/moby#23880 They are not supporting job execution yet.

Custom docker execution service

• Full control over every aspect.
• Full job controller should be implemented.
• Precedent by AdRoll using Luigi (kind of Makefile with Python syntax by Spotify) and a homemade job queue for running and monitoring docker instances. source

Reasons to stay with Kubernetes

• Its being used already for the execution of pipelines and ETL workflows composed by dockerized steps here.