gsemet / kube-airflow Goto Github PK

If you pick a value other than empty string, the ENV vars in the containers will be prepended with this value (eg if you set prefix to "af_" then RABBIT_HOST becomed AF_RABBIT_HOST). This breaks nearly all Pods.

When this chart is mature enough, let's contribute to incubator section of kubernetes/charts

It is currently configured to:
path: "{{ .Values.airflow.url_prefix }}/health"

Looking at the logs of the web container, I see this:
10.244.3.1 - - [13/Dec/2017:15:20:36 +0000] "GET /airflow/health HTTP/1.1" 404 3703 "-" "Go-http-client/1.1"

Kubernetes then kills the Pod, and the web Pod eventually goes into a CrashLoopBackOff.

When I have changed the value to: path: "/health" things work as intended.

Note that is a vanilla deployment of this repo, i.e. I did not add or modify any variables.

It had previously been discussed on slack that we could use the the git-sync sidecar for updating dags from github. The environment variables have already been added. We could add the git-sync sidecar to the scheduler, worker and web deployments. If this is done, do we make it an option in the values.yaml that has to be enabled?

I am trying to understand what happens inside this script but I don't see it in the repository

https://github.com/Stibbons/kube-airflow/blob/229a05e9b15a538e995172ded511bbbbed346c2e/airflow/templates/deployments-worker.yaml#L33

We should consider using a requirements.yaml for RabbitMQ and Postgres and stop managing these charts ourselves. This also allows for these dependencies to be optional. In some cases, some people may choose to run a Postgres outside their k8s cluster (such as RDS).

I installed using the default values, but enabled persistence.

A PV was dynamically created by the PVC and correctly mounted in the relevant Pods. So far so good.

However, the default location where the PV is mounted in the pods is /dags.
The value of dags_folder in airflow.cfg is pointing to /usr/local/airflow/dags.

Hence, more work is needed to use persistent volumes to store DAG files.

This is a placeholder for adding tests. This is probably a requirement for getting this into kubernetes/charts

https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md

kubectl get events:

kubectl events says: spec.containers{airflow-flower} Warning Unhealthy kubelet, k8s-development-41339263-2 Liveness probe failed: HTTP probe failed with statuscode: 404

The probe is defined as path: “{{ .Values.ingress.flower.path }}/” and the value in values.yaml is /flower

So the probe checks localhost:5555/flower, which returns 404. Changing the value in the values.yaml to empty string (“”) fixes the probe.

I'm new in kubernetes. I deploy 4 workers and somehow they always restarted within < 10 minutes. When i check with kubectl get pods, the restart counter is not increased, and i can't access task's log file because the log files in that workers disappeared. Is that expected?

*** Log file isn't local.
*** Fetching here: http://worker-3.workers.default.svc.cluster.local:8793/log/dagsname/task1/2018-02-19T00:00:00/1.log
*** Failed to fetch log file from worker. 404 Client Error: NOT FOUND for http://worker-3.workers.default.svc.cluster.local:8793/log/dagsname/task1/2018-02-19T00:00:00/1.log

With a access policy of ReadWriteOnce, which is the only access policy supported by aws ebs volumes. This will mean that each pvc will need to be given a unique name, like "fullname"-scheduler, etc.

An alternative proposal would be to using a pvc that supports ReadWriteMany. Users on AWS could possible use the elastic file system available from the external storage project https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs

If this were the case, only one deployment would need to run the optional git-sync which may be desirable.

Personally, I think the second approach is more desireable

Hey @stibbons I've been tinkering around with your repo and one thing I noticed is that somehow a worker will fail at all task while another one will succeed. As I'm tuning Airflow, I'm curious to see what would be the tradeoff between more worker statefulsets vs increasing the amount of dag_concurrency within the Airflow configuration?