kubeflow / testing Goto Github PK
View Code? Open in Web Editor NEWTest infrastructure and tooling for Kubeflow.
License: Apache License 2.0
Test infrastructure and tooling for Kubeflow.
License: Apache License 2.0
Right now our instructions require that every repository build a Docker image and create a custom run.sh script.
Maintaining a separate container per repository is more work than necessary. The sole purpose of this is to specify the command line arguments to run_e2e_workflow.py that control the workflow to run.
We can get rid of that by adopting a convention that the arguments will be defined in the root of the repository e.g. /prow_config.yaml
Then we can have a single container and entrypoint used by our prow jobs that loads the workflows and submits them.
In kubeflow/training-operator#401 we started using yapf to enforce a consistent Python style.
We should a presubmit test so that we can ensure all python code is properly formatted.
/cc @jimexist
The ultimate goal would be to have Argo able to run multiple testing workflows on different platforms.
A first step would be a setup that can be used to run the e2e tests locally on an OpenShift cluster.
We should setup appropriate access to the test infrastructure so we can share it with appropriate members of the community.
The current thinking is to
To start I think we might want to groups
test-admin
test-viewer
We want to move our test infrastructure into kubeflow.org and share it with other members of the community willing to take on responsibility for maintaining it.
Here's what we need
Our dev environment depends on ISTIO for monitoring.
We should document the ISTIO setup in
https://github.com/kubeflow/testing/blob/master/dev-kubeflow-org/README.md
We should also update the shell scripts necessary.
/assign @lluunn
Example PR: kubeflow/kubeflow#524
Error we observed was
+ /usr/local/bin/checkout.sh /src
+ SRC_DIR=/src
+ mkdir -p /src/kubeflow
+ git clone https://github.com/kubeflow/kubeflow.git /src/kubeflow/kubeflow
Cloning into '/src/kubeflow/kubeflow'...
+ echo Job Name = kubeflow-presubmit
+ REPO_DIR=/src/kubeflow/kubeflow
+ cd /src/kubeflow/kubeflow
Job Name = kubeflow-presubmit
+ '[' '!' -z 524 ']'
+ git fetch origin pull/524/head:pr
From https://github.com/kubeflow/kubeflow
* [new ref] refs/pull/524/head -> pr
+ '[' '!' -z 7d9c2633d7ddf2a1957e685f9e7f287340be0e34 ']'
+ git checkout 7d9c2633d7ddf2a1957e685f9e7f287340be0e34
fatal: reference is not a tree: 7d9c2633d7ddf2a1957e685f9e7f287340be0e34
We should delete the GKE cluster in mlkube-testing after migrating to kubeflow-ci
Blocked by kubeflow/examples#63
Our tests create a bunch of resources
Most of these resources (exception is Argo) should be GC'd by the teardown steps in our tests. But some failures/bugs prevent resources from being GC'd.
It would be good to have a cron job to periodically garbage collect old resources like namespaces.
We should see if we can add a link to the Argo UI for the job so that its readily accessible in gubernator.
I wonder if we can just specify it as metadata?
In our E2E jsonnet workflows we pass prow environment variables as a string of comma separated key value pairs
e.g
ks param set workflows prow_env "JOB_NAME=${JOB_NAME},JOB_TYPE=presubmit,PULL_NUMBER=${PULL_NUMBER},REPO_NAME=${REPO_NAME},REPO_OWNER=${REPO_OWNER},BUILD_NUMBER=${BUILD_NUMBER}"
It would be nice to parse this into a dictionary of key value pairs so that we could easily reference these parameters in jsonnet by name. Here is a snippet to do that. It would be nice to define this as a utility function in a common location.
listToDict:: function(v)
{
// We enclose v[0] in brackets because we want to treat it as a variable and use its value
// as the field and not treat it as a string literal corresponding to the field name.
[ v[0] ]: v[1],
},
// parseEnvToMap takes a string "key1=value1,key2=value2,..."
// and turns it into an object with keys key1, key2, ... and values value1,...,value2
parseEnvToMap::function(v)
local pieces = std.split(v, ",");
if v != "" && std.length(pieces) > 0 then
std.foldl(
function(a, b) a + $.listToDict(std.split(b, "=")),
pieces, {}
)
else {},
We need a bot Github account with (minimum necessary) access to repos in kubeflow org in order to automate repo management.
This bot will be responsible for tasks like label sync (kubeflow/community#24), and in future PR scan, issue update.
We'll have a kubeflow.org group as owner of bot user.
On this PR
#126
travis ran and failed. Did someone enable travis? To my knowledge we don't use Travis in this repo.
We need to figure out how to test our UIs like the TFJobs UI.
We should create a standard workflow to run basic tests that can be shared by all our repositories; eg
I think its an open question whether we would run this as a separate prow job or have a single prow job for each workflow and run this as a sub workflow.
It would be nice if we could auto-push dev.kubeflow.org or perhaps create another environment nightly-dev.kubeflow.org.
There are certain changes like UI that are difficult to test manually. It would be useful if we had an up todate environment.
I think we could easily adapt our E2E tests to do this and then just run it as a cron job.
This'll cause a momentary blip.
See for example
kubeflow/kubeflow#787
Relevant bit:
INFO|2018-05-10T18:32:32|/src/kubeflow/testing/py/kubeflow/testing/util.py|58| Updating workflows kubeflow-test-infra.jlewi-kubeflow-gke-deploy-test-4-3a8b
INFO|2018-05-10T18:32:33|/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py|161| URL for workflow: http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-kubeflow-gke-deploy-787-f893ce6-1462-040e?tab=workflow
INFO|2018-05-10T18:32:33|/src/kubeflow/testing/py/kubeflow/testing/argo_client.py|21| Workflow kubeflow-presubmit-kubeflow-e2e-gke-787-f893ce6-1462-b429 in namespace kubeflow-test-infra; phase=Running
INFO|2018-05-10T18:32:33|/src/kubeflow/testing/py/kubeflow/testing/argo_client.py|21| Workflow kubeflow-presubmit-kubeflow-e2e-minikube-787-f893ce6-1462-b59b in namespace kubeflow-test-infra; phase=Running
INFO|2018-05-10T18:32:33|/src/kubeflow/testing/py/kubeflow/testing/argo_client.py|21| Workflow kubeflow-presubmit-tf-serving-image-787-f893ce6-1462-0913 in namespace kubeflow-test-infra; phase=Running
INFO|2018-05-10T18:32:34|/src/kubeflow/testing/py/kubeflow/testing/util.py|512| Writing gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/787/kubeflow-presubmit/1462/finished.json
INFO|2018-05-10T18:32:34|/src/kubeflow/testing/py/kubeflow/testing/util.py|522| Uploading file /tmp/tmpRunE2eWorkflowbjXGWwlog to gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/787/kubeflow-presubmit/1462/build-log.txt.
So looks like we ended while workflows were still running.
In Gubernator, we display metadata that has links to the Argo workflows.
This metadata is stored in finished.json
https://github.com/kubernetes/test-infra/tree/master/gubernator
We currently write finsihed.json after the test finishes. But this means the handy links to the Argo workflows aren't available until the job finishes.
I wonder if we could wrote finished.json at the start of the test but with just the metadata field (no timestamp or result) and then just overwrite it at the end. The purpose being to display the Argo links even before the job finishes.
@BenTheElder @krzyzacy Do you know if this will work?
We need to look at the junit XML files to decide whether to report the job as failed or success
We need to do something like kubeflow/training-operator#319
If we want to be strict about which xml files are expected we could specify the xml files expected in the prow_config.yaml file.
We should setup some prow jobs for this repo to run lint checks and python unittests
Found that if the workflow name is too long in prow_config.yaml I get this error:
Pod "kubeflow-pytorch-operator-presubmit-pytorchjob-e2e-13-1a13f4d-3-17ac-2950311768" is invalid: metadata.labels: Invalid value: "kubeflow-pytorch-operator-presubmit-pytorchjob-e2e-13-1a13f4d-3-17ac": must be no more than 63 characters
Should probably truncate or provide a warning to to the user.
We should add instructions to our test-infra setup to add GPUs.
Here are the commands I used
gcloud beta --project=kubeflow-ci container node-pools create k80 --accelerator type=nvidia-tesla-k80,count=1 --zone us-east1-d --cluster kubeflow-testing --machine-type=n1-standard-8 --num-nodes 1 --min-nodes 0 --max-nodes 5 --enable-autoscaling
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/cos/daemonset-preloaded.yaml
This is straight from the docs
https://cloud.google.com/kubernetes-engine/docs/concepts/gpus
Need to setup prow for kubeflow/testing
I tried rebuilding the Docker image used by our test workers and I got the following error.
In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
enum34>=1.1.3; python_version < "3.4" from https://pypi.python.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl#md5=68f6982cc07dde78f4b500db829860bd (from astroid==1.6.1->-r /tmp/requirements.txt (line 1))
singledispatch; python_version < "3.4" from https://pypi.python.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl#md5=d633bac187d681455ab065c645be845d (from astroid==1.6.1->-r /tmp/requirements.txt (line 1))
backports.functools-lru-cache; python_version < "3.4" from https://pypi.python.org/packages/03/8e/2424c0e65c4a066e28f539364deee49b6451f8fcd4f718fefa50cc3dcf48/backports.functools_lru_cache-1.5-py2.py3-none-any.whl#md5=5a93b4dac0cc414ceff28ff678713e31 (from astroid==1.6.1->-r /tmp/requirements.txt (line 1))
setuptools>=34.0.0 from https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5 (from google-api-core==0.1.4->-r /tmp/requirements.txt (line 6))
futures>=3.2.0; python_version < "3.2" from https://pypi.python.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl#md5=cfd62ab6c9852b04bb6048480fefaa75 (from google-api-core==0.1.4->-r /tmp/requirements.txt (line 6))
funcsigs>=1; python_version < "3.3" from https://pypi.python.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl#md5=701d58358171f34b6d1197de2923a35a (from mock==2.0.0->-r /tmp/requirements.txt (line 23))
configparser; python_version == "2.7" from https://pypi.python.org/packages/7c/69/c2ce7e91c89dc073eb1aa74c0621c3eefbffe8216b3f9af9d3885265c01c/configparser-3.5.0.tar.gz#md5=cfdd915a5b7a6c09917a64a573140538 (from pylint==1.8.2->-r /tmp/requirements.txt (line 30))
backports.ssl_match_hostname from https://pypi.python.org/packages/76/21/2dc61178a2038a5cb35d14b61467c6ac632791ed05131dda72c20e7b9e23/backports.ssl_match_hostname-3.5.0.1.tar.gz#md5=c03fc5e2c7b3da46b81acf5cbacfe1e6 (from websocket-client==0.40.0->-r /tmp/requirements.txt (line 41))
The command '/bin/sh -c pip install -U pip wheel && pip install -r /tmp/requirements.txt && pip3 install -U pip wheel && pip3 install -r /tmp/requirements.txt'
This might be fixed by #36 by switching to pipenv.
In gubernator we report the url for the worfklow UI as
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-tf-serving-image-387-f608b1a-532-1c41;tab=workflow
This doesn't appear to be valid anymore with the new version of the Argo UI.
I think we just want to use
http://testing-argo.kubeflow.org/workflows/kubeflow-test-infra/kubeflow-presubmit-tf-serving-image-387-f608b1a-532-1c41
@jimexist would you mind sending another PR to fix this?
See kubeflow/kubeflow#558
When adding a new E2E test we might want to turn it on but not mark failures as blocking/failures because we want to bake the test before making it blocking.
We could do this as follows
The JUnit XML files could still indicate whether the test passed or not so that in test grid we could see whether the test was reliable or not.
The old hostname testing-argo.kubeflow.io is no longer valid.
testing-argo.kubeflow.org should work.
We need to update the test scripts to print out the correctly URLs.
As a temporary work around people can manually edit the URLs.
During label auto-sync, we had a label rename: "do-not-merge/work-in-progress" -> "status/in progress", which might cause prow merge uncompleted PR.
One false positive merge here: kubeflow/community#41 (comment)
To fix it:
Affected repos (might merge PR incomplete):
reporting
examples
community
pytorch-operator
caffe2-operator
experimental-beagle
hp-tuning
Those repos are not affected due to bot user's lack of permission (include our most active repos):
kubeflow
tf-operator
testing
example-seldon
experimental-kvc
See #47
Background:
kubeflow/community#20
tide seems to be working for kubeflow/kubeflow.
We'd like to setup our infrastructure so that we can build Docker images on pre/postsubmit so that we always have up to date images.
Our current thinking is that we will use Argo to create pipelines that do the builds and run any tests. We can use Prow to trigger this in response to pre/postsubmit
Here's some detail how this could work
Related issues:
Error
"ERROR:root:Exception occured creating cluster: <HttpError 403 when requesting https://container.googleapis.com/v1/projects/kubeflow-ci/zones/us-east1-d/clusters?alt=json returned "Insufficient regional quota to satisfy request for resource: "DISKS_TOTAL_GB". The request requires '100.0' and is short '4.0'. The regional quota is '4096.0' with '96.0' available.">, status: 403
"
This happened on kubeflow/training-operator#485
The kubeflow-testing cluster was recreated yesterday.
/assign jlewi
PTAL
#36 took out the ssh
install from #76 in the test-worker image. But a bunch of commits since then too. ๐ Somehow the updated image just got pushed to kubeflow-ci in the past 3 days.
pmackinn@kubeflow-ci:~$ gcloud docker -- run -it --rm --entrypoint=/bin/sh gcr.io/kubeflow-ci/test-worker:latest
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. Please use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR regi
stries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning, e.g. `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/suppo
rt/deprecation-notices#gcloud-docker
# dpkg -l | grep ssh
# exit
pmackinn@kubeflow-ci:~$ gcloud docker -- run -it --rm --entrypoint=/bin/sh gcr.io/kubeflow-ci/test-worker:v20180320-8c28838-e3b0c4
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03. Please use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a credential helper, then use `docker` as you would for non-GCR regi
stries, e.g. `docker pull gcr.io/project-id/my-image`. Add `--verbosity=error` to silence this warning, e.g. `gcloud docker --verbosity=error -- pull gcr.io/project-id/my-image`. See: https://cloud.google.com/container-registry/docs/suppo
rt/deprecation-notices#gcloud-docker
# dpkg -l | grep ssh
ii openssh-client 1:7.2p2-4ubuntu2.4 amd64 secure shell (SSH) client, for secure access to remote machines
ii openssh-server 1:7.2p2-4ubuntu2.4 amd64 secure shell (SSH) server, for secure access from remote machines
ii openssh-sftp-server 1:7.2p2-4ubuntu2.4 amd64 secure shell (SSH) sftp server module, for SFTP access from remote machines
ii ssh 1:7.2p2-4ubuntu2.4 all secure shell client and server (metapackage)
# exit
It would be good to have a presubmit check that will automatically fail if a string like "DO NOT SUBMIT" appears anywhere in the patch.
I'm not sure if this is something prow already supports.
We should run E2E tests on minikube.
kube_test is supposed to help with this.
checkout.sh supports the PROW environment variables.
But it also supports the more convenient syntax we introduced to specify for extra repos.
For workflows triggered outside of prow (e.g. manually) its more convenient just to specify EXTRA_REPOS.
We should update checkout.sh so that it doesn't assume REPO_OWNER and other prow environment variables.
We use gcr.io/kubeflow-ci/test-worker as the worker image in our CI/CD workflows.
We would like to use the same Docker image in our release workflows.
But we'd like to make sure untruted code running in PRs can't modify those images.
So it would be good to move these docker images into a separate registry such that code running in kubeflow-ci can read but not write that registry.
On our test cluster in project mlkube-testing. I'm noting Argo workflows going back as far as 22 days old.
I'm also knowing that the UI and kubectl are sluggish.
I suspect this might be because there are so many workflows.
What is the garbage collection policy for workflows that have completed? Is there any?
For our release cluster we should setup IAP so we can securely connect to the Argo UI.
We will need to wait for all the issues with IAP and Envoy JWT validation to be fixed so we can do this securely.
Job is marked as success and no test failures. There is no error in the log.
The issue is that the pod is exiting with non zero exit code. If we look at the pod logs (via prow jobs UI) we see the following exceptoin
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 294, in <module>
success = main()
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 284, in main
return run(args, file_handler)
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 184, in run
if results["status"]["phase"] != "Succeeded":
TypeError: list indices must be integers, not str
This exception happens after the logs have been uploaded to GCS which is why it doesn't show up in the gubernator logs.
/cc @lluunn
INFO|2018-02-10T02:56:21|/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py|129| URL for workflow: http://testing-argo.kubeflow.io/timeline/kubeflow-test-infra/kubeflow-presubmit-232-a452812-181-cf21;tab=workflow
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 232, in <module>
success = main()
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 222, in main
return run(args, file_handler)
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 130, in run
status = github_status.GithubStatus()
File "/src/kubeflow/testing/py/kubeflow/testing/github_status.py", line 51, in __init__
Github.__init__(self)
File "/src/kubeflow/testing/py/kubeflow/testing/github_status.py", line 24, in __init__
raise Exception('Missing environment variable GIT_TOKEN')
Exception: Missing environment variable GIT_TOKEN
This is caused by #17 . The environment variable GIT_TOKEN isn't defined in the prow job. This code runs in the prow container.
We'll need to rethink how we get the GIT API token; its possible prow attaches a secret already for the API token.
Alternatively it might just be easier to see if we can set it in the gubernator see #10 .
In the meantime we should roll this back since it breaks our tests.
Katacoda just added another Kubeflow scenario (the GH issue summarization example).
It would be great if we could work with Katacoda on figuring out how to test these scenarios against the latest version of Kubeflow.
In Gubernator the Refs links don't work for us. It ends up trying to go to kubernetes/kubernetes
kubernetes/kubernetes@a7511ff
I'm referring to this set of files: https://github.com/kubeflow/tf-operator/tree/master/py
For example there should be a generic version deploy.py
to deploy a cluster without tf specific params such as https://github.com/kubeflow/tf-operator/blob/master/py/deploy.py#L144
Would like a link directly from a PR to the corresponding argo workflow to make debugging easier
This was originally aded in #17 but this broke things #20 and was rolled back.
The environment variable GIT_TOKEN isn't defined in the prow job. This code runs in the prow container.
We'll need to rethink how we get the GIT API token; its possible prow attaches a secret already for the API token.
Alternatively it might just be easier to see if we can set it in gubernator see #10 .
/cc @jose5918
Related to #72 (truncate workflow name)
Right now we encode a lot of information in the workflow name. The pattern is
{ORG}-{REPOSITORY}-{PROW-JOB-TYPE}-{WORKFLOW-NAME}-{PR-NUMBER}-{RANDOMNESS}
It would be good to start using labels to represent this information. This would make it easier to filter and select workflows.
A lot of times our tests will fail because we timeout waiting for a K8s deployment.
An example is when a deployment fails to start because the image couldn't be pulled.
To debug the test failure we'd like to capture the events for pod because this provides helpful debug information.
For example here is the output of kubectl describe for a pod that failed to start because the name of the binary to invoke was wrong.
kubectl -n kubeflow-presubmit-kubeflow-e2e-299-439a983-360-cee1 describe pods tf-job-operator-7899645f6c-cgxd2
Name: tf-job-operator-7899645f6c-cgxd2
Namespace: kubeflow-presubmit-kubeflow-e2e-299-439a983-360-cee1
Node: gke-kubeflow-testing-default-pool-3cd81eea-2dp8/10.142.0.2
Start Time: Mon, 26 Feb 2018 18:02:23 -0800
Labels: name=tf-job-operator
pod-template-hash=3455201927
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kubeflow-presubmit-kubeflow-e2e-299-439a983-360-cee1","name":"tf-job-operator-789...
Status: Running
IP: 10.36.4.231
Created By: ReplicaSet/tf-job-operator-7899645f6c
Controlled By: ReplicaSet/tf-job-operator-7899645f6c
Containers:
tf-job-operator:
Container ID: docker://e1d6dfd6955d2c4c1d89030626e8f143293f14bd55081421ce0d3da26ab64845
Image: gcr.io/kubeflow-images-staging/tf_operator:v20180226-403
Image ID: docker-pullable://gcr.io/kubeflow-images-staging/tf_operator@sha256:6c80d00498103a87d2acbdad34ce56886bacf29fb819e4424fa909dfc4c5b2e6
Port: <none>
Command:
/opt/mlkube/tf_operator
--controller-config-file=/etc/config/controller_config_file.yaml
--alsologtostderr
-v=1
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: oci runtime error: container_linux.go:247: starting container process caused "exec: \"/opt/mlkube/tf_operator\": stat /opt/mlkube/tf_operator: no such file or directory"
Exit Code: 127
Started: Mon, 26 Feb 2018 18:18:16 -0800
Finished: Mon, 26 Feb 2018 18:18:16 -0800
Ready: False
Restart Count: 8
Environment:
MY_POD_NAMESPACE: kubeflow-presubmit-kubeflow-e2e-299-439a983-360-cee1 (v1:metadata.namespace)
MY_POD_NAME: tf-job-operator-7899645f6c-cgxd2 (v1:metadata.name)
Mounts:
/etc/config from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from tf-job-operator-token-kk7pm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: tf-job-operator-config
Optional: false
tf-job-operator-token-kk7pm:
Type: Secret (a volume populated by a Secret)
SecretName: tf-job-operator-token-kk7pm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned tf-job-operator-7899645f6c-cgxd2 to gke-kubeflow-testing-default-pool-3cd81eea-2dp8
Normal SuccessfulMountVolume 19m kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 MountVolume.SetUp succeeded for volume "config-volume"
Normal SuccessfulMountVolume 19m kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 MountVolume.SetUp succeeded for volume "tf-job-operator-token-kk7pm"
Normal Pulled 18m (x4 over 19m) kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 Container image "gcr.io/kubeflow-images-staging/tf_operator:v20180226-403" already present on machine
Normal Created 18m (x4 over 19m) kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 Created container
Warning Failed 18m (x4 over 19m) kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 Error: failed to start container "tf-job-operator": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"/opt/mlkube/tf_operator\": stat /opt/mlkube/tf_operator: no such file or directory"
Warning BackOff 9m (x44 over 19m) kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 Back-off restarting failed container
Warning FailedSync 4m (x74 over 19m) kubelet, gke-kubeflow-testing-default-pool-3cd81eea-2dp8 Error syncing pod
prow supports using your own build cluster.
Since we already use a K8s cluster to run our Argo workflows, we should ask prow to run our prow jobs in that cluster as well this will give us more access to the logs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.