Comments (7)
it works ,thx
from yatai.
Thanks for your report! Can you show me what the helm release values are for your yatai-image-builder?
helm -n ${namespace} get values yatai-image-builder
from yatai.
helm -n hk-ai-test get values yatai-image-builder
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/houfei/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/houfei/.kube/config
USER-SUPPLIED VALUES:
aws:
accessKeyID: ""
secretAccessKeyExistingSecretKey: ""
secretAccessKeyExistingSecretName: ""
dockerRegistry:
bentoRepositoryName: yatai-bentos
inClusterServer: docker-registry.hk-ai-test.svc.cluster.local:5000
password: ""
secure: false
server: 127.0.0.1:5000
username: ""
yatai:
endpoint: http://${yatai-host}
yataiSystem:
namespace: hk-ai-test
helm -n hk-ai-test get values yatai-deployment
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/houfei/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/houfei/.kube/config
USER-SUPPLIED VALUES:
bentoDeploymentNamespaces:
- hk-ai-test
enableRestrictedSecurityContext: true
layers:
network:
automaticDomainSuffixGeneration: false
domainSuffix: ""
ingressClass: nginx-aws-private
yatai:
endpoint: http://${yatai-host}
yataiServiceAccount:
namespace: hk-ai-test
yataiSystem:
namespace: hk-ai-test
from yatai.
Can you show me the actual env in yatai-image-buider and yatai-deployment?
kubectl -n ${yatai-image-builder-namespace} get secret yatai-image-builder-env -o jsonpath='{.data.YATAI_SYSTEM_NAMESPACE}' | base64 -d
kubectl -n ${yatai-deployment-namespace} get secret yatai-deployment-env -o jsonpath='{.data.YATAI_SYSTEM_NAMESPACE}' | base64 -d
If the env is hk-ai-test
, you should restart yatai-image-builder and yatai-deployment:
kubectl -n ${yatai-image-builder-namespace} rollout restart deploy/yatai-image-builder
kubectl -n ${yatai-deployment-namespace} rollout restart deploy/yatai-deployment
from yatai.
kubectl -n hk-ai-test get secret yatai-image-builder-env -o jsonpath='{.data.YATAI_SYSTEM_NAMESPACE}' | base64 -d
hk-ai-test
kubectl -n hk-ai-test get secret yatai-deployment-env -o jsonpath='{.data.YATAI_SYSTEM_NAMESPACE}' | base64 -d
hk-ai-test
kubectl -n hk-ai-test rollout restart deploy/yatai-image-builder
deployment.apps/yatai-image-builder restarted
kubectl -n hk-ai-test rollout restart deploy/yatai-deployment
deployment.apps/yatai-deployment restarted
yatai-image-builder error log
Version: 1.1.3
GitCommit: ea82e2b
BuildDate: 2023-01-17T10:02:40Z
1.6777302369828238e+09 INFO setup starting manager
1.6777302369829607e+09 INFO getting yatai client {"func": "doRegisterYataiComponent"}
1.6777302369836986e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
1.677730236983708e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0302 04:10:36.983933 1 leaderelection.go:248] attempting to acquire leader lease hk-ai-test/58b75536.yatai.ai...
time="2023-03-02T04:10:37Z" level=error msg="POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}"
1.6777302370496786e+09 ERROR registerYataiComponent {"func": "registerYataiComponent", "error": "register yatai component: DoJsonRequest Error: [POST]http://${yatai-host}/api/v1/clusters/bento/yatai_components: POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}", "errorVerbose": "POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:193\ngithub.com/bentoml/yatai-image-builder/yatai-client.(*YataiClient).RegisterYataiComponent\n\t/workspace/yatai-client/client.go:152\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).doRegisterYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1861\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).registerYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1882\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nDoJsonRequest Error: [POST]http://${yatai-host}/api/v1/clusters/bento/yatai_components\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do.func1\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:128\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:194\ngithub.com/bentoml/yatai-image-builder/yatai-client.(*YataiClient).RegisterYataiComponent\n\t/workspace/yatai-client/client.go:152\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).doRegisterYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1861\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).registerYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1882\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nregister yatai component\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).doRegisterYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1876\ngithub.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).registerYataiComponent\n\t/workspace/controllers/resources/bentorequest_controller.go:1882\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
github.com/bentoml/yatai-image-builder/controllers/resources.(*BentoRequestReconciler).registerYataiComponent
/workspace/controllers/resources/bentorequest_controller.go:1884
1.6777302616634731e+09 DEBUG events yatai-image-builder-59f757d555-cjltk_b486d0ca-cfdf-4a48-9114-1c588498a195 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"hk-ai-test","name":"58b75536.yatai.ai","uid":"b203a26b-240d-4bfa-a0da-114e7ecee122","apiVersion":"coordination.k8s.io/v1","resourceVersion":"585834952"}, "reason": "LeaderElection"}
I0302 04:11:01.664956 1 leaderelection.go:258] successfully acquired lease hk-ai-test/58b75536.yatai.ai
yatai-deployment error log
Version: 1.1.8
GitCommit: 0bd8ecf
BuildDate: 2023-02-14T08:16:37Z
1.6777302431243255e+09 INFO controller-runtime.builder skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called {"GVK": "serving.yatai.ai/v1alpha2, Kind=BentoDeployment"}
1.6777302431243482e+09 INFO controller-runtime.builder skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called {"GVK": "serving.yatai.ai/v1alpha2, Kind=BentoDeployment"}
1.6777302431244867e+09 INFO start cleaning up abandoned runner services {"func": "doCleanUpAbandonedRunnerServices"}
1.6777302431245127e+09 INFO getting yatai client {"func": "doRegisterYataiComponent"}
1.6777302431249635e+09 INFO controller-runtime.webhook Registering webhook {"path": "/convert"}
1.6777302431250212e+09 INFO controller-runtime.builder Conversion webhook enabled {"GVK": "serving.yatai.ai/v1alpha2, Kind=BentoDeployment"}
1.6777302431250324e+09 INFO setup starting manager
1.6777302431252067e+09 INFO controller-runtime.webhook.webhooks Starting webhook server
1.6777302431252685e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
1.6777302431253283e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0302 04:10:43.125364 1 leaderelection.go:248] attempting to acquire leader lease hk-ai-test/b292d523.yatai.ai...
1.6777302431254594e+09 INFO controller-runtime.certwatcher Updated current TLS certificate
1.677730243125541e+09 INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443}
1.6777302431255946e+09 INFO controller-runtime.certwatcher Starting certificate watcher
time="2023-03-02T04:10:43Z" level=error msg="POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}"
1.677730243200576e+09 ERROR registerYataiComponent {"func": "registerYataiComponent", "error": "DoJsonRequest Error: [POST]http://${yatai-host}/api/v1/clusters/bento/yatai_components: POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}", "errorVerbose": "POST http://${yatai-host}/api/v1/clusters/bento/yatai_components status=403, {\"message\":\"get api token: failed to get yatai config in cluster bento in organization default: secret yatai-common-env not found in namespace yatai-system: secrets \\\"yatai-common-env\\\" not found\"}\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:193\ngithub.com/bentoml/yatai-deployment/yatai-client.(*YataiClient).RegisterYataiComponent\n\t/workspace/yatai-client/client.go:145\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).doRegisterYataiComponent\n\t/workspace/controllers/bentodeployment_controller.go:2979\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).registerYataiComponent\n\t/workspace/controllers/bentodeployment_controller.go:2999\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nDoJsonRequest Error: [POST]http://${yatai-host}/api/v1/clusters/bento/yatai_components\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do.func1\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:128\ngithub.com/bentoml/yatai-common/reqcli.(*JsonRequestBuilder).Do\n\t/go/pkg/mod/github.com/bentoml/[email protected]/reqcli/http.go:194\ngithub.com/bentoml/yatai-deployment/yatai-client.(*YataiClient).RegisterYataiComponent\n\t/workspace/yatai-client/client.go:145\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).doRegisterYataiComponent\n\t/workspace/controllers/bentodeployment_controller.go:2979\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).registerYataiComponent\n\t/workspace/controllers/bentodeployment_controller.go:2999\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
github.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).registerYataiComponent
/workspace/controllers/bentodeployment_controller.go:3001
from yatai.
@janljd Thank you very much for reporting the bug. I have fixed it and released a new version. You need to update yatai to version 1.1.7
.
from yatai.
The latest version of yatai is 1.1.13, and the same error has occurred again.
from yatai.
Related Issues (20)
- Error: [yatai] `login` failed: request failed with status code 403: {"message":"get api token: record not found"} HOT 8
- feat: Yatai to flatten YAML configuration into JQ format
- builder OOM Killed HOT 7
- Not able to build the Bentoml from the .mar file HOT 1
- Failed on deploying Yatai to EKS HOT 5
- Run Triton Server in local docker registry
- How to update docker registry secret? HOT 2
- Error: Cannot connect to AWS ECR repository
- does yatai support deploy bento to other cluster? HOT 3
- docker crash on readyz call
- fix trans in BentoCloud HOT 1
- Error: kaniko(bentoImageBuildEngine) apt-get install error HOT 1
- Error: get Bento CR bentoes.resources.yatai.ai not found HOT 1
- Will yatai login ldap be supported? HOT 1
- Deploy a "custom Bentoml image" using "yatai-deployment".
- Support labels from values.yaml
- Error: YAML parse error on yatai/templates/deployment.yaml HOT 4
- Installation of yatai-deployment failed to [ERROR registerYataiComponent...] HOT 5
- Provide a way to disable home page cards
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yatai.