Comments (4)
jlewi
commented on July 19, 2024
Our test cluster has 7 nodes (I think autoscaling is on)
Our boot disks are 100 Gb so our test cluster should be using ~700Gb
Our NFS VM uses 1Tb.
The tf-operator E2E spins up GKE clusters which requires additional disks which is where we get the error listed in the previous comment.
from testing.
jlewi
commented on July 19, 2024
I appear to have leaked some PDs from my minikube testing.
gcloud --project=kubeflow-ci compute disks list
NAME ZONE SIZE_GB TYPE STATUS
gke-kubeflow-testing-default-pool-06b8d016-7wfc us-east1-d 100 pd-standard READY
gke-kubeflow-testing-default-pool-06b8d016-cbxh us-east1-d 100 pd-standard READY
gke-kubeflow-testing-default-pool-06b8d016-kp51 us-east1-d 100 pd-standard READY
gke-kubeflow-testing-k80-pool-a651fc91-bl6t us-east1-d 100 pd-standard READY
gke-kubeflow-testing-k80-pool-a651fc91-fpfc us-east1-d 100 pd-standard READY
gke-kubeflow-testing-k80-pool-a651fc91-tdb7 us-east1-d 100 pd-standard READY
gke-kubeflow-testing-k80-pool-a651fc91-vw09 us-east1-d 100 pd-standard READY
jlewi-kubeflow-kubeflow-presubmit-test-473-290a us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-105bb3c-746-8bec us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-335637f-743-6773 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-3baad6d-771-08c7 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-7116e87-747-5534 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-743b807-769-eea9 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-86d4720-765-c9d4 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-a57032f-745-7525 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-c2349d2-748-64ce us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-cc44b5b-744-4b7e us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-e08d4d9-775-b74d us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-gke-473-f79a24f-742-f950 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-105bb3c-746-7c44 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-335637f-743-0cea us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-3baad6d-771-872f us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-7116e87-747-1119 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-743b807-769-d1b7 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-86d4720-765-e094 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-a57032f-745-db0c us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-c2349d2-748-dda6 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-cc44b5b-744-192c us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-e08d4d9-775-c853 us-east1-d 100 pd-standard READY
kubeflow-presubmit-kubeflow-e2e-minikube-473-f79a24f-742-39e6 us-east1-d 100 pd-standard READY
kubeflow-test-nfs-vm us-east1-d 10 pd-ssd READY
kubeflow-test-nfs-vm-data us-east1-d 1000 pd-standard READY
from testing.
jlewi
commented on July 19, 2024
My guess is this was a result from not configuring auto-delete on the PDs. I'm manually deleting them now.
from testing.
jlewi
commented on July 19, 2024
Fixed.
from testing.
Related Issues (20)
- Alternative solution to removal of test on optional-test-infra HOT 31
- Deprecate ECR repo provided by optional-test-infra HOT 17
- Image Scanning for CVs HOT 8
- Image Scanning HOT 2
- IAM as Code HOT 5
- [GCP] Migrate machine type to e2 family to save costs HOT 8
- [AWS] Configure dependabot for new-built image PR
- [AWS] Optional-Test-Infra Migration HOT 2
- Go license tools no longer returning licenses for k8s libraries like apimachinery, controller-runtime, etc HOT 4
- The Optional-test infra should run presubmit jobs for kubeflow/kubeflow
- [AWS] Infrastructure as Code HOT 3
- Improve unit tests for kubeflow/testing repo codebase
- Let optional test infra manage kubeflow/testing presubmit/postsubmit HOT 6
- Migrate to CDK-deployed AWS Resources HOT 2
- rebuild test-worker image HOT 4
- Postsubmit link formatting error
- tekton cluster has been deleted in AWS Optional Test Infrastructure? HOT 3
- eksctl latest release will break cluster setup HOT 3
- Optional Test Infra Deprecation Notice HOT 11
- Support AWS EKS cluster version 1.22 in CI HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from testing.