Comments (5)
Thanks for the report! Searching around for this problem, there doesn't seem to be a definitive answer. In the kubernetes code base it's mostly set to minutes. Can you try something like kubeflowInformerFactory := informers.NewSharedInformerFactory(kubeflowClient, 10*time.Minute)
to see if that solves your problem?
from mpi-operator.
By the way I'm really interested in where you are running these thousands of mpi jobs and for what purpose. :) Can I quote you guys as a user/customer of mpi-operator?
from mpi-operator.
I have thousands of mpijob,most of them are successed or failed. If the resync period is short, mpi-operator cost many time to sync the finished mpijob which have nothing change.
I support our group's deep learning platform which has many many jobs.
Yes, you can quote me as a user/customer of mpi-operator
from mpi-operator.
Right, setting the resync period to 0 means to never resync, that should solve your problem.
from mpi-operator.
@answer3y do you mind telling me your group/company's name? If you don't feel comfortable sharing it on github, you can email me directly (rong dot ou at gmail dot com), or hit me up on the kubeflow slack. Thanks!
from mpi-operator.
Related Issues (20)
- pod priority was assigned to 0 though the priorityclassname of the podgroup had been assigned HOT 2
- Port conficts will occur when multiple pods dispatched to the same node under hostnetwork. HOT 3
- MPI Job fails on EKS with > 2 instances of 128 core per instance HOT 2
- Update stale examples HOT 2
- MPI-Operator run example failed HOT 8
- Cant get mpijob status when pod template is invalid HOT 9
- which is the latest mpi job definition between mpi-operator and training operator HOT 4
- the object has been modified; please apply your changes to the latest version and try again HOT 8
- When when WaitForWorkersReady is enabled in MPI operator, MPI operator and gang scheduler are in a deadlock HOT 4
- Work with DeepSpeed for large scale training HOT 28
- Replace the plain pod workers with Indexed Job HOT 4
- "make generate" command run failed
- The operator still creates the launcher when launcherCreationPolicy is "WaitForWorkersReady" and suspend is "true"
- Running in a subset of namespaces HOT 8
- Wrong host info in discover_hosts.sh
- NCCL tests example HOT 1
- how could mpijob of mpi operator worker get the hostname of launcher HOT 2
- Connection reset HOT 4
- "cleanPodPolicy: All" does not clean up launcher pod HOT 1
- ttlSecondsAfterFinished for MPIJob, not only launcher HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mpi-operator.