Slow to start a MPIJob about mpi-operator HOT 5 CLOSED

Thanks for the report! Searching around for this problem, there doesn't seem to be a definitive answer. In the kubernetes code base it's mostly set to minutes. Can you try something like kubeflowInformerFactory := informers.NewSharedInformerFactory(kubeflowClient, 10*time.Minute) to see if that solves your problem?

from mpi-operator.

By the way I'm really interested in where you are running these thousands of mpi jobs and for what purpose. :) Can I quote you guys as a user/customer of mpi-operator?

from mpi-operator.

@rongou

I have thousands of mpijob，most of them are successed or failed. If the resync period is short, mpi-operator cost many time to sync the finished mpijob which have nothing change.

I support our group's deep learning platform which has many many jobs.

Yes, you can quote me as a user/customer of mpi-operator

from mpi-operator.

Right, setting the resync period to 0 means to never resync, that should solve your problem.

from mpi-operator.

@answer3y do you mind telling me your group/company's name? If you don't feel comfortable sharing it on github, you can email me directly (rong dot ou at gmail dot com), or hit me up on the kubeflow slack. Thanks!

from mpi-operator.