[FEA] Add cluster setup scripts for different communication protocols/setups about multi-gpu-tools HOT 3 CLOSED

Sounds like a good idea. Would these be 3 separate versions of run-dask-process.sh? If so, how would the project-specific code (ie. non-shared) specify which to use, since (at the moment) scripts call run-cluster-dask-jobs.sh and do not directly call run-dask-process.sh? The parameter idea you mentioned might be better in that case.

from multi-gpu-tools.

Yeah, thinking about this a bit more, the projects would call into run-cluster-dask-jobs.sh and then based on an additional parameter it would branch out into calling the specific one of 3 files.
I agree that it would be cleaner to parametrize this into the single run-dask-process.sh file which decides which function to use when building the appropriate scheduler and worker arguments, based on the argument provided.

More concretely it would be something like this:

run-dask-process.sh

function build_tcp_args()
...

function build_ucx_nvlink_args()
...

and so on, where the script could be called as bash run-dask-process.sh tcp scheduler workers either from other projects, or via the run-cluster-dask-jobs.sh file.

@rlratzel If this approach makes sense, happy to take this on and work on the updates

from multi-gpu-tools.

That sounds good to me. I think that will make running on different cluster configurations much easier. Just let me know when your PR is ready for review and I'll gladly take a look. Thanks @ayushdg

from multi-gpu-tools.