Can prefect run each subflow in different machine? about prefect HOT 2 CLOSED

HI @jsshizhan, thanks for writing in! Yes, you can definitely run subflows on different machines from their initiating flows. It may help to break down a few Prefect concepts first:

• A flow represents the code you want to run (regardless of where you run it)
• A work pool (a pool of Prefect workers) represents where you want a flow to run (regardless of which flows they are)
• A deployment combines a flow that you want to run with the work pool that you want to run it on, along with default parameters, schedules, and other configuration that "instantiates" your flow
• The Prefect SDK provides a run_deployment function that creates a flow run from a deployment, thus creating a subflow that would be running on whatever infrastructure your work pool is on. You can optionally wait for the results of that flow, or just move on trusting that it will run concurrently

Putting that all together, I think what you'd want is to:

• create a work pool for your Kubernetes cluster
• create a work pool for the other group of machines you want to run on
• run least one worker on each work pool; if you're using plain VM instances or baremetal hosts, use the default ProcessWorker or DockerWorker (see the available worker types)
• create Prefect deployments for the flows you want to run
• from within flow A, call run_deployment to invoke flow B on the other infrastructure

I hope this helps, and please follow up with us in our Slack community or Discourse with any more questions you have.

from prefect.

@chrisguidry how to get return value from flow B

from prefect.