Execute Workflow from a Pub / Sub subscription about workflows-samples HOT 7 CLOSED

Thanks for the reply @grant ,

Apologies for not getting back to you, this one seems to have passed me by.

Appreciate the confirmation, although I think this means Workflow is unlikely to be of much use directly for our desired use case right now.

I didn’t go into detail before, but basically data is written to Pub/Sub and picked up by App Engine for writing to BigQuery.

It felt like a great PoC would have been to bridge this gap from Pub/Sub to BigQuery with a workflow, but if App Engine, Run or Functions would still need to sit in the middle, it’s just adding extra complexity rather than removing it, which is a shame.

Are direct calls from Pub/Sub something you think we may see in the future?

from workflows-samples.

Hi, thanks for the use-case. Maybe this comment also helps.

Right now you'll need Pub/Sub -> Compute (like Functions/Run) -> Workflow

We have a new product Eventarc (http://cloud.google.com/eventarc) that manages event triggers, and can avoid a compute step, but it only supports Cloud Run targets. It's very new (went GA last month) but we only accept 2 inputs (Audit Logs, Pub/Sub) and 1 output (Run) currently. Maybe Workflows could be a target in the future. Then you just need to create the trigger without any compute in the middle.

If you file a feature request in the issue tracker (workflows issue), that helps us track this request:

https://cloud.google.com/workflows/docs/getting-support

I think that waiting for the workflow to be finished is a waste of resources and therefore money.

Oh, to not wait for the workflow to be finished, just return and don't sleep/poll for the status.

  await executionClient.createExecution({
    parent: executionClient.workflowPath(projectId, location, workflow),
  });

Any thoughts on how to implement an idempotent pubsub trigger for workflows which can be retried on failure?

I'm not sure this helps, a Pub/Sub push subscription will use it's retry policy in case of failure, using the response code.

Are direct calls from Pub/Sub something you think we may see in the future?

Eventarc is the Google Cloud event trigger product, so I can imagine a Workflow event sink in the future. Please link the public issue here if you create one using the link above!

from workflows-samples.

@grant thank you. I just want to add here a link to your great article making clear the "Compute (like Functions/Run) -> Workflow" part. https://medium.com/google-cloud/call-a-workflow-from-a-function-51fc7fc8e1ff

Also there are already two similar feature requests in the Google's task tracker:

1. Pub/Sub trigger https://issuetracker.google.com/issues/175903554
2. Make Workflows accept and parse messages from Pub / Sub #integrations #push-format
   https://issuetracker.google.com/issues/175019114

from workflows-samples.

Hey @iamacarpet, I don't think you can directly execute a workflow from a Pub/Sub subscription with GCP right now. I think you'll need to add a compute step (Functions or Run) that executes the workflow.

Here's a little code in Node that might help (we're working on improving the docs and samples), maybe adapt it and put it in a Cloud Function:

const projectId = 'my-project';
const location = 'us-central1';
const workflow = 'quickstart';

const {ExecutionsClient} = require('@google-cloud/workflows');
const executionClient = new ExecutionsClient();

async function runWorkflow() {
  // Execute workflow
  const createExecutionRes = await executionClient.createExecution({
    parent: executionClient.workflowPath(projectId, location, workflow),
  });
  console.log(`Created execution: ${createExecutionRes[0].name}`);

  function sleep(ms) {
    return new Promise((resolve) => {
      setTimeout(resolve, ms);
    });
  }

  // Wait for execution to finish and print results:
  // TODO: poll for result
  await sleep(5000);
  const executionName = createExecutionRes[0].name;
  const execution = await executionClient.getExecution({
    name: executionName,
  });
  console.log(execution[0].result);
}
runWorkflow();

Hope that helps.

from workflows-samples.

Again, hope that helps. If there are further questions, feel free to comment. Thanks.

from workflows-samples.

Either have this question. Thought I can simplify the data processing pipelines with workflows but now I see there is no PubSub trigger for them.

Thanks for sharing the workaround. I think that waiting for the workflow to be finished is a waste of resources and therefore money. Any thoughts on how to implement an idempotent pubsub trigger for workflows which can be retried on failure?

from workflows-samples.

Workflows can now be triggered with Pub/Sub messages via Eventarc.

from workflows-samples.