I want to create a document of Cloud Firestore, and containing the timestamp value. but, there isn't such a function that converts a floating value(sys.now) to timestamp format. Doesn't the function that plays the same role as below datetime.timestamp() exist?

Thank you.

Sample

main:
  params: [args]
  steps:
  - init:
      assign:
        - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}

  - test_firestore:
      call: googleapis.firestore.v1.projects.databases.documents.createDocument
      args:
        parent: ${"projects/" + project_id + "/databases/(default)/documents/xxx"}
        collectionId: yyy
        documentId: xxx-yyy
        body:
          fields:
            name:
              stringValue: kim
            created_at:
              timestampValue: ${datetime.timestamp(sys.now, 'JST')}
      result: resp

  - check:
      return: ${resp}

Parallel step samples need clean up for GA. See comments from PR#126.

Expected Behavior

There should be very clear instructions on how to provide autocompletion for workflow files. And an image/GIF.

Expected Behavior

Samples that use expressions for values of array types should pass schema check.

Actual Behavior

Samples that have an expression in place of an actual array raise an error when checked against schema.

Steps to Reproduce the Problem

1. Test each sample against schema with ajv:
   ajv -c ajv-formats -s workflows.yaml -d iterate_googleapis_translate.workflows.json
2. Output will include error:

     {
       instancePath: '/1/translateToFrench/for/in',
       schemaPath: '#/properties/in/type',
       keyword: 'type',
       params: { type: 'array' },
       message: 'must be array'
    },
   

Right now I implement a system with GCP workflow but I have some problem about how to organizing the yaml file which have a lot of LOC. However, in some platform fix this problem by using multiple file such as gitlab CICD that you can import another yaml file into the another one.

Does anyone have any suggestion ?

Expected Behavior

Expected that Workflows schema allows try property of the step object to be an stepArray object, as demonstrated by samples error_retry_http_other.workflows.yaml and parallel_error_handling.workflows.yaml

Actual Behavior

Current schema defines try property of the step object as a step object itself. The samples are using a stepArray object as a value for try causing an additionalProperty error.

Steps to Reproduce the Problem

1. Test each sample against schema with ajv:
   ajv -c ajv-formats -s workflows.yaml -d error_retry_http_other.workflows.yaml
2. Output will include error:

     {
     instancePath: '/0/catchErrors/try',
     schemaPath: '#/additionalProperties',
     keyword: 'additionalProperties',
     params: { additionalProperty: 'steps' },
     message: 'must NOT have additional properties'
   }
   

Expected Behavior

Working workflow based connector_read_firestore.workflows.yaml example

Actual Behavior

Workflow execution fails with:

in step "readFromFirestore": {"message":"TypeError: unsupported operand type for call: 'dict' (expecting callable)","tags":["TypeError"]}

Steps to Reproduce the Problem

1. Create new workflow
2. Copy workflow definition from connector_read_firestore.workflows.yaml into workflow
3. Execute workflow

Expected Behavior

I should be able to find examples of common text operations, such as global replacements or stripping out an undesired character from a file using text.replace, text.replace_all, text.replace_all_regex, etc.

Actual Behavior

These examples don't exist.

• Version: all
• Platform: all

The documentation on https://cloud.google.com/workflows/docs/create-workflow-gcloud#yaml fails with the final run command which is listed as:

gcloud workflows run myFirstWorkflow

The error occurs at this line:

Error message:

The argument of 'in' must be a dict or an array; got: null

Expected Behavior

connectors/connector_firestore.workflows.yaml describe Firestore Connector

Actual Behavior

BigQuery Connector

also https://cloud.google.com/workflows/docs/reference/googleapis/firestore/Overview

Expected Behavior

The schema is up-to-date with the workflows syntax available in the official docs.

Actual Behavior

The schema is outdated. E.g. except + as + steps is not recognized.

🤖 Auto-generated Issue.

status: cancelled
commit: 8f62a98
run:https://github.com/GoogleCloudPlatform/workflows-samples/runs/16899097271
buildURL: https://console.cloud.google.com/cloud-build/builds/19caf214-1b19-48ad-b40d-6d6fa460075b?project=798560053668

Samples are not working, I was trying to set params by hand with different variations but all the time I'm getting errors, often TypeError :/ Really documentation for Workflows looks outdated.

Expected Behavior

The value set in the result of Cloud Build Connector is accessible from the next step.

Actual Behavior

Invalid value for "vars" parameter: vars map does not contain key "resp"

Steps to Reproduce the Problem

main:
  params: [args]
  steps:
  - init:
      assign:
        - project_id: $${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}

  - setup:
      call: googleapis.cloudbuild.v1.projects.builds.create
      args:
        projectId: ${project_id}
        body:
          source:
            repoSource:
              repoName: xxx
              branchName: main
          steps:
          - name: "ubuntu"
            args: echo $SHORT_SHA > $BUILDER_OUTPUT/output
      result: resp

  - check:
      return: ${resp}

Specifications

• Version: Google Provider v3.70.0
• Platform: Terraform v0.12.31

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update actions/checkout digest to 44c2b7a
• chore(deps): update dependency flask to v3.0.3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Expected Behavior

Removing the LIMIT statement from the example code should allow for Workflows to iterate through all the results in the example table via pagination, thus circumventing the memory limit of Workflows.

Actual Behavior

When removing the LIMIT statement, the workflow execution fails with the following error message:

HTTP response exceeded the limit of 2097152 bytes
in step "startQuery", routine "main", line: 8
{
  "message": "HTTP response exceeded the limit of 2097152 bytes",
  "tags": [
    "ValueError"
  ]
}

Steps to Reproduce the Problem

Run the following workflows code:

main:
  params: [input]
  steps:
    - init:
        assign:
          - pageToken: null
    - startQuery:
        call: googleapis.bigquery.v2.jobs.query
        args:
          projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          body:
            useLegacySql: false
            # Remove LIMIT from the query to iterate through all results
            query: SELECT name, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name ORDER BY total DESC
        result: query
    - getPage:
        call: googleapis.bigquery.v2.jobs.getQueryResults
        args:
          projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          jobId: ${query.jobReference.jobId}
          maxResults: 10
          pageToken: ${pageToken}
        result: page
    - processPage:
        for:
          value: row
          in: ${page.rows}
          steps:
            - processRow:
                call: sys.log
                args:
                  data: ${row}
    - checkIfDone:
        switch:
          - condition: ${"pageToken" in page and page.pageToken != ""}
            assign:
              - pageToken: ${page.pageToken}
            next: getPage

Fix

The example code needs to be updated to use the endpoint googleapis.bigquery.v2.jobs.insert (with a config block etc.) instead of googleapis.bigquery.v2.jobs.query as the insert endpoint returns a job object instead of the full results.

Suggestion (YAML, but JSON example also needs to be updated)

main:
  params: [input]
  steps:
    - init:
        assign:
          - pageToken: null
    - startQuery:
        call: googleapis.bigquery.v2.jobs.insert
        args:
          projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          body:
            configuration:
              query: 
                useLegacySql: false
                # Remove LIMIT from the query to iterate through all results
                query: SELECT name, SUM(number) AS total FROM `bigquery-public-data.usa_names.usa_1910_2013` GROUP BY name ORDER BY total DESC LIMIT 50
        result: query
    - getPage:
        call: googleapis.bigquery.v2.jobs.getQueryResults
        args:
          projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          jobId: ${query.jobReference.jobId}
          maxResults: 10
          pageToken: ${pageToken}
        result: page
    - processPage:
        for:
          value: row
          in: ${page.rows}
          steps:
            - processRow:
                call: sys.log
                args:
                  data: ${row}
    - checkIfDone:
        switch:
          - condition: ${"pageToken" in page and page.pageToken != ""}
            assign:
              - pageToken: ${page.pageToken}
            next: getPage

Expected Behavior

The code from https://github.com/GoogleCloudPlatform/workflows-samples/blob/main/src/bigquery_pagetoken.workflows.yaml to allow you to page through BigQuery results as reported.

Found code via https://cloud.google.com/workflows/docs/paginate-bigquery-results

Actual Behavior

The code sample uses googleapis.bigquery.v2.jobs.query instead of googleapis.bigquery.v2.jobs.insert so it creates a synchronous BigQuery API call, returning all rows which overflows Workflows execution if of any size.

Steps to Reproduce the Problem

1. Run code on https://github.com/GoogleCloudPlatform/workflows-samples/blob/main/src/bigquery_pagetoken.workflows.yaml with a large BigQuery table

Fix

I replaced the code with googleapis.bigquery.v2.jobs.insert and it worked:

    - startQuery:
        call: googleapis.bigquery.v2.jobs.insert
        args:
            projectId: ${projectId}
            body:
                configuration:
                    query:
                        useLegacySql: false
                        query: ${bq_query}

Expected Behavior

iterate_googleapis_translate.workflows.yaml args should use supported properties as indicated by schema: url, method, headers, body, query, auth, timeout, seconds

Actual Behavior

Samples uses unsupported properties for step.args object: q, target, format, source

Steps to Reproduce the Problem

1. Test sample against schema with ajv: ajv -c ajv-formats -s workflows.yaml -d iterate_googleapis_translate.workflows.yaml
2. Output will include error:

     {
       instancePath: '/1/translateToFrench/for/steps/1/translate/args',
       schemaPath: '#/properties/args/additionalProperties',
       keyword: 'additionalProperties',
       params: { additionalProperty: 'q' },
       message: 'must NOT have additional properties'
     }
   

There should instructions for how to update a schema when developing (in VS Code).

Example settings:

"json.schemas": [
        {
            "name": "workflows.json",
            "description": "Google Cloud Workflows configuration file",
            "fileMatch": [
                "workflows.json",
                "*.workflows.json",
            ],
            "url": "https://json.schemastore.org/workflows"
          }
    ],
    "yaml.schemas": {
        "https://json.schemastore.org/workflows": [
            "workflows.yaml",
            "workflows.yml",
            "*.workflows.yaml",
            "*.workflows.yml"
        ]
    },

Expected Behavior

Some developers prefer JSON instead of YAML. We should generate JSON samples in addition to JSON samples.

Actual Behavior

We only have YAML samples.

There are a few developer resource we could link in this repo:

• https://github.com/meteatamel/workflow-samples
• https://github.com/filipknapik/google_workflows
• https://www.youtube.com/watch?v=Uz8G8fTwwXs
• https://atamel.dev/posts/2020/09-08_first_look_at_workflows

Maybe we can make it easier to add community samples, somehow?

https://github.com/GoogleCloudPlatform/community

How to add a condition for contains?
e.g

"response" : {
"Bundle" : "1"
}

How is check if the response is empty?

Actual Behavior

Samples use "dummy" in example names

Expected Behavior

Use a different value such as, "placeholder", "test", "example", "sample", etc

@camiekim I can fix this for the samples, but can you let me know if we need to do a synchronous change i.e. should I wait for any corresponding doc updates?

I have been trying saga pattern in cloud workflows, since it is workflow orchestration error handling and rollback is important thing here so i have been trying this, but i needed a feature request on passing custom next functions on sub workflows in order to achieve this pattern to work, otherwise it would have more bloated code to do this same without that feature.

---
main:
  steps:
    - book-flight:
        call: post
        result: flightData
        args:
            url: https://***.ngrok.io/book-flight
            next: cancel-flight
    - book-hotel:
        call: post
        result: hotelData
        args:
             url: https://***.ngrok.io/book-hotel
             next: cancel-hotel
    - book-taxi:
        call: post
        result: taxiData
        args:
          url: https://***.ngrok.io/book-taxi
          next: cancel-taxi
    - cancel-flight:
        call: post
        result: flightCancelData
        args:
          url: https://***.ngrok.io/cancel-flight
          next: returnData
    - cancel-hotel:
        call: post
        result: hotelCancelData
        args:
          url: https://***.ngrok.io/cancel-hotel
          next: returnData
    - cancel-taxi:
        call: post
        result: taxiCancelData
        args:
          url: https://***.ngrok.io/cancel-taxi
          next: returnData
    - returnData:
        return: ${"Flight Res "+flightData.body.message+" Hotel Res "+ hotelData.body.message+ " Tax Res "+taxiData.body.message}
post:
  params: [url, next]
  steps:
    - postAPI:
        try:
          call: http.post
          args:
            url: ${url}
            headers:
              Content-Type: text/plain
              Content-Accept: text/plain
            query: {}
          result: data
        retry: "${http.default_retry_non_idempotent}"
        except:
          as: e
          steps:
            - callNext:
                next: ${next}
    - returnData:
        return: ${data}

When i deploy this workflow i get a error message failed to build: error calculating label indices: attempt to jump to non-existing step ${next}.

Expected Behavior

api called inside the workflow should get the datatype of the input parameters as given in json.

input json as below
{
"candidate_project_key": 1,
"development_phase": "Phase III",
"end_date": "",
"is_archive_only": false,
"start_date": "",
"study_description": "TEST_7OCT21_3",
"study_id": "TEST_11OCT21_3",
"study_title": "TEST_11OCT21_3",
"therapeutic_area_key": 78,
"token": "eyJraWQiOiJsdf1cm9mU1RtNGNwRnFvM3hJVjdVblVZUzVfVXRzcE1JNk1pXzhEY2tPdHFBIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULjNUR1Bic0kxNVFMNFB6NHlIVzdHQTdBZUtCVTUxQXdQTHYsd1bS1jQ3YxOTAiLCJpc3MiOiJodHRwczovL2ludGllbnQtZGV2Lm9rdGFwcmV2aWV3LmNvbS9vYXV0aDIvYXVzcmVxNGJoY1dQdlFna1YwaDciLCJhdWQiOiJpbnRpZW50IiwiaWF0IjoxNjMzOTMzNzU0LCJleHAiOjE2MzM5"
}

Actual Behavior

here even though the json input is having integer, boolean values the api is being passed with all the values as string.

Steps to Reproduce the Problem

1. create an dummy api which expects integer, boolean and string values as inputs
2. create an workflow to call the above created api and print the types of the input parameters
3. run the workflow with a json having boolean, string and interger values

Below is the workflow yaml
main:
params: [run_args]
steps:

• step1:

  call: http.post
  args:
      url: https://internal_application.com/web/clin/api/dataservices/studies
      headers: 
        content-type: application/json
        authorization: ${"Bearer "+run_args.token}
      body:
          "study_title": run_args.study_title
          "therapeutic_area_key": run_args.therapeutic_area_key
          "development_phase": run_args.development_phase
          "study_description": run_args.study_description
          "study_id": run_args.study_id
          "candidate_project_key": run_args.candidate_project_key
          "start_date": run_args.start_date
          "end_date": run_args.end_date
          "is_archive_only": run_args.is_archive_only
  result: api_response
  

Specifications

google cloud workflow

• Version:
• Platform: google cloud platform

#81 attempted to fix a syntax error in connect_firestore_read:

document_found step referenced camelCased documentVale but read_item result uses document_value.

This needs to be corrected in the YAML or in the generation process. I have not verified the casing is a bug.

The connector_sheets.workflows.yaml sample shows how to create a new spreadsheet and update it using the connector. But it doesn't show how to update an existing spreadsheet. I believe this is more complicated as it'll have to involve authenticating the workflow with the user account where the spreadsheet lives. However, I think this would be a common use case that we should have a sample for.

Expected Behavior

"main" is the default branch

Actual Behavior

"master" is the default branch

Expected Behavior

To better support autocomplete, the workflows schema requires a definition for the for property in the step definition

Documented usage: https://cloud.google.com/workflows/docs/reference/syntax/iteration

Expected Behavior

Could you please add a sample for bigquery request using queryParameters[] and googleapis.bigquery.v2.jobs.query ?

After many tries, i'm still not able to use a query with a variable named : @table_suffix.

My query is quite simple as we can do using an other client library :

        const sqlQuery = `SELECT * FROM \`project_id.dataset.*\` WHERE _TABLE_SUFFIX = @table_suffix LIMIT 5`;

        const options = {
            query: sqlQuery,
            location: 'EU',
            params: { table_suffix: 'events_intraday_20211202' },
        };

if i follow the snippet i should use "queryParameters" parameters but if i check the documentation of the object "QueryRequest" i should use queryParameters[]
https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/v2/jobs/query#subworkflow_snippet
https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/v2/Overview#queryrequest

I try this

- init:
    assign:
    - table_suffix: "events_intraday_20211202"
    - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
    - query: "SELECT * FROM `project_id.dataset.*` WHERE _TABLE_SUFFIX = @table_suffix LIMIT 5"

- query:
    call: googleapis.bigquery.v2.jobs.query
    args:
        projectId: ${project_id}
        body:
            query: ${query}
            useLegacySql: false
            queryParameters[]: 
                name: "table_suffix"
                parameterType:
                    type: "string"
                parameterValue:
                    value: "events_intraday_20211202"
    result: queryResult
- documentFound:
    return: ${queryResult}

I will appreciate your help.

Expected Behavior

To better support autocomplete, the workflows schema requires a definition for the parallel property in the step definition

Documented usage: https://cloud.google.com/workflows/docs/reference/syntax/parallel-steps

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

The instructions for running a sample should be 1 step, and not produce artifact files, like this:

WORKFLOW=myFirstWorkflow

# Create workflow
gcloud beta workflows deploy $WORKFLOW \
--source src/$WORKFLOW.workflows.yaml

# Execute workflow
EXE=$(basename $(gcloud beta workflows execute $WORKFLOW --format='value(name)')) &&
gcloud beta workflows executions describe $EXE \
--workflow $WORKFLOW \
--format='value(result)' | python -m json.tool

Is there a trick to using the sys.now() function — I'm having trouble getting that to work properly in my workflows.
I couldn't find any specific documentation around it though.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Struggling to find where to ask this question, but has anyone had luck triggering a Workflow from a Pub / Sub message?

It looks like it's possible to configure the execution to fire using a HTTP subscription.

However, it doesn't look like the Workflow would be able to access the event data sent in the request.

Our backend already writes to Pub / Sub, so being able to start handling events in Workflows would be amazing.

Expected Behavior

Tuesday

Actual Behavior

error 404

Steps to Reproduce the Problem

curl https://us-central1-workflowsample.cloudfunctions.net/datetime
1.
1.
1.

Specifications

• Version:
• Platform:

Expected Behavior

Sample provided here should work without changes.

Actual Behavior

args.bucketName used on output_storage_file_prefix parameter is not listed anywhere

Steps to Reproduce the Problem

1. Copy & paste the sample code to a new workflow
2. Try to execute it
3. Error should appears

Specifications

• Version: Console UI
• Platform: Console UI