fission / fission-workflows Goto Github PK
View Code? Open in Web Editor NEWWorkflows for Fission: Fast, reliable and lightweight function composition for serverless functions
License: Apache License 2.0
Workflows for Fission: Fast, reliable and lightweight function composition for serverless functions
License: Apache License 2.0
The env create command has the wrong image tag (it is 0.3.0
not v0.3.0
):
The right command is:
fission env create --name binary --image fission/binary-env:0.3.0
but the NOTES contains the command with an extra 'v' in the tag.
Sometimes when a workflow is invoked once, some of the tasks are invoked more than once.
(Sorry, I don't have more info right now.)
Currently the default pull policy of the environments are "IfNotPresent" and it is not possible to change it without manually updating the deployments.
It would be nice to be able to update the env with a --force which would trigger a forced update even if running on "latest"
This is not a big deal, especially with packaging coming up, just a nice to have.
Let's use this issue to keep track of workflow visualization. We also want to implement a GUI editor for workflows. There's enough in common between these two efforts that we can use the same issue for now.
Does this project really need to use Bazel?
It seems that the workflow-env gets killed/terminated. Does get rebooted properly.
Given that a major emphasis is made on re-using workflows and recursion over iteration. It is vital for the workflow engine to have some sort of recursion detection.
Solution 1
A "parent" parameter could be added (as a header for example) that links a workflow to its parent workflow. From there the workflow engine can analyze the workflow dependency graph to determine whether a infinite recursion is occurring
Potential issues:
Solution 2
Add a stack parameter to the request, containing all preceding workflow invocation ids. The workflow engine can infer can simply call a stack overflow if the stack becomes to large.
Potential issues:
...
Performance of the workflow engine is key. So it should be measured accordingly. This issue tracks the progress of that effort.
Thoughts on it:
Currently FW uses the poolmgr and controller to resolve and invoking. This was a temporary solution as the router lacked the capability to route to a specific function, based on UID rather than name.
It makes more sense to use the router, because we avoid duplication in things such as caching. However we need to explore/think a bit about how the router fits into this, and what still needs to be changed
@soamvasani fyi
There used to be (in the JSON format) support to force the type of an input. This functionality should also be supported in the YAML format.
From the start the internal functions have been designed to be fully pluggable. This means that the user should be able to dynamically add internal functions. Due to prototyping and time constraints this functionality has not made it in yet.
The solution will probable need a HTTP/gRPC endpoint to which users can upload a golang plugin. The system validates it and makes it available to the workflows
Similar to #64 we also might want to allow users to extend the functionality of the expression engine. Making it pluggable will most likely result in something similar to internal functions: a user uploads a golang plugin to an endpoint, the system validates it and makes it available to the expressions in the workflows.
Once it is merged into Fission the builder functionality should be reused to translate YAML to the JSON workflow format. This allows us to keep the workflow engine from having to deal with other formats other than the JSON format.
I have a very simple workflow like:
apiVersion: 1
description: Collect a form - computes a risk then sends SMS with result
output: CreateResult
tasks:
# Fetch incoming data and update status. Computes risk.
SetToPending:
run: set-to-pending
inputs: "{$.Invocation.Inputs.default}"
# Adds a status depending on value of the risk
MakeDecision:
run: make-decision
inputs: "{$.Tasks.SetToPending.Output}"
requires:
- SetToPending
SendSMS:
run: if
inputs:
if: "{$.Tasks.SetToPending.Output.risk > 70}"
then:
run: send-sms
inputs: "{$.Tasks.MakeDecision.Output}"
else:
run: send-sms
inputs: "{$.Tasks.MakeDecision.Output}"
requires:
- SetToPending
- MakeDecision
CreateResult:
run: compose
inputs:
id: "{$.Tasks.SetToPending.Output.id}"
status: "{$.Tasks.MakeDecision.Output.status}"
risk: "{$.Tasks.MakeDecision.Output.risk}"
sid: "{$.Tasks.SendSMS.Output.sid}"
requires:
- SetToPending
- MakeDecision
- SendSMS
the inputs are supposed to be JSON. I made a mistake calling the function and my JSON was malformed. Then the workflow engine goes crazy and retries for ever:
$ stern workflow --namespace fission-function
"2017-10-20T08:47:17Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:17Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:17Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:17Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:17Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a2e9050}
"2017-10-20T08:47:18Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:18Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:18Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:18Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:18Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:18Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:18Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:18Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:18Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:18Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a41aa70}
"2017-10-20T08:47:19Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:19Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:19Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:19Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:19Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:19Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:19Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:19Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:19Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:19Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a326120}
"2017-10-20T08:47:20Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:20Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:20Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:20Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:20Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:20Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:20Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:20Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:20Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:20Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a098310}
"2017-10-20T08:47:21Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:21Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:21Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:21Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:21Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:21Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:21Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:21Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:21Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:21Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3bdb00}
"2017-10-20T08:47:22Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:22Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:22Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:22Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:22Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:22Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:22Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:22Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1b678bf0}
"2017-10-20T08:47:23Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:23Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:23Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:23Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:23Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:23Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:23Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:23Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:23Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:23Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a09e090}
"2017-10-20T08:47:24Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:24Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:24Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:24Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:24Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:24Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:24Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:24Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:24Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:24Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a098160}
"2017-10-20T08:47:25Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:25Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:25Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:25Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:25Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:25Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:25Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:25Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:25Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:25Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3d7ef0}
"2017-10-20T08:47:26Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:26Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult
"2017-10-20T08:47:26Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS
"2017-10-20T08:47:26Z" level=info msg="task found on horizon" task=SendSMS
"2017-10-20T08:47:26Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b
"2017-10-20T08:47:26Z" level=info msg="Invoking function 'if' for task 'SendSMS'"
"2017-10-20T08:47:26Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:26Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000" val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"
"2017-10-20T08:47:26Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"
"2017-10-20T08:47:26Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3f1df0}
I could not find a way to stop that. Currently the only option I found is to do
Is there another and better way?
Thanks!
Users shouldn't have to learn and install two separate CLIs.
Currently the workflow-engine, due to prototyping reasons, is a separate component within the fission ecosystem. The plan is to ensure that the workflow engine is just another environment.
The benefits of it being a environment:
To achieve this
The fission controller proxy for the workflows apiserver doesn't work if workflows is installed in a different namespace from the fission controller.
$ wfcli status
panic: unknown error (status 502): {resp:0x10a78190}
$ kubectl -n fission logs controller...
2017/10/25 15:53:42 http: proxy error: dial tcp: lookup workflows-apiserver on 10.43.240.10:53: no such host
127.0.0.1 - - [25/Oct/2017:15:53:42 +0000] "GET /proxy/workflows-apiserver/status HTTP/1.1" 502 0
To fix this, the proxy must either discover the workflows installation somehow (maybe through a configmap) or we should require workflows to be installed in the same namespace as fission (not even sure that's possible given that helm allows the user to override the installation namespace).
Currently users can only specify a single task or a reference to another workflow in a dynamic task. To avoid having to write workflows for small, trivial or application-specific use cases, it should be possible to write a inline workflow, analogous to how lambda's are used in Java/python/etc.
Ideally the inline workflow should match the workflow definition as closely as possible.
Currently the user needs to manually compile their yaml workflow definition to json using the wfparser
cli tool.
The intention is have this abstracted away from the user by using the builder in Fission. This should allow users to provide the yaml definitions directly to Fission.
Currently internal functions (like if, noop and sleep) are hardcoded into the workflow engine, though they are built to be extensible. A user should be able to add internal functions during runtime, without the need to rebuild/redeploy the workflow engine. Probably requires a separate endpoint that allows loading (http or shared volume) of Golang plugins.
We need a way to either manually or automatically clean up old workflow invocation data.
Mention the following:
$
, task
, output
uid()
, input(task, <key>)
, output(task)
, param(<key>)
User should be able to define dynamic tasks (such as loops and other higher order functions) within a workflow definition, using Javascript or Go, and have them interpreted by the workflow engine as part of workflow execution.
To make dynamic task definition easy, the workflow engine should do most of the work of type checks etc. around dynamic task inputs and outputs. The user should have to write as little code as possible.
Both implementations (Javascript and Go) would involve the workflow engine loading the dynamic task and running it.
The Javascript implementation can use the existing Javascript interpreter, which already allows us to isolate functions as well as run them with timeouts. That will ensure that a misbehaved dynamic task from one workflow won't affect other workflows. If we solve #76 using Javascript, then that same set of primitives can be used within the dynamic task definition.
The Go implementation will need the builder to compile the Go tasks as one or more plugins. The workflow engine would load these as plugins and run them. It would have to do this in a separate process, for isolation; otherwise a misbehaving plugin could affect other workflows. There's a little more complexity in the Go implementation because of this out-of-process requirement. Also, the user would need to learn how to specify a workflow in Go using the workflow engine's type system, which is non-trivial because of some of the dynamic typing that the workflow engine does.
Overall, Javascript sounds like a better fit from a developer experience point of view, with the only advantage of the Go implementation being that you can probably get more performance out of it than the interpreted Javascript.
Pick either Python or Javascript, and write a set of classes and functions that allow a user to specify a workflow in that language, instead of YAML.
Most of the workflow engines contain some sort of a visualizer to show the execution of a workflow, which improves the monitor-ability and debug-ability of workflows.
The UI can be divided in two parts:
The editor is currently considered out of scope for the coming months, but the visualizer is definitely an important part of the workflow experience (especially for demonstrations).
Potential visualizer features:
Partial application should allow users to create workflows by just specifying a part of the arguments.
This would already be useful when giving demo's. For example, you want to provide a webhook for a slack command, but the workflow also requires apikeys and other configs to be set. You create a partial application of the workflow by providing all the apikeys. This leaves just the argument (in this case the formdata-body) to be provided by Slack.
The current examples in the readme are old and 'abstract'. They should be updated to actually functional examples
As discussed with @soamvasani
Documents:
Currently the JSON workflow definition representation is a direct translation from the internal protobuf workflow spec. Due to the limitations of Protobuf, TypedValues cannot be represented as arbitrary types in the JSON format, which leads to unreadable serialized values (see github.com/fission/fission-workflow/tree/master/examples/whales/maybewhale.wf.json)
We should separate the JSON format from the internal Protobuf data structure to ensure that the inputs remain readable to the user.
Currently only internal functions can be dynamic tasks (tasks that return a task). Support should be added to the fission runtime environment to also allow fission functions to be dynamic tasks.
One approach would be that in the presence of a special header X-Fission-Workflow-Dynamic
the body is interpreted as a task. Or, maybe even better, having the content-type be something like application/vnd+fission.workflow+json
. By specifically requiring the header we avoid outputs to be accidentally being interpreted as dynamic tasks.
This issue tracks the state of the incorporation of workflow management in the existing Fission CLI or into a separate CLI. It might just be that we decide to not integrate any thing into the client specifically to workflows.
(Proposed) API functionality:
Roadmap:
The output
field can be used to manipulate or override the output of a task. This would prevent the need of an additional transformation task or tasks having to manipulate it in their inputs.
The actual output of a task is calculated by taking the actual function output and applying the output
on to it.
Currently the values are mapped to and from HTTP requests/responses using the content-type header implicitly. Although it currently works out okay for the current use cases, it might not work for things like binary data. Improvement are needed to have a more explicit mapping between typedvalues and http requests/responses.
The currently implemented event store was written with a limited knowledge of NATS streaming, abusing some of the features.
Currently the naieve setup consists of the following subject structure
workflow.<id>
: each subchannel contains one workflow.invocation.<id>
: each subchannel contains the activities of one workflow invocation._activity
: NATS streaming doesn't support wildcard subscriptions as of the moment of writing (see nats-io/nats-streaming-server#340 for progress on the feature). In order to replicate the behavior of a wildcard subscription this channel is used to publish notifications of when something happens in one of the various workflow/invocation subchannels. The event store implementation has one 'metasubscriptions' on this channel that creates new subscribers when new subjects appear.Issues with this setup:
Possible solutions:
Currently, for the prototype, this is a low priority issue, as for small usage (<1000 invocations) it works just fine. Nothing is persisted yet, as fission-nats.yml deployment is still using in-memory, and can be cleared by simply restarting that deployment. So, until the prototype is advanced enough that it becomes clear what is needed from the event store, the current implementation is okay.
Feature request: allow single-stepping through workflows while being able to inspect task inputs and outputs.
Already partly supported, but needs to be fleshed out.
We need a service separate from the 'workflow engine' that only contains the api server logic, without any active controller. This will be useful with the integration in Fission, essentially creating a read-only fission workflows instance.
The performance optimization that would make use of the advantages of the workflow definition, is to enable the scheduler to pre-warm functions. The controller will poke the Fission functions to warm them up before actually invoking them.
The process would look like this:
Currently there is no check, and a workflow will basically timeout after getting stuck. Due to the static declaration of dependencies between tasks, within the workflow the validation should be trivial.
Currently implicit expressions are recognized only when there is a trailing $
. This lacks flexibility in the expressions. There should be ways to improve this identification of expression types vs. regular strings.
Currently only the default
input is mapped to the body of fission functions.
To support passing inputs to the headers and query of the fission function invocation, we need to add the following mappings in the fission runtime environment.
Inputs:
Optionally, each entry in the headers and queries could also be map to support multiple headers/query params with the same key, but that is not a terrible thing not to support.
Currently workflows are defined using a backward dependencies. For example:
myTask:
...
requires:
otherDepTask1: ~
otherDepTask2: ~
While provides more self-contained tasks, for simple use cases it might be more straightforward to allow for forward dependencies.
myTask:
...
on-complete:
- nextTask1
- nextTask2
On the backend the dependencies should resolve similarly: a task waits for all dependencies, whether specified forward or backward, to resolve. dependencyAwait
allows tasks to wait for only a part of their dependencies. However, with the constraint that individual tasks can only be invoked once, this means that you can only create discriminators not 'multiple instances' (see http://workflowpatterns.com/).
Similar to Openstack's mistral, we can support both. The user should be able to specify which type of dependency definition they want to use.
apiVersion: v1
dependencies: forward|backward
tasks:
...
The default, though open for discussion, would be backward.
Currently the short control loop that loops over the cache in the controller. However, in (edge) cases the invocation might no longer be in the cache, but still need to be handled by the controller. The long control loop (as described in the docs) will run less frequent, but willbe more thorough in finding relevant workflow invocations.
This is a aggregate issue of all minor TODOs not worth a separate issue. Noted here mainly for my own organization
In order to account for some of the requirements of this project: performance, scalability, reliability. A simple benchmark is needed. As the workflow engine is not different from any other Fission function, this benchmark tool should easily be generalizable for any Fission function (some prior work of before my internship exists for this)
Functionality:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.